)]}'
{"/PATCHSET_LEVEL":[{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"132dd0e77b634592d52b799cae88a870096f9e33","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":11,"id":"f9900b4c_f6184c80","updated":"2021-10-28 21:52:09.000000000","message":"I updated the spec and addressed various concerns around the topics discussed during the Yoga PTG (a word of thanks to the Nova core developers for provided feedback).\n\nhttps://etherpad.opendev.org/p/nova-yoga-ptg#L559\n\nNamely:\n\n* Upgrade-related:\n  * Nova compute service versions;\n  * Neutron upgrade considerations;\n* Scheduling-related:\n  * Described the prefilter to be implemented based on a new compute capability (relevant os-traits change will be raised in due course);\n* Instance lifecycle operations and API stubs;\n  * Described the operations that will raise API errors. I may revisit this list after some more investigation but for now the list is as in the uploaded change;\n* Neutron proposal status:\n  * Added a note about the RFE being approved already (the spec is being reviewed).\n\nOther changes:\n\n* Changed the wording regarding PCI VPD serial number extraction since we no longer need to handle that in Nova and can rely on Libvirt to provide this information since my changes to Libvirt got merged;\n* Updated references to relevant OVN series;\n* Added a section around potential future work based on the PCI device tracking in the placement service spec.\n\n","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"82a4915d2026e9d9392daea09a8ca225dd9b84b2","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":11,"id":"6547fe1a_9f5e3afc","updated":"2021-11-12 16:01:46.000000000","message":"Responded to a couple of comments. Will respond to others and resumbit.\n\nThanks a lot for the review!","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"6d0741dc01fb1da50c6203fb6c0414fc6c8557bd","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":11,"id":"663e53c9_2adbd045","updated":"2021-11-10 17:19:04.000000000","message":"The neutron spec has been merged and the nova spec looks good to me. I will wait at least for Sean to review then I will upgrade my vote to +2. ","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"346b4af75bb926b9a5729bdc20e2499d8071d0fa","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":11,"id":"04fac531_fbb7d503","updated":"2021-11-12 15:28:31.000000000","message":"i have 1 or 2 nits in-line but i am happy with the design as presented overall.\ni would be fine with those being addressed in a follow up patch but if you do re spin im happy to re-review on monday.","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"82a4915d2026e9d9392daea09a8ca225dd9b84b2","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":11,"id":"dd65e703_b8b95947","in_reply_to":"04fac531_fbb7d503","updated":"2021-11-12 16:01:46.000000000","message":"Ack, I\u0027ll go over the comments and resubmit.\n\nThanks a lot!","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"37778294da3ed748023f0011beaca0411c520932","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":13,"id":"96175068_1f15e680","updated":"2021-11-15 19:14:24.000000000","message":"The last update adds a clarification about extending ``_get_neutron_events`` in the virt driver code.","commit_id":"80ae5a5fd4af20182679601d62fe9159673bd909"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"d3064e65417b086c74c9b9355801da027ae3ec17","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":14,"id":"4af27a8b_5954b73f","updated":"2021-11-16 15:42:58.000000000","message":"I have no high level design issues with this spec as written bar the whitelist extension which we disucssed on IRC. i have left a nit inline but i think we can move forward with this to the implementation review phase. If we elect to filter out remote manage device by default if we dont request them to prevent them being claimed by pci alisa or more generically filter out all deivec with physical_network we can update this spec to reflect that.\n\ni think not excluding device with physical_network form use with pci passthough si arguable a bug which is why i want to separate that discussion form this spec.\n\nwith that in mind im +2 on this and since gibi is also ill +w as well.","commit_id":"ba6772ae23d3a713fc297431366dd5b26ed02809"},{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"d32565d794682981b668bbc3b604586d9b3749be","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":14,"id":"e79aa98d_d582cba3","updated":"2021-11-16 08:32:12.000000000","message":"The recent changes looks good to me. Sean had +2 before so I\u0027m confident upgrading my +1 to +2 too.","commit_id":"ba6772ae23d3a713fc297431366dd5b26ed02809"}],"specs/xena/approved/integration-with-off-path-network-backends.rst":[{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"87cdb48ebff279ca4fdc74da75312bd4188cd903","unresolved":true,"context_lines":[{"line_number":257,"context_line":""},{"line_number":258,"context_line":"Largely, the goal is to gather the information necessary for representor"},{"line_number":259,"context_line":"plugging via Nova and pass it to the right place."},{"line_number":260,"context_line":""},{"line_number":261,"context_line":"In case PCIe used at the SmartNIC DPU for NIC access, both the hypervisor host"},{"line_number":262,"context_line":"and the SmartNIC DPU host that belong to the same physical machine can see"},{"line_number":263,"context_line":"PCI(e) functions exposed by the controllers on the same card, therefore, they"}],"source_content_type":"text/x-rst","patch_set":9,"id":"8688af09_13447ec6","line":260,"updated":"2021-06-14 19:04:09.000000000","message":"+1 nice short summary of the nova portion.","commit_id":"d2a19227767b9dd5f40bb939921b5263898560e7"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"21f4a7e9a1a10cd54f6cf5359d3be5ad1824be5e","unresolved":false,"context_lines":[{"line_number":257,"context_line":""},{"line_number":258,"context_line":"Largely, the goal is to gather the information necessary for representor"},{"line_number":259,"context_line":"plugging via Nova and pass it to the right place."},{"line_number":260,"context_line":""},{"line_number":261,"context_line":"In case PCIe used at the SmartNIC DPU for NIC access, both the hypervisor host"},{"line_number":262,"context_line":"and the SmartNIC DPU host that belong to the same physical machine can see"},{"line_number":263,"context_line":"PCI(e) functions exposed by the controllers on the same card, therefore, they"}],"source_content_type":"text/x-rst","patch_set":9,"id":"6c093b12_da619bb4","line":260,"in_reply_to":"8688af09_13447ec6","updated":"2021-06-24 18:08:24.000000000","message":"Ack","commit_id":"d2a19227767b9dd5f40bb939921b5263898560e7"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"87cdb48ebff279ca4fdc74da75312bd4188cd903","unresolved":true,"context_lines":[{"line_number":311,"context_line":"    driver will only be bound to VFs, not PFs and that PFs will be utilized for"},{"line_number":312,"context_line":"    hypervisor host purposes (e.g. connecting to the rest of the control"},{"line_number":313,"context_line":"    plane);"},{"line_number":314,"context_line":"  * Storing of VF logical number and PF MAC could be in ``extra_info`` could"},{"line_number":315,"context_line":"    be done to avoid extra database lookups;"},{"line_number":316,"context_line":"* Add logic to handle ports of type VNIC_SMARTNIC (\"smart-nic\")."},{"line_number":317,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"58c12f0b_fcdf33dd","line":314,"range":{"start_line":314,"start_character":44,"end_line":314,"end_character":53},"updated":"2021-06-14 19:04:09.000000000","message":"nit:remove","commit_id":"d2a19227767b9dd5f40bb939921b5263898560e7"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"21f4a7e9a1a10cd54f6cf5359d3be5ad1824be5e","unresolved":true,"context_lines":[{"line_number":311,"context_line":"    driver will only be bound to VFs, not PFs and that PFs will be utilized for"},{"line_number":312,"context_line":"    hypervisor host purposes (e.g. connecting to the rest of the control"},{"line_number":313,"context_line":"    plane);"},{"line_number":314,"context_line":"  * Storing of VF logical number and PF MAC could be in ``extra_info`` could"},{"line_number":315,"context_line":"    be done to avoid extra database lookups;"},{"line_number":316,"context_line":"* Add logic to handle ports of type VNIC_SMARTNIC (\"smart-nic\")."},{"line_number":317,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"862e4445_88e37d07","line":314,"range":{"start_line":314,"start_character":44,"end_line":314,"end_character":53},"in_reply_to":"58c12f0b_fcdf33dd","updated":"2021-06-24 18:08:24.000000000","message":"Will do.","commit_id":"d2a19227767b9dd5f40bb939921b5263898560e7"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"87cdb48ebff279ca4fdc74da75312bd4188cd903","unresolved":true,"context_lines":[{"line_number":312,"context_line":"    hypervisor host purposes (e.g. connecting to the rest of the control"},{"line_number":313,"context_line":"    plane);"},{"line_number":314,"context_line":"  * Storing of VF logical number and PF MAC could be in ``extra_info`` could"},{"line_number":315,"context_line":"    be done to avoid extra database lookups;"},{"line_number":316,"context_line":"* Add logic to handle ports of type VNIC_SMARTNIC (\"smart-nic\")."},{"line_number":317,"context_line":""},{"line_number":318,"context_line":"Identifying Port Representors"}],"source_content_type":"text/x-rst","patch_set":9,"id":"2467e01d_fbf7ed58","line":315,"range":{"start_line":315,"start_character":36,"end_line":315,"end_character":43},"updated":"2021-06-14 19:04:09.000000000","message":"lookups and schema changes","commit_id":"d2a19227767b9dd5f40bb939921b5263898560e7"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"21f4a7e9a1a10cd54f6cf5359d3be5ad1824be5e","unresolved":true,"context_lines":[{"line_number":312,"context_line":"    hypervisor host purposes (e.g. connecting to the rest of the control"},{"line_number":313,"context_line":"    plane);"},{"line_number":314,"context_line":"  * Storing of VF logical number and PF MAC could be in ``extra_info`` could"},{"line_number":315,"context_line":"    be done to avoid extra database lookups;"},{"line_number":316,"context_line":"* Add logic to handle ports of type VNIC_SMARTNIC (\"smart-nic\")."},{"line_number":317,"context_line":""},{"line_number":318,"context_line":"Identifying Port Representors"}],"source_content_type":"text/x-rst","patch_set":9,"id":"d5d0a211_305d9cc3","line":315,"range":{"start_line":315,"start_character":36,"end_line":315,"end_character":43},"in_reply_to":"2467e01d_fbf7ed58","updated":"2021-06-24 18:08:24.000000000","message":"Ack, will fix.","commit_id":"d2a19227767b9dd5f40bb939921b5263898560e7"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"87cdb48ebff279ca4fdc74da75312bd4188cd903","unresolved":true,"context_lines":[{"line_number":466,"context_line":"  capability (initially, just the board serial number but it may be extended at"},{"line_number":467,"context_line":"  a later point if needed)."},{"line_number":468,"context_line":""},{"line_number":469,"context_line":"Periodic hypervisor resource updates will add newly discovered PciDevices and"},{"line_number":470,"context_line":"get the associated card serial number information. However, old devices will"},{"line_number":471,"context_line":"not get this information without explicit action."},{"line_number":472,"context_line":""},{"line_number":473,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"af6758d9_cab8333b","line":470,"range":{"start_line":469,"start_character":0,"end_line":470,"end_character":50},"updated":"2021-06-14 19:04:09.000000000","message":"-1 to this.\n\ncurrently the perodic task does retrive the pci info in update_avaiable_resouces but it never updates\nit in the db or in memory.\n\nthe db should only be updated when the compute agent starts as it is done today.\nits debatabel if we we even want to keep the current behavior in the perodic task that we have today.\n\nnova doe s not currently support adding or removing pci device to the hypervior without a compute agent and somethimes\nlibvirt restart.\n\nWith my downstream hat on we dont support hardware updated inplace as we do not support chaning the number of VF or updating\npci whitelist whiel vms are on the hosts. its too easy to break exisitng vms if we allowed that. you have to drain\nthe host of vms then perform the hardware updates and then you can use it for new vms.\n\ni am not sure that it makes sense to extend upstream to track device change at runtime in a periodic upstream either\nas nova has no support for moving allocation betwween equivalent devices if a device that was in use was removed.\n\n\nif other agree we could adress this in a followup as this is more of an implemeation nit.","commit_id":"d2a19227767b9dd5f40bb939921b5263898560e7"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"21f4a7e9a1a10cd54f6cf5359d3be5ad1824be5e","unresolved":true,"context_lines":[{"line_number":466,"context_line":"  capability (initially, just the board serial number but it may be extended at"},{"line_number":467,"context_line":"  a later point if needed)."},{"line_number":468,"context_line":""},{"line_number":469,"context_line":"Periodic hypervisor resource updates will add newly discovered PciDevices and"},{"line_number":470,"context_line":"get the associated card serial number information. However, old devices will"},{"line_number":471,"context_line":"not get this information without explicit action."},{"line_number":472,"context_line":""},{"line_number":473,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"de6fbdcf_523febbe","line":470,"range":{"start_line":469,"start_character":0,"end_line":470,"end_character":50},"in_reply_to":"af6758d9_cab8333b","updated":"2021-06-24 18:08:24.000000000","message":"Fair enough, I was trying to be as generic as possible since saw this in the Nova code base:\n\nhttps://opendev.org/openstack/nova/src/tag/23.0.1/nova/pci/manager.py#L106-L108\n\"        To support pci device hot plug, we sync with the hypervisor\n        periodically, fetching all devices information from hypervisor,\n        update the tracker and sync the DB information.\"\n\nI have not seen a lot (if any) requests for PCIe hot-plug in my practice so this is definitely not something I am looking for in the initial implementation.\n\nWill adjust the text accordingly.","commit_id":"d2a19227767b9dd5f40bb939921b5263898560e7"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"87cdb48ebff279ca4fdc74da75312bd4188cd903","unresolved":true,"context_lines":[{"line_number":587,"context_line":"primary power from a PCIe slot so their power lifecycle is tied to the"},{"line_number":588,"context_line":"main board lifecycle. This should be taken into consideration when performing"},{"line_number":589,"context_line":"power off/power on operations on the hypervisor hosts as it will affect"},{"line_number":590,"context_line":"services running on the SmartNIC DPU (a reboot of the hypervisor host should"},{"line_number":591,"context_line":"not)."},{"line_number":592,"context_line":""},{"line_number":593,"context_line":"Developer impact"}],"source_content_type":"text/x-rst","patch_set":9,"id":"ad50c588_d8ef068c","line":590,"range":{"start_line":590,"start_character":40,"end_line":590,"end_character":46},"updated":"2021-06-14 19:04:09.000000000","message":"here you are refering to a warm reboot vs a cold reboot.\n\nwarm reboots  are initacited by the host os and typically do not result in the disruption of power to the pci slot although that\ndepened on the uefi/kernel behavior. could reboot such as whne you do a reset via ipmi disconnect the power and reconencts it\n\ncold reboot involve fully powering fo the server and renergising it which would reload firmware exctra form flash on all cards.\n\nso in a warm reboot the DPUs proably will continue to be online during the host os reboot process but for a cold reboot\nthen they would be powered off with the server and the other pci devices.","commit_id":"d2a19227767b9dd5f40bb939921b5263898560e7"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"21f4a7e9a1a10cd54f6cf5359d3be5ad1824be5e","unresolved":true,"context_lines":[{"line_number":587,"context_line":"primary power from a PCIe slot so their power lifecycle is tied to the"},{"line_number":588,"context_line":"main board lifecycle. This should be taken into consideration when performing"},{"line_number":589,"context_line":"power off/power on operations on the hypervisor hosts as it will affect"},{"line_number":590,"context_line":"services running on the SmartNIC DPU (a reboot of the hypervisor host should"},{"line_number":591,"context_line":"not)."},{"line_number":592,"context_line":""},{"line_number":593,"context_line":"Developer impact"}],"source_content_type":"text/x-rst","patch_set":9,"id":"2c5e651b_771f6a09","line":590,"range":{"start_line":590,"start_character":40,"end_line":590,"end_character":46},"in_reply_to":"ad50c588_d8ef068c","updated":"2021-06-24 18:08:24.000000000","message":"Yes, there may be platform-dependent behaviors for sure.\n\nI can change this to: \"in contrast, a warm reboot is likely to keep a SmartNIC DPU powered on\".","commit_id":"d2a19227767b9dd5f40bb939921b5263898560e7"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"87cdb48ebff279ca4fdc74da75312bd4188cd903","unresolved":true,"context_lines":[{"line_number":597,"context_line":"need to gain similar functionality to discover card serial numbers if they"},{"line_number":598,"context_line":"want to support the same workflow."},{"line_number":599,"context_line":""},{"line_number":600,"context_line":"Upgrade impact"},{"line_number":601,"context_line":"--------------"},{"line_number":602,"context_line":""},{"line_number":603,"context_line":"N/A"}],"source_content_type":"text/x-rst","patch_set":9,"id":"d398af38_4d3191ca","line":600,"range":{"start_line":600,"start_character":0,"end_line":600,"end_character":14},"updated":"2021-06-14 19:04:09.000000000","message":"so i think this is proably the part of the spec that we might need to consider more closely.\ni agree that in general i dont think this should have a large upgrade impact.\n\nany host that does not use this feature should have 0 impact and i think that will more or less be the case with this proposal.\n\nthe only aspect might be an issue is the initall agent start after upgrade may need to upgrade teh pci_device tables entries to sotre the extra info for the serial number ectra\n\ni think we should call this out here so that larger clouds are aware of this as that coudl cause issues if you rebooted them all at once.\nother then that however i the current form of the spec avoid all the upgrade issues that the original one had as there are no db or nova api changes involved.\nthere is also a dependency on neutron being upgraded too to use the feature but","commit_id":"d2a19227767b9dd5f40bb939921b5263898560e7"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"21f4a7e9a1a10cd54f6cf5359d3be5ad1824be5e","unresolved":true,"context_lines":[{"line_number":597,"context_line":"need to gain similar functionality to discover card serial numbers if they"},{"line_number":598,"context_line":"want to support the same workflow."},{"line_number":599,"context_line":""},{"line_number":600,"context_line":"Upgrade impact"},{"line_number":601,"context_line":"--------------"},{"line_number":602,"context_line":""},{"line_number":603,"context_line":"N/A"}],"source_content_type":"text/x-rst","patch_set":9,"id":"ebe405a4_a3167e42","line":600,"range":{"start_line":600,"start_character":0,"end_line":600,"end_character":14},"in_reply_to":"d398af38_4d3191ca","updated":"2021-06-24 18:08:24.000000000","message":"Agreed.\n\nAs for Neutron, this is more of an operational concern in terms of service upgrade ordering. However, it would be good to think through the situation with usage of Neutron API (and this feature) happening before a Neutron upgrade. Since Neutron is assumed to have an older version, it cannot react to the newly added data in the request. It also does not have microversions that could have been used in the case of Nova. So we need something else.","commit_id":"d2a19227767b9dd5f40bb939921b5263898560e7"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"87cdb48ebff279ca4fdc74da75312bd4188cd903","unresolved":true,"context_lines":[{"line_number":645,"context_line":"In order to make this useful overall there are additional cross-project"},{"line_number":646,"context_line":"changes required. Specifically, to make this work with OVN:"},{"line_number":647,"context_line":""},{"line_number":648,"context_line":"* ovn-controller needs to learn how to plug representors into correct bridges"},{"line_number":649,"context_line":"  at the SmartNIC DPU node side since the os-vif-like functionality to hook VFs"},{"line_number":650,"context_line":"  up is still needed;"},{"line_number":651,"context_line":""},{"line_number":652,"context_line":"  * Changes to support devlink (netlink-devlink) are tracked here: [13]_"},{"line_number":653,"context_line":"    (published on the OVS mailing list: [14]_);"}],"source_content_type":"text/x-rst","patch_set":9,"id":"ec25213a_3053aacb","line":650,"range":{"start_line":648,"start_character":0,"end_line":650,"end_character":21},"updated":"2021-06-14 19:04:09.000000000","message":"has this been implemented yet in ovn?","commit_id":"d2a19227767b9dd5f40bb939921b5263898560e7"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"21f4a7e9a1a10cd54f6cf5359d3be5ad1824be5e","unresolved":true,"context_lines":[{"line_number":645,"context_line":"In order to make this useful overall there are additional cross-project"},{"line_number":646,"context_line":"changes required. Specifically, to make this work with OVN:"},{"line_number":647,"context_line":""},{"line_number":648,"context_line":"* ovn-controller needs to learn how to plug representors into correct bridges"},{"line_number":649,"context_line":"  at the SmartNIC DPU node side since the os-vif-like functionality to hook VFs"},{"line_number":650,"context_line":"  up is still needed;"},{"line_number":651,"context_line":""},{"line_number":652,"context_line":"  * Changes to support devlink (netlink-devlink) are tracked here: [13]_"},{"line_number":653,"context_line":"    (published on the OVS mailing list: [14]_);"}],"source_content_type":"text/x-rst","patch_set":9,"id":"cc1926da_2782f603","line":650,"range":{"start_line":648,"start_character":0,"end_line":650,"end_character":21},"in_reply_to":"ec25213a_3053aacb","updated":"2021-06-24 18:08:24.000000000","message":"We are trying to get it fully agreed on upstream: there is a certain challenge in orchestrating the inclusion of various bits across 3 projects (one of them being outside OpenStack).\n\nhttps://patchwork.ozlabs.org/project/ovn/patch/20210509140305.1910796-1-frode.nordahl@canonical.com/#2699171","commit_id":"d2a19227767b9dd5f40bb939921b5263898560e7"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"87cdb48ebff279ca4fdc74da75312bd4188cd903","unresolved":true,"context_lines":[{"line_number":651,"context_line":""},{"line_number":652,"context_line":"  * Changes to support devlink (netlink-devlink) are tracked here: [13]_"},{"line_number":653,"context_line":"    (published on the OVS mailing list: [14]_);"},{"line_number":654,"context_line":"  * Representor plugging changes: [15]_ [16]_;"},{"line_number":655,"context_line":"* The OVN driver code in Neutron needs to learn about SmartNIC DPU node"},{"line_number":656,"context_line":"  hostnames and respective PCIe add-in-card serial numbers gathered via devlink"},{"line_number":657,"context_line":"  or VPD:"}],"source_content_type":"text/x-rst","patch_set":9,"id":"aa1d1b6c_2101ab0c","line":654,"range":{"start_line":654,"start_character":34,"end_line":654,"end_character":46},"updated":"2021-06-14 19:04:09.000000000","message":"oh oke this is the code for the ovn plugging.\n\ni think we would need this to be merged upstream in ovn and in a release before we can merge the nova code.\n\nand the same woudl be ture of the ovs netlink-devlnk support.\n\nis this likely to be inculded in an upstream ovs/ovn release before we reach milestone 3?\n\nwe do not speculativly merge code in nova to support a feature in an external project like ovs or\novn before its released or at least merged in tehre master branch so this is a hard depency for the nova code to\nmerge but not nessisarly for the spec to be approved.","commit_id":"d2a19227767b9dd5f40bb939921b5263898560e7"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"21f4a7e9a1a10cd54f6cf5359d3be5ad1824be5e","unresolved":true,"context_lines":[{"line_number":651,"context_line":""},{"line_number":652,"context_line":"  * Changes to support devlink (netlink-devlink) are tracked here: [13]_"},{"line_number":653,"context_line":"    (published on the OVS mailing list: [14]_);"},{"line_number":654,"context_line":"  * Representor plugging changes: [15]_ [16]_;"},{"line_number":655,"context_line":"* The OVN driver code in Neutron needs to learn about SmartNIC DPU node"},{"line_number":656,"context_line":"  hostnames and respective PCIe add-in-card serial numbers gathered via devlink"},{"line_number":657,"context_line":"  or VPD:"}],"source_content_type":"text/x-rst","patch_set":9,"id":"d5faaaa9_c370f6e0","line":654,"range":{"start_line":654,"start_character":34,"end_line":654,"end_character":46},"in_reply_to":"aa1d1b6c_2101ab0c","updated":"2021-06-24 18:08:24.000000000","message":"\u003e i think we would need this to be merged upstream in ovn and in a release before we can merge the nova code.\n\u003e and the same woudl be ture of the ovs netlink-devlnk support.\n\nYes, I agree - hopefully we can do it as a lot of spec work in Nova and Neutron depends on it.\n\nThere is an alternative approach similar to having it directly in OVN and having a separate daemon running on a DPU to be responsible for representor plugging. It makes things more difficult in several ways:\n\n1) Separate component to maintain outside OVN;\n2) Communication patterns become more complex;\n3) Changes at the Neutron side to talk to a separate daemon (whether to do it in the OVN mech driver or having 2 mech drivers is another question);\n4) Neutron has been moving away from having separate agents to just using OVN NB API and it feels like a step in the opposite direction.\n\n\u003e is this likely to be inculded in an upstream ovs/ovn release before we reach milestone 3?\n\nAt the moment I don\u0027t have a good answer - Frode and I need some more time to get a concrete +2/-2 upstream for including this in OVN directly. Otherwise we will aim for having a separate daemon and adjust the Neutron spec accordingly.\n\nI think regardless of which direction wins the Nova spec will stay the same.\n\nThe next step for us is to have a discussion during the Neutron Drivers Meeting and to try and get more OVN core developer opinions on Frode\u0027s patchset.","commit_id":"d2a19227767b9dd5f40bb939921b5263898560e7"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"87cdb48ebff279ca4fdc74da75312bd4188cd903","unresolved":true,"context_lines":[{"line_number":659,"context_line":"  * Port binding needs to be aware of the hypervisor and SmartNIC DPU"},{"line_number":660,"context_line":"    hostname mismatches and mappings between card serial numbers and SmartNIC"},{"line_number":661,"context_line":"    DPU node hostnames (the relevant Neutron specification is published at"},{"line_number":662,"context_line":"    [17]_)."},{"line_number":663,"context_line":"* It would be good to add the support for serial number retrieval to Libvirt"},{"line_number":664,"context_line":"  in order to avoid having this code in Nova in the long run [18]_."},{"line_number":665,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"a3e9c39e_29d4f170","line":662,"updated":"2021-06-14 19:04:09.000000000","message":"ack. i think we would also want the neutron spec to be approved or close to approval before we approve the nova spec.\n\neffectively we should aim to merge both at the same time once both communities are happy with the design.","commit_id":"d2a19227767b9dd5f40bb939921b5263898560e7"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"21f4a7e9a1a10cd54f6cf5359d3be5ad1824be5e","unresolved":false,"context_lines":[{"line_number":659,"context_line":"  * Port binding needs to be aware of the hypervisor and SmartNIC DPU"},{"line_number":660,"context_line":"    hostname mismatches and mappings between card serial numbers and SmartNIC"},{"line_number":661,"context_line":"    DPU node hostnames (the relevant Neutron specification is published at"},{"line_number":662,"context_line":"    [17]_)."},{"line_number":663,"context_line":"* It would be good to add the support for serial number retrieval to Libvirt"},{"line_number":664,"context_line":"  in order to avoid having this code in Nova in the long run [18]_."},{"line_number":665,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"6f2e8091_80ffdcb2","line":662,"in_reply_to":"a3e9c39e_29d4f170","updated":"2021-06-24 18:08:24.000000000","message":"Ack","commit_id":"d2a19227767b9dd5f40bb939921b5263898560e7"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"87cdb48ebff279ca4fdc74da75312bd4188cd903","unresolved":true,"context_lines":[{"line_number":661,"context_line":"    DPU node hostnames (the relevant Neutron specification is published at"},{"line_number":662,"context_line":"    [17]_)."},{"line_number":663,"context_line":"* It would be good to add the support for serial number retrieval to Libvirt"},{"line_number":664,"context_line":"  in order to avoid having this code in Nova in the long run [18]_."},{"line_number":665,"context_line":""},{"line_number":666,"context_line":"Testing"},{"line_number":667,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":9,"id":"959230ad_47c3b236","line":664,"updated":"2021-06-14 19:04:09.000000000","message":"yep. i dont think this need to be a hard depency like the ovn support but in the long run it would be better to get this form libvirt\n+1","commit_id":"d2a19227767b9dd5f40bb939921b5263898560e7"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"21f4a7e9a1a10cd54f6cf5359d3be5ad1824be5e","unresolved":false,"context_lines":[{"line_number":661,"context_line":"    DPU node hostnames (the relevant Neutron specification is published at"},{"line_number":662,"context_line":"    [17]_)."},{"line_number":663,"context_line":"* It would be good to add the support for serial number retrieval to Libvirt"},{"line_number":664,"context_line":"  in order to avoid having this code in Nova in the long run [18]_."},{"line_number":665,"context_line":""},{"line_number":666,"context_line":"Testing"},{"line_number":667,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":9,"id":"c2143f86_d03eb88c","line":664,"in_reply_to":"959230ad_47c3b236","updated":"2021-06-24 18:08:24.000000000","message":"Ack","commit_id":"d2a19227767b9dd5f40bb939921b5263898560e7"}],"specs/xena/approved/remote-networking-agents.rst":[{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"eba8738e4b779a4ebd7133f7e3296d31e02e0ff1","unresolved":true,"context_lines":[{"line_number":8,"context_line":"Integration with Remote Networking Agents"},{"line_number":9,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/nova/+spec/integration-with-remote-networking-agents"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"Off-path SmartNIC DPUs introduce an architecture change where"},{"line_number":14,"context_line":"network agents responsible for NIC switch configuration and representor"}],"source_content_type":"text/x-rst","patch_set":5,"id":"544cd796_e9785d96","line":11,"range":{"start_line":11,"start_character":44,"end_line":11,"end_character":85},"updated":"2021-05-25 13:40:47.000000000","message":"by the way i know i suggested this orginally but antoher alternitive based on the definiton you now have below would be  \"integration-with-off-path-network-backends.\n\nbut the current version works for me alhtough you should udate the file name\n\nit should be integration-with-remote-networking-agents.rst\nwe have some scripts that rely on that to move implemeted specs to the correct folder at the end fo the cycle.\n\nwe can fix it later but if you respin this can you ensure the file name and blueprint url match.","commit_id":"f4333c8817c847fdd318c4fc9b4a9d683b5e006c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"86e5d0c46115ae33479600e063a1617b8645de36","unresolved":false,"context_lines":[{"line_number":8,"context_line":"Integration with Remote Networking Agents"},{"line_number":9,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/nova/+spec/integration-with-remote-networking-agents"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"Off-path SmartNIC DPUs introduce an architecture change where"},{"line_number":14,"context_line":"network agents responsible for NIC switch configuration and representor"}],"source_content_type":"text/x-rst","patch_set":5,"id":"d6ae09bb_b50d4307","line":11,"range":{"start_line":11,"start_character":44,"end_line":11,"end_character":85},"in_reply_to":"544cd796_e9785d96","updated":"2021-05-27 09:43:20.000000000","message":"Ack, will change the title to \"Integration With Off-path Network Backends\" and the file name and the blueprint name to \"integration-with-off-path-network-backends\".","commit_id":"f4333c8817c847fdd318c4fc9b4a9d683b5e006c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"eba8738e4b779a4ebd7133f7e3296d31e02e0ff1","unresolved":true,"context_lines":[{"line_number":38,"context_line":"  cores are responsible for programming a NIC Switch and are bypassed when"},{"line_number":39,"context_line":"  rules programmed into the NIC Switch are enough to make a decision on where"},{"line_number":40,"context_line":"  to deliver packets. Normally, NIC cores only participate in packet forwarding"},{"line_number":41,"context_line":"  for the \"slow path\" only and the \"fast path\" is handled hardware like an"},{"line_number":42,"context_line":"  ASIC;"},{"line_number":43,"context_line":"* On-path SmartNIC DPU architecture [1]_ [2]_ - an architecture where NIC cores"},{"line_number":44,"context_line":"  participate in processing of every packet going through the NIC as a whole."}],"source_content_type":"text/x-rst","patch_set":5,"id":"91708b30_46237638","line":41,"range":{"start_line":41,"start_character":57,"end_line":41,"end_character":58},"updated":"2021-05-25 13:40:47.000000000","message":"nit: in","commit_id":"f4333c8817c847fdd318c4fc9b4a9d683b5e006c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"86e5d0c46115ae33479600e063a1617b8645de36","unresolved":false,"context_lines":[{"line_number":38,"context_line":"  cores are responsible for programming a NIC Switch and are bypassed when"},{"line_number":39,"context_line":"  rules programmed into the NIC Switch are enough to make a decision on where"},{"line_number":40,"context_line":"  to deliver packets. Normally, NIC cores only participate in packet forwarding"},{"line_number":41,"context_line":"  for the \"slow path\" only and the \"fast path\" is handled hardware like an"},{"line_number":42,"context_line":"  ASIC;"},{"line_number":43,"context_line":"* On-path SmartNIC DPU architecture [1]_ [2]_ - an architecture where NIC cores"},{"line_number":44,"context_line":"  participate in processing of every packet going through the NIC as a whole."}],"source_content_type":"text/x-rst","patch_set":5,"id":"b8e8e510_ca925c57","line":41,"range":{"start_line":41,"start_character":57,"end_line":41,"end_character":58},"in_reply_to":"91708b30_46237638","updated":"2021-05-27 09:43:20.000000000","message":"Ack, ty.","commit_id":"f4333c8817c847fdd318c4fc9b4a9d683b5e006c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"eba8738e4b779a4ebd7133f7e3296d31e02e0ff1","unresolved":true,"context_lines":[{"line_number":40,"context_line":"  to deliver packets. Normally, NIC cores only participate in packet forwarding"},{"line_number":41,"context_line":"  for the \"slow path\" only and the \"fast path\" is handled hardware like an"},{"line_number":42,"context_line":"  ASIC;"},{"line_number":43,"context_line":"* On-path SmartNIC DPU architecture [1]_ [2]_ - an architecture where NIC cores"},{"line_number":44,"context_line":"  participate in processing of every packet going through the NIC as a whole."},{"line_number":45,"context_line":"  In other words, NIC cores are always on the \"fast path\" of all packets;"},{"line_number":46,"context_line":"* NIC Switch (or eSwitch) - a programmable embedded switch present in various"}],"source_content_type":"text/x-rst","patch_set":5,"id":"6c2f75dc_5f6f8eba","line":43,"range":{"start_line":43,"start_character":2,"end_line":43,"end_character":9},"updated":"2021-05-25 13:40:47.000000000","message":"ack, we can work with these definitons for this spec","commit_id":"f4333c8817c847fdd318c4fc9b4a9d683b5e006c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"86e5d0c46115ae33479600e063a1617b8645de36","unresolved":false,"context_lines":[{"line_number":40,"context_line":"  to deliver packets. Normally, NIC cores only participate in packet forwarding"},{"line_number":41,"context_line":"  for the \"slow path\" only and the \"fast path\" is handled hardware like an"},{"line_number":42,"context_line":"  ASIC;"},{"line_number":43,"context_line":"* On-path SmartNIC DPU architecture [1]_ [2]_ - an architecture where NIC cores"},{"line_number":44,"context_line":"  participate in processing of every packet going through the NIC as a whole."},{"line_number":45,"context_line":"  In other words, NIC cores are always on the \"fast path\" of all packets;"},{"line_number":46,"context_line":"* NIC Switch (or eSwitch) - a programmable embedded switch present in various"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fb40102f_fd6598ac","line":43,"range":{"start_line":43,"start_character":2,"end_line":43,"end_character":9},"in_reply_to":"6c2f75dc_5f6f8eba","updated":"2021-05-27 09:43:20.000000000","message":"Ack","commit_id":"f4333c8817c847fdd318c4fc9b4a9d683b5e006c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"eba8738e4b779a4ebd7133f7e3296d31e02e0ff1","unresolved":true,"context_lines":[{"line_number":137,"context_line":"destination over the fast path bypassing the \"Application CPU\". Otherwise, the"},{"line_number":138,"context_line":"packet is processed in software at the Application CPU and then forwarded to"},{"line_number":139,"context_line":"the destination."},{"line_number":140,"context_line":""},{"line_number":141,"context_line":"There are more sophisticated scenarios as well:"},{"line_number":142,"context_line":""},{"line_number":143,"context_line":"* Two or more SmartNICs/DPUs per server attached to different NUMA nodes;"}],"source_content_type":"text/x-rst","patch_set":5,"id":"bb17e6c2_9ec2a5e8","line":140,"updated":"2021-05-25 13:40:47.000000000","message":"+1 this is a good highlevel overview of the packet flow without adding too much lowlevel detail.","commit_id":"f4333c8817c847fdd318c4fc9b4a9d683b5e006c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"86e5d0c46115ae33479600e063a1617b8645de36","unresolved":false,"context_lines":[{"line_number":137,"context_line":"destination over the fast path bypassing the \"Application CPU\". Otherwise, the"},{"line_number":138,"context_line":"packet is processed in software at the Application CPU and then forwarded to"},{"line_number":139,"context_line":"the destination."},{"line_number":140,"context_line":""},{"line_number":141,"context_line":"There are more sophisticated scenarios as well:"},{"line_number":142,"context_line":""},{"line_number":143,"context_line":"* Two or more SmartNICs/DPUs per server attached to different NUMA nodes;"}],"source_content_type":"text/x-rst","patch_set":5,"id":"3e73ff8d_9ee1c278","line":140,"in_reply_to":"bb17e6c2_9ec2a5e8","updated":"2021-05-27 09:43:20.000000000","message":"Ack","commit_id":"f4333c8817c847fdd318c4fc9b4a9d683b5e006c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"eba8738e4b779a4ebd7133f7e3296d31e02e0ff1","unresolved":true,"context_lines":[{"line_number":155,"context_line":"(with the help of os-vif) can no longer be responsible for VIF plugging in the"},{"line_number":156,"context_line":"same way it is in the hardware offload scenario: OVS bridges and port"},{"line_number":157,"context_line":"representors are no longer exposed to the OS on the main board."},{"line_number":158,"context_line":""},{"line_number":159,"context_line":"Since Nova and networking agents run on different hosts, there needs to be a"},{"line_number":160,"context_line":"different set of interactions in order to:"},{"line_number":161,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"3d1f4377_a2af72b3","line":158,"updated":"2021-05-25 13:40:47.000000000","message":"ack, this is a detail for the neutron implmenation by the way but we should also not\nthat in this congifuation, assuming no other agent is running on the hypervior host os direclty\nthat we will not be supporting vnic_type normal on this host.\n\nneutron can correctly candel that at the port binding stage by rejecting to bind the port but we likely want to consider schduling based on this as well.\n\nefectivly what im thinking is that we woudl modify neutron so that it alwasy create a port resouce request. that request would by default not request an placemnet resouce classes adn only contain traits request for the vnic_type, network type (vxlan, geneve...) and optionaly the physnet if applicable.\n\nwe could emulate this on the nova side with a placement prefilter in the sort term but ultimately i would like to see nova expressing these types of requireemtns.\n\nif this was in place vms with  ports of vnic_type normal could never by schduled to host that only had off path acclerators and you would not have to do anything special form a sculding point of view.\n\nit would also be genericaly useful as we would know that a host could potentially bind the port to the correct network type and physnet before we actually select teh host in scudling and try to do bind the port on the compute node.\n\nthis is likely a sperate enhanment form this feature but i think we shoudl keep this in mind as somethign to aim for.","commit_id":"f4333c8817c847fdd318c4fc9b4a9d683b5e006c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"86e5d0c46115ae33479600e063a1617b8645de36","unresolved":false,"context_lines":[{"line_number":155,"context_line":"(with the help of os-vif) can no longer be responsible for VIF plugging in the"},{"line_number":156,"context_line":"same way it is in the hardware offload scenario: OVS bridges and port"},{"line_number":157,"context_line":"representors are no longer exposed to the OS on the main board."},{"line_number":158,"context_line":""},{"line_number":159,"context_line":"Since Nova and networking agents run on different hosts, there needs to be a"},{"line_number":160,"context_line":"different set of interactions in order to:"},{"line_number":161,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"c0d553d7_d3bfc75b","line":158,"in_reply_to":"3d1f4377_a2af72b3","updated":"2021-05-27 09:43:20.000000000","message":"\u003e ack, this is a detail for the neutron implmenation by the way but we should also not\nthat in this congifuation, assuming no other agent is running on the hypervior host os direclty that we will not be supporting vnic_type normal on this host.\n\nAgreed, I will put references to specific agents in parentheses just for information. In fact, it may be something totally different from OVS and OVN at the SmartNIC DPU side - Nova doesn\u0027t even need to know what it is. I am just trying to avoid being too abstract here as somebody might be reading this out of context but it\u0027s certainly related to Neutron more than to Nova.\n\nWill also make a note that no networking agents are expected to run on the hypervisor host in this configuration as you say - this is a pretty important distinction to note.\n\n\u003e if this was in place vms with  ports of vnic_type normal could never by schduled to host that only had off path acclerators and you would not have to do anything special form a sculding point of view.\n\u003e it would also be genericaly useful as we would know that a host could potentially bind the port to the correct network type and physnet before we actually select teh host in scudling and try to do bind the port on the compute node.\n\nRight, part of the reason why I was trying to account for resources in the initial spec was that suddenly ports are no longer an \"infinite\" resource that can just be created - there are limits to the amount of VFs that can be allocated and \"normal\" ports aren\u0027t available at all.\n\nI think what you suggest covers it well.","commit_id":"f4333c8817c847fdd318c4fc9b4a9d683b5e006c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"eba8738e4b779a4ebd7133f7e3296d31e02e0ff1","unresolved":true,"context_lines":[{"line_number":182,"context_line":"    switchdev-capable NIC: via PCIe, a platform device or other means of I/O."},{"line_number":183,"context_line":"    The hypervisor host would see PCIe endpoints regardless of that but relying"},{"line_number":184,"context_line":"    on PCI addresses in the implementation to match functions and their"},{"line_number":185,"context_line":"    representors is not feasible."},{"line_number":186,"context_line":""},{"line_number":187,"context_line":"In order to track SmartNIC DPUs and associations of PFs/VFs with them, there"},{"line_number":188,"context_line":"is a need for a unique and persistent identifier that is discoverable from both"}],"source_content_type":"text/x-rst","patch_set":5,"id":"42618c17_7a1440f2","line":185,"updated":"2021-05-25 13:40:47.000000000","message":"+1","commit_id":"f4333c8817c847fdd318c4fc9b4a9d683b5e006c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"86e5d0c46115ae33479600e063a1617b8645de36","unresolved":false,"context_lines":[{"line_number":182,"context_line":"    switchdev-capable NIC: via PCIe, a platform device or other means of I/O."},{"line_number":183,"context_line":"    The hypervisor host would see PCIe endpoints regardless of that but relying"},{"line_number":184,"context_line":"    on PCI addresses in the implementation to match functions and their"},{"line_number":185,"context_line":"    representors is not feasible."},{"line_number":186,"context_line":""},{"line_number":187,"context_line":"In order to track SmartNIC DPUs and associations of PFs/VFs with them, there"},{"line_number":188,"context_line":"is a need for a unique and persistent identifier that is discoverable from both"}],"source_content_type":"text/x-rst","patch_set":5,"id":"8a4a4eac_3e857a42","line":185,"in_reply_to":"42618c17_7a1440f2","updated":"2021-05-27 09:43:20.000000000","message":"Ack","commit_id":"f4333c8817c847fdd318c4fc9b4a9d683b5e006c"},{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"155b000199ec3da7d42238b487f4b25fcc8d8c75","unresolved":true,"context_lines":[{"line_number":421,"context_line":"before a port update since those are done by the Nova Compute manager during"},{"line_number":422,"context_line":"instance creation. Alternatively, it can be stored in the database in"},{"line_number":423,"context_line":"``extra_info`` of a PciDevice."},{"line_number":424,"context_line":""},{"line_number":425,"context_line":""},{"line_number":426,"context_line":"Alternatives"},{"line_number":427,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":5,"id":"4e460dc0_00778582","line":424,"updated":"2021-05-25 12:05:15.000000000","message":"Do I understand correctly that the existing PCI tracking in nova-compute and the existing PCIFilter in the nova-scheduler are assumed to be not impacted by the proposed change? \n\n@Sean: Do this spec has any impact on your pci device tracking in placment spec https://review.opendev.org/c/openstack/nova-specs/+/791047 ?","commit_id":"f4333c8817c847fdd318c4fc9b4a9d683b5e006c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"86e5d0c46115ae33479600e063a1617b8645de36","unresolved":false,"context_lines":[{"line_number":421,"context_line":"before a port update since those are done by the Nova Compute manager during"},{"line_number":422,"context_line":"instance creation. Alternatively, it can be stored in the database in"},{"line_number":423,"context_line":"``extra_info`` of a PciDevice."},{"line_number":424,"context_line":""},{"line_number":425,"context_line":""},{"line_number":426,"context_line":"Alternatives"},{"line_number":427,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":5,"id":"0c3edb8f_8a201343","line":424,"in_reply_to":"288df934_bed5b3fa","updated":"2021-05-27 09:43:20.000000000","message":"\u003e Do I understand correctly that the existing PCI tracking in nova-compute and the existing PCIFilter in the nova-scheduler are assumed to be not impacted by the proposed change? \n\nNothing except for storing some more information associated with PciDevices (the serial number from VPD, PF MAC, VF logical number).\n\nI wish we had PCIe hierarchy IDs available for PFs and VFs in the contemporary hardware to avoid having to rely on PF MAC and VF logical numbers for VF representor lookup but I guess we have to work with what we have today and later use better means if they become available.\n\n\u003e so i will adapt the geneirc pci spec to supprot this spec assuming this gets approved.\n\nThank you for looking into this and all the work so far - this is much appreciated.","commit_id":"f4333c8817c847fdd318c4fc9b4a9d683b5e006c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"181fbcc05524c2a1c5f10fec0c0b1cf6dd7495e3","unresolved":true,"context_lines":[{"line_number":421,"context_line":"before a port update since those are done by the Nova Compute manager during"},{"line_number":422,"context_line":"instance creation. Alternatively, it can be stored in the database in"},{"line_number":423,"context_line":"``extra_info`` of a PciDevice."},{"line_number":424,"context_line":""},{"line_number":425,"context_line":""},{"line_number":426,"context_line":"Alternatives"},{"line_number":427,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":5,"id":"288df934_bed5b3fa","line":424,"in_reply_to":"4e460dc0_00778582","updated":"2021-05-25 13:48:28.000000000","message":"they are complementary but i think this can proceed without that spec being completed so one does not block the other.\n\ni am going to extend that spec to cater for some of the use cases that this spec raises. namely im going to add the ablity to add traits via the pci whitelist instead fo just resouce classes.\n\nin the current version of my spec im suggesting addign the ablity to request traits via the pci alais and i was going to have the capablies transformed into traits and reported automatically but i think haveign the abllity to add CUSTOM_ traits via the pci whitelist will allow that tagging of the device for use with this spec in a palacment native way if we want that.\n\nso i will adapt the geneirc pci spec to supprot this spec assuming this gets approved.","commit_id":"f4333c8817c847fdd318c4fc9b4a9d683b5e006c"},{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"155b000199ec3da7d42238b487f4b25fcc8d8c75","unresolved":true,"context_lines":[{"line_number":658,"context_line":"    hostname mismatches and mappings between card serial numbers and SmartNIC"},{"line_number":659,"context_line":"    DPU node hostnames (the relevant Neutron specification is published at"},{"line_number":660,"context_line":"    [16]_)."},{"line_number":661,"context_line":""},{"line_number":662,"context_line":"Testing"},{"line_number":663,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":664,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"82926458_5f9e4809","line":661,"updated":"2021-05-25 12:05:15.000000000","message":"Do you expect to have all pieces ready for the Xena release in October including the upstream OVS changes?","commit_id":"f4333c8817c847fdd318c4fc9b4a9d683b5e006c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"86e5d0c46115ae33479600e063a1617b8645de36","unresolved":false,"context_lines":[{"line_number":658,"context_line":"    hostname mismatches and mappings between card serial numbers and SmartNIC"},{"line_number":659,"context_line":"    DPU node hostnames (the relevant Neutron specification is published at"},{"line_number":660,"context_line":"    [16]_)."},{"line_number":661,"context_line":""},{"line_number":662,"context_line":"Testing"},{"line_number":663,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":664,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"f133cf32_dc279963","line":661,"in_reply_to":"82926458_5f9e4809","updated":"2021-05-27 09:43:20.000000000","message":"Yes, that is the goal. Frode and I are trying to get all pieces in place as fast as possible while avoiding snowflakes - we do not see much point in having anything out of tree or unapproved by the relevant upstream communities. This introduces some challenges into being able to reason about exact timelines as the whole effort is cross-project and timing is not always under our control. But I can say that it is our priority to get it done fast (and right).","commit_id":"f4333c8817c847fdd318c4fc9b4a9d683b5e006c"},{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"155b000199ec3da7d42238b487f4b25fcc8d8c75","unresolved":true,"context_lines":[{"line_number":663,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":664,"context_line":""},{"line_number":665,"context_line":"* Unit testing of the added functionality;"},{"line_number":666,"context_line":"* Functional testing on real hardware if possible;"},{"line_number":667,"context_line":""},{"line_number":668,"context_line":"  * Devices with an application CPU and switchdev capabilities are required"},{"line_number":669,"context_line":"    to test that;"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ce78d1c1_9ee72fd8","line":666,"updated":"2021-05-25 12:05:15.000000000","message":"I guess we don\u0027t have real hardware in upstream CI for this. So we cannot expect to have tempest coverage for the feature. Similarly how we don\u0027t have SRIOV coverage for the same reason.\n\nI suggest, top of the unit test, to look at the tests under nova.tests.functional.libvirt I hope that env can be used to cover the proposed changes.","commit_id":"f4333c8817c847fdd318c4fc9b4a9d683b5e006c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"181fbcc05524c2a1c5f10fec0c0b1cf6dd7495e3","unresolved":true,"context_lines":[{"line_number":663,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":664,"context_line":""},{"line_number":665,"context_line":"* Unit testing of the added functionality;"},{"line_number":666,"context_line":"* Functional testing on real hardware if possible;"},{"line_number":667,"context_line":""},{"line_number":668,"context_line":"  * Devices with an application CPU and switchdev capabilities are required"},{"line_number":669,"context_line":"    to test that;"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fb52a94d_cd9643f1","line":666,"in_reply_to":"ce78d1c1_9ee72fd8","updated":"2021-05-25 13:48:28.000000000","message":"perhaps the melonox sriov ci could be extended if that has the correct hardeware.\n\ni do agree that that functional test shoudl be possible and we shoudl emulate this there. that will require some enhancement to the neutron fixture and i woudl guess it would be cleaner to create a spereate neutron fixture for these tests to emulate the beahvior we expect but i think that an implemation detail that we can work out later.","commit_id":"f4333c8817c847fdd318c4fc9b4a9d683b5e006c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"86e5d0c46115ae33479600e063a1617b8645de36","unresolved":true,"context_lines":[{"line_number":663,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":664,"context_line":""},{"line_number":665,"context_line":"* Unit testing of the added functionality;"},{"line_number":666,"context_line":"* Functional testing on real hardware if possible;"},{"line_number":667,"context_line":""},{"line_number":668,"context_line":"  * Devices with an application CPU and switchdev capabilities are required"},{"line_number":669,"context_line":"    to test that;"}],"source_content_type":"text/x-rst","patch_set":5,"id":"46f9ecd3_c8db014c","line":666,"in_reply_to":"fb52a94d_cd9643f1","updated":"2021-05-27 09:43:20.000000000","message":"I wish there was some way to emulate a machine with multiple isolated CPUs talking over PCIe in QEMU and a switchdev-capable switch - this could probably be used to emulate a system with a SmartNIC DPU.\n\nSome useful parts are there but not enough to build a complete system.\n\nhttps://github.com/qemu/qemu/blob/master/docs/specs/rocker.txt\nhttps://people.netfilter.org/pablo/netdev0.1/papers/Rocker-switchdev-prototyping-vehicle.pdf\n\nSo:\n\n1) unit tests - no question about it;\n2) Neutron fixture - will explore what needs to be done but it seems like a decent approach while we do not have a CI with real hardware figured out;\n3) Real hardware - need to explore this further as we definitely need real hw validation long-term.","commit_id":"f4333c8817c847fdd318c4fc9b4a9d683b5e006c"}],"specs/xena/approved/transport-node-resource-providers.rst":[{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"9d4e7017b76e85479eb7e28dc7185595e986126c","unresolved":true,"context_lines":[{"line_number":62,"context_line":"The rest of the description will focus on illustrating why this process needs"},{"line_number":63,"context_line":"improvements to support off-path SmartNICs/DPUs."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"Off-path SmartNICs provide a dedicated CPU for eSwitch programming on which"},{"line_number":66,"context_line":"a dedicated OS is set to run which is separate from the OS running on the"},{"line_number":67,"context_line":"main board. A system with one SmartNIC in a multi-CPU system with PCIe"},{"line_number":68,"context_line":"bifurcation used for the add-in card is shown below::"}],"source_content_type":"text/x-rst","patch_set":1,"id":"afac5f1d_f63adc37","line":65,"range":{"start_line":65,"start_character":0,"end_line":65,"end_character":75},"updated":"2021-04-22 20:44:10.000000000","message":"During a discussion in IRC there was a request to add more information/references to the description of the hardware in question and related technologies (switchdev, eSwitch, port representors).\n\nhttp://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2021-04-22.log.html#t2021-04-22T11:55:52\n\nGoing to add the following ones at least:\n\n* An overview of SmartNIC from device manufacturer engineers (off-path vs on-path terminology and some details) https://netdevconf.info/0x14/pub/slides/39/Netdev%200x14%20--%20Taking%20Control%20of%20your%20SmartNIC%20v1.pdf\n\n* Kernel docs on the switchdev framework https://www.kernel.org/doc/Documentation/networking/switchdev.txt\n\n* OVS offload doc (Neutron) discussing port representors to some degree https://docs.openstack.org/neutron/latest/admin/config-ovs-offload.html;\n\n* Devlink port flavors https://www.kernel.org/doc/html/latest/networking/devlink/devlink-port.html\n\n* Example PCI controller layout https://www.kernel.org/doc/html/latest/networking/devlink/devlink-port.html#pci-controllers\n\n* A kernel patch for mlx5 discussing E-Switch offload and VF representors: https://lwn.net/Articles/692942/","commit_id":"f5594afc2f42be7a81bccc4269577a92bd5d305b"},{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"bd3a47bff64b4a8d86ecdf0ab810d5a5780c3ab1","unresolved":true,"context_lines":[{"line_number":391,"context_line":"between hypervisor hosts and OVN mechanism driver used at the Neutron side"},{"line_number":392,"context_line":"(CN - Compute Node, TN - Transport Node)::"},{"line_number":393,"context_line":""},{"line_number":394,"context_line":"  CN1                   SHR_ROOT         CN2"},{"line_number":395,"context_line":"  (name\u003dCN1_hostname)   /                (name\u003dCN2_hostname)"},{"line_number":396,"context_line":"     \\  agg1           /          agg1   /"},{"line_number":397,"context_line":"      -------------- TN1 ----------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"cb6f0127_f91fcef7","line":394,"range":{"start_line":394,"start_character":24,"end_line":394,"end_character":32},"updated":"2021-04-22 12:49:28.000000000","message":"I assume this is an RP. What is the reason we have a root RP for TNs?","commit_id":"f5594afc2f42be7a81bccc4269577a92bd5d305b"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"9d4e7017b76e85479eb7e28dc7185595e986126c","unresolved":true,"context_lines":[{"line_number":391,"context_line":"between hypervisor hosts and OVN mechanism driver used at the Neutron side"},{"line_number":392,"context_line":"(CN - Compute Node, TN - Transport Node)::"},{"line_number":393,"context_line":""},{"line_number":394,"context_line":"  CN1                   SHR_ROOT         CN2"},{"line_number":395,"context_line":"  (name\u003dCN1_hostname)   /                (name\u003dCN2_hostname)"},{"line_number":396,"context_line":"     \\  agg1           /          agg1   /"},{"line_number":397,"context_line":"      -------------- TN1 ----------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"b6c847fa_b7f1391c","line":394,"range":{"start_line":394,"start_character":24,"end_line":394,"end_character":32},"in_reply_to":"cb6f0127_f91fcef7","updated":"2021-04-22 20:44:10.000000000","message":"I borrowed SHR_ROOT from here for illustration purposes:\n\nhttps://opendev.org/openstack/nova/blame/commit/3f2868ce3646b7e1a4ae5ca789819cfe9c0e48bc/doc/source/reference/update-provider-tree.rst#L116-L121\n\nThe plan was to have each TN as a root of its own tree.\n\nFor example, it would be similar to shared storage providers:\n\nhttps://opendev.org/openstack/nova/blame/commit/3f2868ce3646b7e1a4ae5ca789819cfe9c0e48bc/nova/tests/functional/test_servers_provider_tree.py#L215-L220\n            # Create a shared storage provider as a root\n            prov_tree.new_root(\u0027ssp\u0027, uuids.ssp)\n            prov_tree.update_traits(\n                \u0027ssp\u0027, [\u0027MISC_SHARES_VIA_AGGREGATE\u0027, \u0027STORAGE_DISK_SSD\u0027])\n            prov_tree.update_aggregates(\u0027ssp\u0027, [uuids.agg])\n            prov_tree.update_inventory(\u0027ssp\u0027, {\u0027DISK_GB\u0027: {\u0027total\u0027: 500}})\n\n\nI am going to adjust the picture to avoid confusion.","commit_id":"f5594afc2f42be7a81bccc4269577a92bd5d305b"},{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"bd3a47bff64b4a8d86ecdf0ab810d5a5780c3ab1","unresolved":true,"context_lines":[{"line_number":398,"context_line":"                    / (name\u003dTN1_VPD_SERIAL)"},{"line_number":399,"context_line":"                   /  (RP trait (new in os-traits): \"HW_NIC_TRANSPORT_NODE\")"},{"line_number":400,"context_line":"                  /   (RP trait: \"MISC_SHARES_VIA_AGGREGATE\")"},{"line_number":401,"context_line":"              OVN Chassis"},{"line_number":402,"context_line":"              (name\u003dTN1_hostname)"},{"line_number":403,"context_line":""},{"line_number":404,"context_line":"In this case, all hosts (CN1, CN2, TN1) can observe TN1_VPD_SERIAL - the"}],"source_content_type":"text/x-rst","patch_set":1,"id":"35edf700_e5733451","line":401,"range":{"start_line":401,"start_character":14,"end_line":401,"end_character":25},"updated":"2021-04-22 12:49:28.000000000","message":"is this a child RP under TN1?","commit_id":"f5594afc2f42be7a81bccc4269577a92bd5d305b"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"ed70401bbe1eb14c7d3b50560e0511c595007e9c","unresolved":true,"context_lines":[{"line_number":398,"context_line":"                    / (name\u003dTN1_VPD_SERIAL)"},{"line_number":399,"context_line":"                   /  (RP trait (new in os-traits): \"HW_NIC_TRANSPORT_NODE\")"},{"line_number":400,"context_line":"                  /   (RP trait: \"MISC_SHARES_VIA_AGGREGATE\")"},{"line_number":401,"context_line":"              OVN Chassis"},{"line_number":402,"context_line":"              (name\u003dTN1_hostname)"},{"line_number":403,"context_line":""},{"line_number":404,"context_line":"In this case, all hosts (CN1, CN2, TN1) can observe TN1_VPD_SERIAL - the"}],"source_content_type":"text/x-rst","patch_set":1,"id":"e58a1df2_f81c6c72","line":401,"range":{"start_line":401,"start_character":14,"end_line":401,"end_character":25},"in_reply_to":"35edf700_e5733451","updated":"2021-04-22 13:42:45.000000000","message":"i think this is a neutron created RP like the ones used for bandwidth based scheduling.\n\nalthough it using aggreate for assocation to allow for the usecase fo multi host nics.","commit_id":"f5594afc2f42be7a81bccc4269577a92bd5d305b"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"9d4e7017b76e85479eb7e28dc7185595e986126c","unresolved":true,"context_lines":[{"line_number":398,"context_line":"                    / (name\u003dTN1_VPD_SERIAL)"},{"line_number":399,"context_line":"                   /  (RP trait (new in os-traits): \"HW_NIC_TRANSPORT_NODE\")"},{"line_number":400,"context_line":"                  /   (RP trait: \"MISC_SHARES_VIA_AGGREGATE\")"},{"line_number":401,"context_line":"              OVN Chassis"},{"line_number":402,"context_line":"              (name\u003dTN1_hostname)"},{"line_number":403,"context_line":""},{"line_number":404,"context_line":"In this case, all hosts (CN1, CN2, TN1) can observe TN1_VPD_SERIAL - the"}],"source_content_type":"text/x-rst","patch_set":1,"id":"e016995e_6b1a5e1a","line":401,"range":{"start_line":401,"start_character":14,"end_line":401,"end_character":25},"in_reply_to":"e58a1df2_f81c6c72","updated":"2021-04-22 20:44:10.000000000","message":"The idea here was:\n\n* To associate CN RPs and TN RPs via an aggregate to avoid a parent-child relationship to address the multi-host scenario;\n* Have an OVN Chassis RP (similar to the bw allocation spec here https://review.opendev.org/c/openstack/neutron/+/776701) under a TN RP since uplink interfaces (possibly bonded) will have limited bandwidth that will need to be shared across PFs/VFs exposed to all hosts. If PFs were tracked in the Placement service as resource providers, this would potentially give us a way to make requests such as: \"give me a VF with this amount of bandwidth available via uplink ports of a SmartNIC\".\n\nInitially the goal of this whole structure was to be able to get a hostname of a SmartNIC for a via a serial number, for example:\n\n1) A VF got allocated (by fulfilling a InstancePCIRequest);\n2) A PCI VPD serial number got obtained from it (now we know which TN RP to look for);\n3) An OVN chassis RP got looked up under the TN RP \u003d\u003e the name of the OVN chassis RP is the SmartNIC hostname \u003d\u003e this can be used as a hostname during port binding (could add a trait to the chassis RP for lookup purposes).\n   * As discussed on IRC, it might be desirable to keep binding_host_id equal to the hostname of the hypervisor (http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2021-04-22.log.html#t2021-04-22T12:00:37) but let\u0027s see if that breaks anything at the Neutron side.\n\nWhy would we need to do such a lookup?\n\n1) If Nova were to set binding_host_id to the hostname of the SmartNIC, it would need to know this information. A networking agent running on the SmartNIC needs to somehow make this information available to it while all relevant components (ovn-controller, Neutron, nova-compute) run in parallel and may come up at different times;\n\n2) Alternatively, If Nova were to just pass binding_host_id\u003d\u003dhypervisor_hostname plus the serial number and VF logical number during the port update to Neutron, the mechanism driver (OVN in this case) would then need too look up a hostname of the desired SmartNIC corresponding to the allocated VF based on the serial number.\n\nWe discussed doing (2) by having those associations stored in Neutron (not in Placement) on IRC http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2021-04-22.log.html#t2021-04-22T12:10:14.\n\nWill consider what this might entail but it would be good to assess whether we need this RP structure for bandwidth allocation as well - might get mappings \"for free\" with this.","commit_id":"f5594afc2f42be7a81bccc4269577a92bd5d305b"},{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"bd3a47bff64b4a8d86ecdf0ab810d5a5780c3ab1","unresolved":true,"context_lines":[{"line_number":569,"context_line":""},{"line_number":570,"context_line":"Work Items"},{"line_number":571,"context_line":"----------"},{"line_number":572,"context_line":""},{"line_number":573,"context_line":"* Implement the code to work with devlink-info for serial number extraction;"},{"line_number":574,"context_line":"* Implement the code to parse the binary format of PCI(e) VPD via sysfs per"},{"line_number":575,"context_line":"  PCI 2.1+ Local bus and PCIe 4.0 specifications (the format is the same);"}],"source_content_type":"text/x-rst","patch_set":1,"id":"7ee2e7c7_cecf3b42","line":572,"updated":"2021-04-22 12:49:28.000000000","message":"* Making sure that nova handles sharing RPs. We have rumors that it works with shared storage + provider.yaml but we have no real test coverage in tree for the sharing case with all the lifecycle operations.","commit_id":"f5594afc2f42be7a81bccc4269577a92bd5d305b"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"9d4e7017b76e85479eb7e28dc7185595e986126c","unresolved":true,"context_lines":[{"line_number":569,"context_line":""},{"line_number":570,"context_line":"Work Items"},{"line_number":571,"context_line":"----------"},{"line_number":572,"context_line":""},{"line_number":573,"context_line":"* Implement the code to work with devlink-info for serial number extraction;"},{"line_number":574,"context_line":"* Implement the code to parse the binary format of PCI(e) VPD via sysfs per"},{"line_number":575,"context_line":"  PCI 2.1+ Local bus and PCIe 4.0 specifications (the format is the same);"}],"source_content_type":"text/x-rst","patch_set":1,"id":"269ff5b9_e9ee7b66","line":572,"in_reply_to":"55b4c1e0_914c8580","updated":"2021-04-22 20:44:10.000000000","message":"Ack, I saw this when it comes to testing sharing RP:\n\nhttps://opendev.org/openstack/nova/blame/commit/3f2868ce3646b7e1a4ae5ca789819cfe9c0e48bc/nova/tests/functional/test_servers_provider_tree.py#L215-L220\n\nBut I am going to pay more attention to related lifecycle operations - will add a line here to have some test cases for this.","commit_id":"f5594afc2f42be7a81bccc4269577a92bd5d305b"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"ed70401bbe1eb14c7d3b50560e0511c595007e9c","unresolved":true,"context_lines":[{"line_number":569,"context_line":""},{"line_number":570,"context_line":"Work Items"},{"line_number":571,"context_line":"----------"},{"line_number":572,"context_line":""},{"line_number":573,"context_line":"* Implement the code to work with devlink-info for serial number extraction;"},{"line_number":574,"context_line":"* Implement the code to parse the binary format of PCI(e) VPD via sysfs per"},{"line_number":575,"context_line":"  PCI 2.1+ Local bus and PCIe 4.0 specifications (the format is the same);"}],"source_content_type":"text/x-rst","patch_set":1,"id":"55b4c1e0_914c8580","line":572,"in_reply_to":"7ee2e7c7_cecf3b42","updated":"2021-04-22 13:42:45.000000000","message":"we dont although there is at least minimal placment fucntional test.\nnot sure how it will interact with move operations and it defnetly not tested well.","commit_id":"f5594afc2f42be7a81bccc4269577a92bd5d305b"},{"author":{"_account_id":12171,"name":"Moshe Levi","email":"moshele@nvidia.com","username":"moshele"},"change_message_id":"9f9d00e45d2b7a27ef9e6607a073270d48e0d25f","unresolved":true,"context_lines":[{"line_number":75,"context_line":"* Selecting the right host for the instance to be scheduled;"},{"line_number":76,"context_line":""},{"line_number":77,"context_line":"  * In the switchdev-capable NIC case: based on availability of devices with"},{"line_number":78,"context_line":"    the \"switchdev\" capability of PciDevices recorded in the Nova DB;"},{"line_number":79,"context_line":"* Building and running the instance, which involves:"},{"line_number":80,"context_line":""},{"line_number":81,"context_line":"  * Claiming PCI resources via the ResourceTracker at the target host based on"}],"source_content_type":"text/x-rst","patch_set":2,"id":"b3bea360_4f2c0d39","line":78,"range":{"start_line":78,"start_character":0,"end_line":78,"end_character":69},"updated":"2021-04-24 21:28:51.000000000","message":"This was the initiation but the unfontinality [1] spec was half implemented. Today neutron mech driver define which are the SR-IOV port (direct or direct with switchdev) it allows to bind. \n\n\n[1] - https://review.opendev.org/c/openstack/nova-specs/+/435954","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"78cd5865e1608849ac48d0068b25dcd7eadce9da","unresolved":true,"context_lines":[{"line_number":75,"context_line":"* Selecting the right host for the instance to be scheduled;"},{"line_number":76,"context_line":""},{"line_number":77,"context_line":"  * In the switchdev-capable NIC case: based on availability of devices with"},{"line_number":78,"context_line":"    the \"switchdev\" capability of PciDevices recorded in the Nova DB;"},{"line_number":79,"context_line":"* Building and running the instance, which involves:"},{"line_number":80,"context_line":""},{"line_number":81,"context_line":"  * Claiming PCI resources via the ResourceTracker at the target host based on"}],"source_content_type":"text/x-rst","patch_set":2,"id":"ee1de3e3_ee59ef2b","line":78,"range":{"start_line":78,"start_character":0,"end_line":78,"end_character":69},"in_reply_to":"b3bea360_4f2c0d39","updated":"2021-04-26 11:27:12.000000000","message":"Whether port binding will succeed for a particular port type depends on the exposed capabilities - yes.\n\nhttps://opendev.org/openstack/neutron/src/tag/16.3.1/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py#L769-L772\nhttps://opendev.org/openstack/neutron/src/tag/16.3.1/neutron/common/ovn/constants.py#L296-L298\n\nWhat I am referring to here is the functionality that allows target nodes to be filtered based on the PCI device requests created early in the instance creation process:\n\nhttps://opendev.org/openstack/nova/src/tag/23.0.0/nova/compute/api.py#L1036-L1038 (InstancePCIRequests request creation)\n\nhttps://opendev.org/openstack/nova/src/tag/21.2.0/nova/scheduler/filters/pci_passthrough_filter.py#L23-L56 (the filter code)\nhttps://opendev.org/openstack/nova/src/tag/23.0.0/nova/pci/stats.py#L505-L525 (the support_requests method)\nhttps://opendev.org/openstack/nova/src/tag/23.0.0/nova/pci/stats.py#L415-L503 (_filter_pools)\nhttps://opendev.org/openstack/nova/src/tag/23.0.0/nova/pci/stats.py#L241-L258 (_filter_pools_for_spec)\nhttps://opendev.org/openstack/nova/src/tag/21.2.0/nova/pci/utils.py#L38-L70 (pci_device_prop_match)\n\n\nIf the \"switchdev\" capability is present in the extra_info column for a particular PciDevice entry, then the scheduler is able to make the right instance placement decision.","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"949ec991d37115d125a08649056f3606564802b7","unresolved":true,"context_lines":[{"line_number":75,"context_line":"* Selecting the right host for the instance to be scheduled;"},{"line_number":76,"context_line":""},{"line_number":77,"context_line":"  * In the switchdev-capable NIC case: based on availability of devices with"},{"line_number":78,"context_line":"    the \"switchdev\" capability of PciDevices recorded in the Nova DB;"},{"line_number":79,"context_line":"* Building and running the instance, which involves:"},{"line_number":80,"context_line":""},{"line_number":81,"context_line":"  * Claiming PCI resources via the ResourceTracker at the target host based on"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9b4a3e92_b8fc3d10","line":78,"range":{"start_line":78,"start_character":0,"end_line":78,"end_character":69},"in_reply_to":"ee1de3e3_ee59ef2b","updated":"2021-04-26 16:02:28.000000000","message":"in theory yes the filter can match on that today but we do not ask for the capablity in the pci request spec.\n\nthat does not mean we cannot do so in the future but just a comment on the fact that since we do not ask for that capablity today the filter does not guarentee that.\n\nin fact we even disucssed using it to fix https://bugs.launchpad.net/nova/+bug/1915282 as part of hardening the security of hardware offloaded ovs.\n\ni was planning to propose a patch that would add a request for the swichdev capablity if the vif_type was ovs and vnic_type was direct or macvtap. all the code to enfore the requirement should be there we just need to start using it.","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":12171,"name":"Moshe Levi","email":"moshele@nvidia.com","username":"moshele"},"change_message_id":"9f9d00e45d2b7a27ef9e6607a073270d48e0d25f","unresolved":true,"context_lines":[{"line_number":85,"context_line":""},{"line_number":86,"context_line":"      * binding_host_id with the hypervisor hostname;"},{"line_number":87,"context_line":"      * binding:profile details with PCI device information,"},{"line_number":88,"context_line":"        namely: pci_vendor_info, pci_slot, physical_network;"},{"line_number":89,"context_line":""},{"line_number":90,"context_line":"* Network device assignment for the newly created instance and vif plugging"},{"line_number":91,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"b8e6a9d2_f77995be","line":88,"range":{"start_line":88,"start_character":0,"end_line":88,"end_character":60},"updated":"2021-04-24 21:28:51.000000000","message":"the only relevant information is the pci_slot. \nThe physical_network is used for IB SR-IOV.\nThe pci_vendor_info was used to help neutron to validated that the NIC is supported. We remove that validation long time ago. I don\u0027t think we used. (unless thinks as changed :))","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"949ec991d37115d125a08649056f3606564802b7","unresolved":true,"context_lines":[{"line_number":85,"context_line":""},{"line_number":86,"context_line":"      * binding_host_id with the hypervisor hostname;"},{"line_number":87,"context_line":"      * binding:profile details with PCI device information,"},{"line_number":88,"context_line":"        namely: pci_vendor_info, pci_slot, physical_network;"},{"line_number":89,"context_line":""},{"line_number":90,"context_line":"* Network device assignment for the newly created instance and vif plugging"},{"line_number":91,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"da3f6485_f82a62fa","line":88,"range":{"start_line":88,"start_character":0,"end_line":88,"end_character":60},"in_reply_to":"2b3136e9_a341ddce","updated":"2021-04-26 16:02:28.000000000","message":"yep moshe is correc but i also agree its good to level set on what is present today.\nthe pci_slot is indeed the only thing that is guarenteed to be there for neutron to consome but we always send all 3 even if the vendor info is not used.","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"78cd5865e1608849ac48d0068b25dcd7eadce9da","unresolved":true,"context_lines":[{"line_number":85,"context_line":""},{"line_number":86,"context_line":"      * binding_host_id with the hypervisor hostname;"},{"line_number":87,"context_line":"      * binding:profile details with PCI device information,"},{"line_number":88,"context_line":"        namely: pci_vendor_info, pci_slot, physical_network;"},{"line_number":89,"context_line":""},{"line_number":90,"context_line":"* Network device assignment for the newly created instance and vif plugging"},{"line_number":91,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"2b3136e9_a341ddce","line":88,"range":{"start_line":88,"start_character":0,"end_line":88,"end_character":60},"in_reply_to":"b8e6a9d2_f77995be","updated":"2021-04-26 11:27:12.000000000","message":"Agreed, just describing the process here regardless of whether things are actually used or not.\n\nThe reason I put it here is that I could not find an end-to-end description of the relevant process documented so I thought it would be good to have the relevant parts of it in the spec. Hopefully it will be clearer for reviewers that way.","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":12171,"name":"Moshe Levi","email":"moshele@nvidia.com","username":"moshele"},"change_message_id":"9f9d00e45d2b7a27ef9e6607a073270d48e0d25f","unresolved":true,"context_lines":[{"line_number":112,"context_line":"                                   │             │"},{"line_number":113,"context_line":"                                   │             │"},{"line_number":114,"context_line":"                               ┌───┴────┐    ┌───┴────┐"},{"line_number":115,"context_line":"             IO interconnect 1 │PF NUMA1│    │PF NUMA2│ IO interconnect 2"},{"line_number":116,"context_line":"                  (PCIe)       │VFs     │    │VFs     │    (PCIe)"},{"line_number":117,"context_line":"                               └────┬───┘    └───┬────┘"},{"line_number":118,"context_line":"                                    │            │"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9bf084a0_7de0c166","line":115,"range":{"start_line":115,"start_character":0,"end_line":115,"end_character":73},"updated":"2021-04-24 21:28:51.000000000","message":"why you have 2 NUMA is it socket direct card?\nI think we can simplify it with host with PF and DPU with PF Representor","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"78cd5865e1608849ac48d0068b25dcd7eadce9da","unresolved":true,"context_lines":[{"line_number":112,"context_line":"                                   │             │"},{"line_number":113,"context_line":"                                   │             │"},{"line_number":114,"context_line":"                               ┌───┴────┐    ┌───┴────┐"},{"line_number":115,"context_line":"             IO interconnect 1 │PF NUMA1│    │PF NUMA2│ IO interconnect 2"},{"line_number":116,"context_line":"                  (PCIe)       │VFs     │    │VFs     │    (PCIe)"},{"line_number":117,"context_line":"                               └────┬───┘    └───┬────┘"},{"line_number":118,"context_line":"                                    │            │"}],"source_content_type":"text/x-rst","patch_set":2,"id":"0121c277_d153aa1a","line":115,"range":{"start_line":115,"start_character":0,"end_line":115,"end_character":73},"in_reply_to":"9bf084a0_7de0c166","updated":"2021-04-26 11:27:12.000000000","message":"Yes, the aim is to show that the design factors in a possibility of having a SmartNIC connected to multiple PCIe hierarchies (e.g. in a dual-socket system).\n\nFor the dual-socket case, I am assuming that this should result in 4 PFs exposed to a dual-socket host (2 uplink ports x 2 NUMA nodes) when bonding and PF hiding is not configured.\n\nhttps://community.mellanox.com/s/article/getting-started-with-socket-direct-connectx-5-adapters-on-an-infiniband-network\n\"If you have two port Socket Direct Adapter, you are supposed to see PCI 4 devices, two physical and two logical, on the two PCI slots used.\"\n\nhttps://docs.mellanox.com/display/BlueFieldSWv31011424/Multi-host\n\"In multi-host mode, each host interface can be divided into up to 4 independent PCIe interfaces. All interfaces would share the same physical port, and are managed by the same multi-physical function switch (MPFS). Each host would have its own e-switch and would control its own traffic.\"\n\nhttps://docs.mellanox.com/display/BlueFieldSWv22011000/BlueField+Link+Aggregation\n\"# mlxconfig -d /dev/mst/mt41682_pciconf0 s HIDE_PORT2_PF\u003dTrue\n\nIf you do not perform this, the function will still be visible, but it will not receive any traffic.\"\n\n\nI\u0027ve seen similar functionality in SmartNICs from other vendors too so this should be fairly generic.","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":12171,"name":"Moshe Levi","email":"moshele@nvidia.com","username":"moshele"},"change_message_id":"9f9d00e45d2b7a27ef9e6607a073270d48e0d25f","unresolved":true,"context_lines":[{"line_number":118,"context_line":"                                    │            │"},{"line_number":119,"context_line":"   ┌────────────────────────────────┼────────────┼──────────────────────┐"},{"line_number":120,"context_line":"   │DPU/SmartNIC Board          ▲   │            │    ▲                 │"},{"line_number":121,"context_line":"   │(transport node)            │   │            │    │  Fast path      │"},{"line_number":122,"context_line":"   │      ┌─────────────┐         ┌─┴────────────┴─┐                    │"},{"line_number":123,"context_line":"   │      │ Application │         │     eSwitch    │     ┌────────────┐ │"},{"line_number":124,"context_line":"   │      │    CPU      ├─────────┤      ASIC      ├─────┤uplink ports│ │"}],"source_content_type":"text/x-rst","patch_set":2,"id":"2a3811df_bb75bc98","line":121,"updated":"2021-04-24 21:28:51.000000000","message":"you can add the OVS bridge and how the VF rep connect to it.","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"78cd5865e1608849ac48d0068b25dcd7eadce9da","unresolved":false,"context_lines":[{"line_number":118,"context_line":"                                    │            │"},{"line_number":119,"context_line":"   ┌────────────────────────────────┼────────────┼──────────────────────┐"},{"line_number":120,"context_line":"   │DPU/SmartNIC Board          ▲   │            │    ▲                 │"},{"line_number":121,"context_line":"   │(transport node)            │   │            │    │  Fast path      │"},{"line_number":122,"context_line":"   │      ┌─────────────┐         ┌─┴────────────┴─┐                    │"},{"line_number":123,"context_line":"   │      │ Application │         │     eSwitch    │     ┌────────────┐ │"},{"line_number":124,"context_line":"   │      │    CPU      ├─────────┤      ASIC      ├─────┤uplink ports│ │"}],"source_content_type":"text/x-rst","patch_set":2,"id":"c84636c0_eadbc941","line":121,"in_reply_to":"2a3811df_bb75bc98","updated":"2021-04-26 11:27:12.000000000","message":"Ack","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":12171,"name":"Moshe Levi","email":"moshele@nvidia.com","username":"moshele"},"change_message_id":"9f9d00e45d2b7a27ef9e6607a073270d48e0d25f","unresolved":true,"context_lines":[{"line_number":249,"context_line":"  * **On-path** FPGA-based SmartNICs are intentionally not considered because"},{"line_number":250,"context_line":"    of a different hardware architecture;"},{"line_number":251,"context_line":"* \"direct\" vnic type;"},{"line_number":252,"context_line":"* A new capability called \"transportnode\" used during scheduling (via a"},{"line_number":253,"context_line":"  binding:profile port attribute containing this capability);"},{"line_number":254,"context_line":""},{"line_number":255,"context_line":"  * The \"switchdev\" capability cannot be reused since a hypervisor does not"},{"line_number":256,"context_line":"    see the eswitch via PFs exposed to it by a SmartNIC/DPU;"}],"source_content_type":"text/x-rst","patch_set":2,"id":"e44cf1cf_0ce4b2e0","line":253,"range":{"start_line":252,"start_character":0,"end_line":253,"end_character":60},"updated":"2021-04-24 21:28:51.000000000","message":"I am not sure we want to create another capability. For DPU support in ironic with created on new port \"smart-nic\" [1]. It will be nice to reuse it for this use-case as well\n\n[1] - https://github.com/openstack/neutron-lib/blob/f01b2e9025d33aeff3bf22ea2568bda036878819/neutron_lib/api/definitions/portbindings.py#L119","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"78cd5865e1608849ac48d0068b25dcd7eadce9da","unresolved":true,"context_lines":[{"line_number":249,"context_line":"  * **On-path** FPGA-based SmartNICs are intentionally not considered because"},{"line_number":250,"context_line":"    of a different hardware architecture;"},{"line_number":251,"context_line":"* \"direct\" vnic type;"},{"line_number":252,"context_line":"* A new capability called \"transportnode\" used during scheduling (via a"},{"line_number":253,"context_line":"  binding:profile port attribute containing this capability);"},{"line_number":254,"context_line":""},{"line_number":255,"context_line":"  * The \"switchdev\" capability cannot be reused since a hypervisor does not"},{"line_number":256,"context_line":"    see the eswitch via PFs exposed to it by a SmartNIC/DPU;"}],"source_content_type":"text/x-rst","patch_set":2,"id":"0d25adfa_f4bea1ad","line":253,"range":{"start_line":252,"start_character":0,"end_line":253,"end_character":60},"in_reply_to":"e44cf1cf_0ce4b2e0","updated":"2021-04-26 11:27:12.000000000","message":"This is more for scheduling purposes.\n\nThere is functionality in Nova that allows target nodes to be filtered based on the PCI device requests created early in the instance creation process:\n\nhttps://opendev.org/openstack/nova/src/tag/23.0.0/nova/compute/api.py#L1036-L1038 (InstancePCIRequests request creation)\n\nhttps://opendev.org/openstack/nova/src/tag/21.2.0/nova/scheduler/filters/pci_passthrough_filter.py#L23-L56 (the filter code)\nhttps://opendev.org/openstack/nova/src/tag/23.0.0/nova/pci/stats.py#L505-L525 (the support_requests method)\nhttps://opendev.org/openstack/nova/src/tag/23.0.0/nova/pci/stats.py#L415-L503 (_filter_pools)\nhttps://opendev.org/openstack/nova/src/tag/23.0.0/nova/pci/stats.py#L241-L258 (_filter_pools_for_spec)\nhttps://opendev.org/openstack/nova/src/tag/21.2.0/nova/pci/utils.py#L38-L70 (pci_device_prop_match)\n\n\nCurrently, if the \"switchdev\" capability is present in the extra_info column for a particular PciDevice entry, then the scheduler is able to make the right instance placement decision.\n\nThe goal here is to provide similar functionality but for PCIe devices that do not have \"switchdev\" exposed but are known to be associated with a SmartNIC based on the proposed. transport_node_list config.","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":12171,"name":"Moshe Levi","email":"moshele@nvidia.com","username":"moshele"},"change_message_id":"9f9d00e45d2b7a27ef9e6607a073270d48e0d25f","unresolved":true,"context_lines":[{"line_number":271,"context_line":"* PCI(e) support initially while keeping the design extensible for other types"},{"line_number":272,"context_line":"  of devices (platform devices, CXL 2.0 etc.);"},{"line_number":273,"context_line":"* PF/VFs with vendor-specific control and data plane with future extensions to"},{"line_number":274,"context_line":"  support hardware or software vDPA to use the virtio data plane."},{"line_number":275,"context_line":""},{"line_number":276,"context_line":"The SmartNIC/DPU provisioning is outside of the scope of this specification,"},{"line_number":277,"context_line":"however, the expectation is that it should happen prior to the provisioning"}],"source_content_type":"text/x-rst","patch_set":2,"id":"8970d591_f9b81f60","line":274,"range":{"start_line":274,"start_character":10,"end_line":274,"end_character":35},"updated":"2021-04-24 21:28:51.000000000","message":"is wither software forwarder of vdpa. Anyway I will just remove the hardware or software.","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"78cd5865e1608849ac48d0068b25dcd7eadce9da","unresolved":true,"context_lines":[{"line_number":271,"context_line":"* PCI(e) support initially while keeping the design extensible for other types"},{"line_number":272,"context_line":"  of devices (platform devices, CXL 2.0 etc.);"},{"line_number":273,"context_line":"* PF/VFs with vendor-specific control and data plane with future extensions to"},{"line_number":274,"context_line":"  support hardware or software vDPA to use the virtio data plane."},{"line_number":275,"context_line":""},{"line_number":276,"context_line":"The SmartNIC/DPU provisioning is outside of the scope of this specification,"},{"line_number":277,"context_line":"however, the expectation is that it should happen prior to the provisioning"}],"source_content_type":"text/x-rst","patch_set":2,"id":"30c54484_a1ab5611","line":274,"range":{"start_line":274,"start_character":10,"end_line":274,"end_character":35},"in_reply_to":"8970d591_f9b81f60","updated":"2021-04-26 11:27:12.000000000","message":"OK, I suppose just referencing \"vDPA\" is enough in this spec.","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":12171,"name":"Moshe Levi","email":"moshele@nvidia.com","username":"moshele"},"change_message_id":"9f9d00e45d2b7a27ef9e6607a073270d48e0d25f","unresolved":true,"context_lines":[{"line_number":281,"context_line":"will be up to the deployment automation to do it."},{"line_number":282,"context_line":""},{"line_number":283,"context_line":"From the deployer point of view, besides understanding the provisioning"},{"line_number":284,"context_line":"challenges, it is also expected that a list of devices to be considered"},{"line_number":285,"context_line":"transport nodes will be formed out of PCI device IDs and vendor IDs of PCI"},{"line_number":286,"context_line":"endpoints known to be provided by SmartNICs/DPUs."},{"line_number":287,"context_line":""},{"line_number":288,"context_line":"For an end user, the workflow will be similar to the one with the hardware"},{"line_number":289,"context_line":"offload functionality except for additional multiplexing between the"}],"source_content_type":"text/x-rst","patch_set":2,"id":"ef6cd158_8e050127","line":286,"range":{"start_line":284,"start_character":12,"end_line":286,"end_character":49},"updated":"2021-04-24 21:28:51.000000000","message":"This sentence is not clear to me, basicly the deploy need to provide the device IDs for the smart-nic on each node. right?","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"78cd5865e1608849ac48d0068b25dcd7eadce9da","unresolved":true,"context_lines":[{"line_number":281,"context_line":"will be up to the deployment automation to do it."},{"line_number":282,"context_line":""},{"line_number":283,"context_line":"From the deployer point of view, besides understanding the provisioning"},{"line_number":284,"context_line":"challenges, it is also expected that a list of devices to be considered"},{"line_number":285,"context_line":"transport nodes will be formed out of PCI device IDs and vendor IDs of PCI"},{"line_number":286,"context_line":"endpoints known to be provided by SmartNICs/DPUs."},{"line_number":287,"context_line":""},{"line_number":288,"context_line":"For an end user, the workflow will be similar to the one with the hardware"},{"line_number":289,"context_line":"offload functionality except for additional multiplexing between the"}],"source_content_type":"text/x-rst","patch_set":2,"id":"740702c7_d432bcb5","line":286,"range":{"start_line":284,"start_character":12,"end_line":286,"end_character":49},"in_reply_to":"ef6cd158_8e050127","updated":"2021-04-26 11:27:12.000000000","message":"Yes, this is mainly for adding the proposed \"transportnode\" capability to discovered PciDevices such that scheduling can be done correctly.\n\nWe don\u0027t have a distinct capability exposed by a driver via devlink so a config seems to be the only option (besides hard-coding vendor and device IDs in the code which is undesirable in my view).","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":12171,"name":"Moshe Levi","email":"moshele@nvidia.com","username":"moshele"},"change_message_id":"9f9d00e45d2b7a27ef9e6607a073270d48e0d25f","unresolved":true,"context_lines":[{"line_number":301,"context_line":""},{"line_number":302,"context_line":"Largely, the goal is to create transport node resource providers in"},{"line_number":303,"context_line":"the Placement service based on observed PCI(e) cards and a config value in the"},{"line_number":304,"context_line":"[pci] section declaring which cards to treat as transport nodes based on the"},{"line_number":305,"context_line":"vendor and device IDs of PCI(e) PFs/VFs those cards expose. Since both the"},{"line_number":306,"context_line":"hypervisor host and the transport node host that belong to the same physical"},{"line_number":307,"context_line":"machine can see PCI(e) functions exposed by the same card, they can see the"},{"line_number":308,"context_line":"same unique serial number. Nova can register transport node resource providers"}],"source_content_type":"text/x-rst","patch_set":2,"id":"250ae914_51171cdd","line":305,"range":{"start_line":304,"start_character":0,"end_line":305,"end_character":58},"updated":"2021-04-24 21:28:51.000000000","message":"What is the transport nodes config Device ID or BDF?\nWhy you need the VFs? can\u0027t you infer it from the pci_whitelist?","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"37e6197ddca87f10096c3dc30f401f53460837a7","unresolved":true,"context_lines":[{"line_number":301,"context_line":""},{"line_number":302,"context_line":"Largely, the goal is to create transport node resource providers in"},{"line_number":303,"context_line":"the Placement service based on observed PCI(e) cards and a config value in the"},{"line_number":304,"context_line":"[pci] section declaring which cards to treat as transport nodes based on the"},{"line_number":305,"context_line":"vendor and device IDs of PCI(e) PFs/VFs those cards expose. Since both the"},{"line_number":306,"context_line":"hypervisor host and the transport node host that belong to the same physical"},{"line_number":307,"context_line":"machine can see PCI(e) functions exposed by the same card, they can see the"},{"line_number":308,"context_line":"same unique serial number. Nova can register transport node resource providers"}],"source_content_type":"text/x-rst","patch_set":2,"id":"0938097b_9adf6511","line":305,"range":{"start_line":304,"start_character":0,"end_line":305,"end_character":58},"in_reply_to":"250ae914_51171cdd","updated":"2021-04-26 15:00:07.000000000","message":"I planned to reuse the PCI whitelist code for this config and maybe restricting it to vendor_id + device ID.\n\nIt could be BDF but I don\u0027t think this is very reliable as a device could be relocated into a different slot and BDF numbers may change if things like \"bus padding\" are configured some time after the initial deployment.","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":12171,"name":"Moshe Levi","email":"moshele@nvidia.com","username":"moshele"},"change_message_id":"9f9d00e45d2b7a27ef9e6607a073270d48e0d25f","unresolved":true,"context_lines":[{"line_number":335,"context_line":"      * 5.1 - devlink-info API;"},{"line_number":336,"context_line":"      * 5.9 - \"board.serial_number\" added to devlink-info (subject to NIC"},{"line_number":337,"context_line":"        driver support);"},{"line_number":338,"context_line":"  * using PCIe VPD data exposed by PCI(e) endpoints: as a fallback if devlink"},{"line_number":339,"context_line":"    is not usable for some reason (old kernel, no driver support etc.);"},{"line_number":340,"context_line":"* Extend the libvirt virt driver code to gather serial numbers for PFs and VFs"},{"line_number":341,"context_line":"  and report this information back to the ResourceTracker code for processing;"},{"line_number":342,"context_line":"* Extend PciDevTracker code to create and PciCard entries in the database based"}],"source_content_type":"text/x-rst","patch_set":2,"id":"8d09fef2_41aabeb1","line":339,"range":{"start_line":338,"start_character":0,"end_line":339,"end_character":71},"updated":"2021-04-24 21:28:51.000000000","message":"how you plan to extract it with lspci?","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"37e6197ddca87f10096c3dc30f401f53460837a7","unresolved":true,"context_lines":[{"line_number":335,"context_line":"      * 5.1 - devlink-info API;"},{"line_number":336,"context_line":"      * 5.9 - \"board.serial_number\" added to devlink-info (subject to NIC"},{"line_number":337,"context_line":"        driver support);"},{"line_number":338,"context_line":"  * using PCIe VPD data exposed by PCI(e) endpoints: as a fallback if devlink"},{"line_number":339,"context_line":"    is not usable for some reason (old kernel, no driver support etc.);"},{"line_number":340,"context_line":"* Extend the libvirt virt driver code to gather serial numbers for PFs and VFs"},{"line_number":341,"context_line":"  and report this information back to the ResourceTracker code for processing;"},{"line_number":342,"context_line":"* Extend PciDevTracker code to create and PciCard entries in the database based"}],"source_content_type":"text/x-rst","patch_set":2,"id":"fca11bca_0219c93f","line":339,"range":{"start_line":338,"start_character":0,"end_line":339,"end_character":71},"in_reply_to":"8d09fef2_41aabeb1","updated":"2021-04-26 15:00:07.000000000","message":"2 ways:\n\n1) via devlink (if the kernel and userspace versions allow for it and the driver exposes the board_serial via devlink-info);\n2) by reading PCI VPD via sysfs (see below).\n\nI have a python implementation for a VPD format reader (the format is described in the PCI/PCIe specs). It reads PCI VPD from a sysfs location and extracts the serial number.\n\nhttps://gist.github.com/dshcherb/40e982989599a757e5b1e25999501019\n\nThe support for this sysfs file was added a long time ago so it is a good fallback:\nhttps://www.kernel.org/doc/Documentation/ABI/testing/sysfs-bus-pci\n\"What:\t\t/sys/bus/pci/devices/.../vpd\nDate:\t\tFebruary 2008\nContact:\tBen Hutchings \u003cbwh@kernel.org\u003e\nDescription:\n\t\tA file named vpd in a device directory will be a\n\t\tbinary file containing the Vital Product Data for the\n\t\tdevice.  It should follow the VPD format defined in\n\t\tPCI Specification 2.1 or 2.2, but users should consider\n\t\tthat some devices may have malformatted data.  If the\n\t\tunderlying VPD has a writable section then the\n\t\tcorresponding section of this file will be writable.\n\"","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":12171,"name":"Moshe Levi","email":"moshele@nvidia.com","username":"moshele"},"change_message_id":"9f9d00e45d2b7a27ef9e6607a073270d48e0d25f","unresolved":true,"context_lines":[{"line_number":365,"context_line":""},{"line_number":366,"context_line":"Whether Nova or the OVN mechanism driver in Neutron will do the lookup of a"},{"line_number":367,"context_line":"transport node hostname during the port update step to properly set"},{"line_number":368,"context_line":"binding_host_id is TBD - it should become more apparent during prototyping."},{"line_number":369,"context_line":"What is clear is that Nova needs to send some information to Neutron:"},{"line_number":370,"context_line":""},{"line_number":371,"context_line":"* transport node hostname or serial number associated with a selected VF so"}],"source_content_type":"text/x-rst","patch_set":2,"id":"5b9c76f1_d30380cc","line":368,"range":{"start_line":368,"start_character":0,"end_line":368,"end_character":22},"updated":"2021-04-24 21:28:51.000000000","message":"you want the host_id to be the DPU hostname? Maybe you should extend neutron to support\nbinding_host_id  - hypervisor \nbinding_network_host_id - DPU","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":12171,"name":"Moshe Levi","email":"moshele@nvidia.com","username":"moshele"},"change_message_id":"9f9d00e45d2b7a27ef9e6607a073270d48e0d25f","unresolved":true,"context_lines":[{"line_number":366,"context_line":"Whether Nova or the OVN mechanism driver in Neutron will do the lookup of a"},{"line_number":367,"context_line":"transport node hostname during the port update step to properly set"},{"line_number":368,"context_line":"binding_host_id is TBD - it should become more apparent during prototyping."},{"line_number":369,"context_line":"What is clear is that Nova needs to send some information to Neutron:"},{"line_number":370,"context_line":""},{"line_number":371,"context_line":"* transport node hostname or serial number associated with a selected VF so"},{"line_number":372,"context_line":"  that the right transport node is selected for representor plugging at the"}],"source_content_type":"text/x-rst","patch_set":2,"id":"bffd4d74_316bcdfd","line":369,"range":{"start_line":369,"start_character":0,"end_line":369,"end_character":69},"updated":"2021-04-24 21:28:51.000000000","message":"I assume that you refer to nova-compute because he will know the VF index after the claim. \nhow will you pass that information? Do we have support for such communication to day?\nI know in regular case after os-vif called nova-compute send notification to neutron-server. I I wonder how it will work in this case...","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":13686,"name":"Frode Nordahl","email":"fnordahl@ubuntu.com","username":"fnordahl"},"change_message_id":"9a442f045618aaed809f481ce6c656d7c1f93b5d","unresolved":true,"context_lines":[{"line_number":366,"context_line":"Whether Nova or the OVN mechanism driver in Neutron will do the lookup of a"},{"line_number":367,"context_line":"transport node hostname during the port update step to properly set"},{"line_number":368,"context_line":"binding_host_id is TBD - it should become more apparent during prototyping."},{"line_number":369,"context_line":"What is clear is that Nova needs to send some information to Neutron:"},{"line_number":370,"context_line":""},{"line_number":371,"context_line":"* transport node hostname or serial number associated with a selected VF so"},{"line_number":372,"context_line":"  that the right transport node is selected for representor plugging at the"}],"source_content_type":"text/x-rst","patch_set":2,"id":"231a535f_2a9d62b7","line":369,"range":{"start_line":369,"start_character":0,"end_line":369,"end_character":69},"in_reply_to":"bffd4d74_316bcdfd","updated":"2021-04-25 06:14:22.000000000","message":"Yes, when nova-compute builds an instance today it will contact Neutron and provide binding_host_id and update the binding_profile field of the port. In the case of regular SR-IOV it will provide extra detail such as PCI address and such already.\n\nSo what we need to do is to extend that with the information required for looking up the correct DPU host and information about logical PF and VF that can be forwarded to OVN for representor port lookup by the ovn-controller running on the NIC.","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":13686,"name":"Frode Nordahl","email":"fnordahl@ubuntu.com","username":"fnordahl"},"change_message_id":"551c24e482f9990d8abffc7fcdb26d3a5bdf04fb","unresolved":true,"context_lines":[{"line_number":371,"context_line":"* transport node hostname or serial number associated with a selected VF so"},{"line_number":372,"context_line":"  that the right transport node is selected for representor plugging at the"},{"line_number":373,"context_line":"  Neutron/OVN side (as there may be multiple of those in one physical host);"},{"line_number":374,"context_line":"* VF logical number information since those numbers are tracked in the NIC"},{"line_number":375,"context_line":"  firmware and can be observed on both the main host side and the SmartNIC/DPU"},{"line_number":376,"context_line":"  side."},{"line_number":377,"context_line":""},{"line_number":378,"context_line":"Alternatives"},{"line_number":379,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":2,"id":"54de328a_fd6933dd","line":376,"range":{"start_line":374,"start_character":0,"end_line":376,"end_character":7},"updated":"2021-04-24 07:02:33.000000000","message":"In addition to providing information about which NIC SoC and logical VF number we need to provide information that allows identifying to which physical port on the NIC the logical VF number belongs in order to successfully look up the correct representor port.\n\nIn some scenarios the user might also want to actually plug the PF into an instance.\n\nDuring prototyping of the OVN Representor Port plugging support I found that using the MAC address of the PF to be the most reliable identifier that would be visible from both the host and NIC SoC side for presently consumable kernel versions.\n\nUsing that also allows for the OVN Representor Port plugging code to plug both PF and VF representors using the same database options and code paths.","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":13686,"name":"Frode Nordahl","email":"fnordahl@ubuntu.com","username":"fnordahl"},"change_message_id":"341efd8a0358a646e87013fad107d2dd105e5833","unresolved":true,"context_lines":[{"line_number":371,"context_line":"* transport node hostname or serial number associated with a selected VF so"},{"line_number":372,"context_line":"  that the right transport node is selected for representor plugging at the"},{"line_number":373,"context_line":"  Neutron/OVN side (as there may be multiple of those in one physical host);"},{"line_number":374,"context_line":"* VF logical number information since those numbers are tracked in the NIC"},{"line_number":375,"context_line":"  firmware and can be observed on both the main host side and the SmartNIC/DPU"},{"line_number":376,"context_line":"  side."},{"line_number":377,"context_line":""},{"line_number":378,"context_line":"Alternatives"},{"line_number":379,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":2,"id":"c0171bd1_da8e0a60","line":376,"range":{"start_line":374,"start_character":0,"end_line":376,"end_character":7},"in_reply_to":"11425bc3_902f580d","updated":"2021-04-27 10:49:15.000000000","message":"Never mind the last comment, the logical numbering does not appear to be visible to the host side through devlink-port after all. We can still determine the logical VF number from host side with other means though.","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":13686,"name":"Frode Nordahl","email":"fnordahl@ubuntu.com","username":"fnordahl"},"change_message_id":"d51bd80544efed7cc8fe2e54681f91c434b3aaf1","unresolved":true,"context_lines":[{"line_number":371,"context_line":"* transport node hostname or serial number associated with a selected VF so"},{"line_number":372,"context_line":"  that the right transport node is selected for representor plugging at the"},{"line_number":373,"context_line":"  Neutron/OVN side (as there may be multiple of those in one physical host);"},{"line_number":374,"context_line":"* VF logical number information since those numbers are tracked in the NIC"},{"line_number":375,"context_line":"  firmware and can be observed on both the main host side and the SmartNIC/DPU"},{"line_number":376,"context_line":"  side."},{"line_number":377,"context_line":""},{"line_number":378,"context_line":"Alternatives"},{"line_number":379,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":2,"id":"11425bc3_902f580d","line":376,"range":{"start_line":374,"start_character":0,"end_line":376,"end_character":7},"in_reply_to":"54de328a_fd6933dd","updated":"2021-04-24 08:04:03.000000000","message":"Actually, we could probably also use the logical PF number as provided by the devlink-port infrastructure. But Nova already stores the PF MAC and has existing helpers for PCI \u003c-\u003e MAC lookups, so I guess that is why I went with that initially.","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":12171,"name":"Moshe Levi","email":"moshele@nvidia.com","username":"moshele"},"change_message_id":"9f9d00e45d2b7a27ef9e6607a073270d48e0d25f","unresolved":true,"context_lines":[{"line_number":396,"context_line":"  better to extend OVN to do that and handle VF plugging at the Nova side)."},{"line_number":397,"context_line":""},{"line_number":398,"context_line":"One alternative approach involves tracking cards using a separate service with"},{"line_number":399,"context_line":"its own API and possibly introducing a different VNIC type: this does not have"},{"line_number":400,"context_line":"a benefit of code reuse and requires another service to be added and integrated"},{"line_number":401,"context_line":"with Nova and Neutron at minimum. Evolving the work that was done to enable"},{"line_number":402,"context_line":"hardware offloaded ports seems like a more effective way to address this"},{"line_number":403,"context_line":"use-case."},{"line_number":404,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"e3c013a1_2887341c","line":401,"range":{"start_line":399,"start_character":25,"end_line":401,"end_character":33},"updated":"2021-04-24 21:28:51.000000000","message":"can you elaborate on this point? why you can do code reuse with different VNIC type? Ithink it will be super important to align this work with the ironic smart-nic work that is already exist.","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"37e6197ddca87f10096c3dc30f401f53460837a7","unresolved":true,"context_lines":[{"line_number":396,"context_line":"  better to extend OVN to do that and handle VF plugging at the Nova side)."},{"line_number":397,"context_line":""},{"line_number":398,"context_line":"One alternative approach involves tracking cards using a separate service with"},{"line_number":399,"context_line":"its own API and possibly introducing a different VNIC type: this does not have"},{"line_number":400,"context_line":"a benefit of code reuse and requires another service to be added and integrated"},{"line_number":401,"context_line":"with Nova and Neutron at minimum. Evolving the work that was done to enable"},{"line_number":402,"context_line":"hardware offloaded ports seems like a more effective way to address this"},{"line_number":403,"context_line":"use-case."},{"line_number":404,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"801b6369_07b4f790","line":401,"range":{"start_line":399,"start_character":25,"end_line":401,"end_character":33},"in_reply_to":"e3c013a1_2887341c","updated":"2021-04-26 15:00:07.000000000","message":"What I mean here is that there could be a completely different API service to track SmartNICs. I would like to avoid that and re-use the logic that already exists in Nova and Neutron with some minimal modifications.\n\nWhether type \"direct\" or type \"smart-nic\" (introduced with the Ironic work) will be used is a secondary concern: I am trying to avoid adding yet another type besides those two and relying solely on Nova, Neutron and OVN when it comes to the services used.","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":12171,"name":"Moshe Levi","email":"moshele@nvidia.com","username":"moshele"},"change_message_id":"9f9d00e45d2b7a27ef9e6607a073270d48e0d25f","unresolved":true,"context_lines":[{"line_number":511,"context_line":"  PCI(e) add-in cards from the PFs and VFs exposed by them;"},{"line_number":512,"context_line":"* Extra lookups need to be performed in order to create transport node"},{"line_number":513,"context_line":"  resource providers."},{"line_number":514,"context_line":""},{"line_number":515,"context_line":"Other deployer impact"},{"line_number":516,"context_line":"---------------------"},{"line_number":517,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"a8d539eb_f99fd7f0","line":514,"updated":"2021-04-24 21:28:51.000000000","message":"so it seem that also nova-compute need to provide information to neutron on which VF port to bind. I didn\u0027t understand if it adds another communication between them","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":13686,"name":"Frode Nordahl","email":"fnordahl@ubuntu.com","username":"fnordahl"},"change_message_id":"9a442f045618aaed809f481ce6c656d7c1f93b5d","unresolved":true,"context_lines":[{"line_number":511,"context_line":"  PCI(e) add-in cards from the PFs and VFs exposed by them;"},{"line_number":512,"context_line":"* Extra lookups need to be performed in order to create transport node"},{"line_number":513,"context_line":"  resource providers."},{"line_number":514,"context_line":""},{"line_number":515,"context_line":"Other deployer impact"},{"line_number":516,"context_line":"---------------------"},{"line_number":517,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"b0a866a7_e33e41bd","line":514,"in_reply_to":"a8d539eb_f99fd7f0","updated":"2021-04-25 06:14:22.000000000","message":"This communication already exists, either Nova will update the binding_host_id and binding_profile of the port provided at instance creation time.","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":12171,"name":"Moshe Levi","email":"moshele@nvidia.com","username":"moshele"},"change_message_id":"9f9d00e45d2b7a27ef9e6607a073270d48e0d25f","unresolved":true,"context_lines":[{"line_number":534,"context_line":""},{"line_number":535,"context_line":"During the deployment planning it is also important to take control traffic"},{"line_number":536,"context_line":"paths into account. Nova compute is expected to trigger representor plugging"},{"line_number":537,"context_line":"indirectly via Neutron as a client via the control network: Neutron is then"},{"line_number":538,"context_line":"responsible for propagating this information back to the responsible agents on"},{"line_number":539,"context_line":"a relevant transport node (via the OVN mechanism driver in this case). Also,"},{"line_number":540,"context_line":"Placement service updates from hypervisor nodes happen over the control"},{"line_number":541,"context_line":"network. This may happen via dedicated ports programmed on the eSwitch which"},{"line_number":542,"context_line":"needs to be done via some form of a deployment automation. Alternatively, LoMs"}],"source_content_type":"text/x-rst","patch_set":2,"id":"f480e7f7_454b1834","line":539,"range":{"start_line":537,"start_character":60,"end_line":539,"end_character":69},"updated":"2021-04-24 21:28:51.000000000","message":"how will nova know that neutron was able to plug the port?","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":13686,"name":"Frode Nordahl","email":"fnordahl@ubuntu.com","username":"fnordahl"},"change_message_id":"9a442f045618aaed809f481ce6c656d7c1f93b5d","unresolved":true,"context_lines":[{"line_number":534,"context_line":""},{"line_number":535,"context_line":"During the deployment planning it is also important to take control traffic"},{"line_number":536,"context_line":"paths into account. Nova compute is expected to trigger representor plugging"},{"line_number":537,"context_line":"indirectly via Neutron as a client via the control network: Neutron is then"},{"line_number":538,"context_line":"responsible for propagating this information back to the responsible agents on"},{"line_number":539,"context_line":"a relevant transport node (via the OVN mechanism driver in this case). Also,"},{"line_number":540,"context_line":"Placement service updates from hypervisor nodes happen over the control"},{"line_number":541,"context_line":"network. This may happen via dedicated ports programmed on the eSwitch which"},{"line_number":542,"context_line":"needs to be done via some form of a deployment automation. Alternatively, LoMs"}],"source_content_type":"text/x-rst","patch_set":2,"id":"27a4c478_d5b32962","line":539,"range":{"start_line":537,"start_character":60,"end_line":539,"end_character":69},"in_reply_to":"f480e7f7_454b1834","updated":"2021-04-25 06:14:22.000000000","message":"Today nova-compute will do the following:\n1) With instance paused, update Neutron binding_host_id, binding_profile\n2) Plug port\n2) Wait for notification from Neutron about port binding being successful\n3) Resume instance\n\nWhat we propose to do is to based on VNIC-type tell Nova to not attempt to do the port plugging at all. Instead it will provide neutron with the necessary information, which is then forwarded to ovn-controller running on the NIC, which then both plugs and binds the representor port.\n\nAs soon as the plugging and binding is successful Neutron will notify Nova (it already does this today) and the VM will be resumed and instance creation complete.","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":12171,"name":"Moshe Levi","email":"moshele@nvidia.com","username":"moshele"},"change_message_id":"9f9d00e45d2b7a27ef9e6607a073270d48e0d25f","unresolved":true,"context_lines":[{"line_number":602,"context_line":""},{"line_number":603,"context_line":"Processes on the hypervisor host would use the PF associated with an uplink"},{"line_number":604,"context_line":"port or a bond (or VLAN interfaces on top of those) in order to communicate"},{"line_number":605,"context_line":"with control processes. Special VFs for control traffic would only be needed"},{"line_number":606,"context_line":"if control traffic needs to be offloaded. The same applies to the processes"},{"line_number":607,"context_line":"running on the application CPU."},{"line_number":608,"context_line":""},{"line_number":609,"context_line":"DPUs/SmartNICs themselves do not typically have a BMC themselves and draw"}],"source_content_type":"text/x-rst","patch_set":2,"id":"74c7fc71_9919987d","line":606,"range":{"start_line":605,"start_character":24,"end_line":606,"end_character":41},"updated":"2021-04-24 21:28:51.000000000","message":"why? we can offload PF traffic as well","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"37e6197ddca87f10096c3dc30f401f53460837a7","unresolved":true,"context_lines":[{"line_number":602,"context_line":""},{"line_number":603,"context_line":"Processes on the hypervisor host would use the PF associated with an uplink"},{"line_number":604,"context_line":"port or a bond (or VLAN interfaces on top of those) in order to communicate"},{"line_number":605,"context_line":"with control processes. Special VFs for control traffic would only be needed"},{"line_number":606,"context_line":"if control traffic needs to be offloaded. The same applies to the processes"},{"line_number":607,"context_line":"running on the application CPU."},{"line_number":608,"context_line":""},{"line_number":609,"context_line":"DPUs/SmartNICs themselves do not typically have a BMC themselves and draw"}],"source_content_type":"text/x-rst","patch_set":2,"id":"2417abda_d9b08668","line":606,"range":{"start_line":605,"start_character":24,"end_line":606,"end_character":41},"in_reply_to":"74c7fc71_9919987d","updated":"2021-04-26 15:00:07.000000000","message":"In that case this comment is wrong - will remove it.","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":12171,"name":"Moshe Levi","email":"moshele@nvidia.com","username":"moshele"},"change_message_id":"9f9d00e45d2b7a27ef9e6607a073270d48e0d25f","unresolved":true,"context_lines":[{"line_number":646,"context_line":"Liaison Needed"},{"line_number":647,"context_line":""},{"line_number":648,"context_line":"Work Items"},{"line_number":649,"context_line":"----------"},{"line_number":650,"context_line":""},{"line_number":651,"context_line":"* Implement the code to work with devlink-info for serial number extraction;"},{"line_number":652,"context_line":"* Implement the code to parse the binary format of PCI(e) VPD via sysfs per"}],"source_content_type":"text/x-rst","patch_set":2,"id":"48f2bda4_fac4cc09","line":649,"updated":"2021-04-24 21:28:51.000000000","message":"where the work item to pass neutron the information. Is there a neutron spec for the networking-ovn changes?","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"37e6197ddca87f10096c3dc30f401f53460837a7","unresolved":true,"context_lines":[{"line_number":646,"context_line":"Liaison Needed"},{"line_number":647,"context_line":""},{"line_number":648,"context_line":"Work Items"},{"line_number":649,"context_line":"----------"},{"line_number":650,"context_line":""},{"line_number":651,"context_line":"* Implement the code to work with devlink-info for serial number extraction;"},{"line_number":652,"context_line":"* Implement the code to parse the binary format of PCI(e) VPD via sysfs per"}],"source_content_type":"text/x-rst","patch_set":2,"id":"cab9ba61_5b2823e6","line":649,"in_reply_to":"48f2bda4_fac4cc09","updated":"2021-04-26 15:00:07.000000000","message":"Drafting it at the moment - will publish it soon.","commit_id":"a8c670dee68210020e5223a7ab94de4582212167"}],"specs/xena/approved/transport-nodes.rst":[{"author":{"_account_id":18511,"name":"Brian Wickersham","email":"bkw86@bellsouth.net","username":"bw6938"},"change_message_id":"c6dae54630cd833cd0d7e73d0cbbdb33894a6a9b","unresolved":true,"context_lines":[{"line_number":436,"context_line":"So the logical numbers for representor flavors are correctly identified at the"},{"line_number":437,"context_line":"transport node but are not visible at the hypervisor host."},{"line_number":438,"context_line":""},{"line_number":439,"context_line":"VF PCI addresses at the hypevisor side are calculated per the PCIe and SR-IOV"},{"line_number":440,"context_line":"specs using the PF PCI address, \"First VF Offset\" and \"VF Stride\" values and"},{"line_number":441,"context_line":"the logical per-PF numbering is maintained by the kernel and exposed via sysfs."},{"line_number":442,"context_line":"Therefore, we can take logical VF numbers from the following sysfs entries::"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a9c299a_8113abed","line":439,"range":{"start_line":439,"start_character":24,"end_line":439,"end_character":33},"updated":"2021-04-29 19:07:57.000000000","message":"hypervisor","commit_id":"2e3e13781872d593aaf4708f2fabfb048b1a55d0"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"fc85234cb23b464b96c15d0888dfcbf08e7951c2","unresolved":false,"context_lines":[{"line_number":436,"context_line":"So the logical numbers for representor flavors are correctly identified at the"},{"line_number":437,"context_line":"transport node but are not visible at the hypervisor host."},{"line_number":438,"context_line":""},{"line_number":439,"context_line":"VF PCI addresses at the hypevisor side are calculated per the PCIe and SR-IOV"},{"line_number":440,"context_line":"specs using the PF PCI address, \"First VF Offset\" and \"VF Stride\" values and"},{"line_number":441,"context_line":"the logical per-PF numbering is maintained by the kernel and exposed via sysfs."},{"line_number":442,"context_line":"Therefore, we can take logical VF numbers from the following sysfs entries::"}],"source_content_type":"text/x-rst","patch_set":3,"id":"651c8f89_1bdb8781","line":439,"range":{"start_line":439,"start_character":24,"end_line":439,"end_character":33},"in_reply_to":"5a9c299a_8113abed","updated":"2021-04-30 10:38:04.000000000","message":"Ack","commit_id":"2e3e13781872d593aaf4708f2fabfb048b1a55d0"},{"author":{"_account_id":18511,"name":"Brian Wickersham","email":"bkw86@bellsouth.net","username":"bw6938"},"change_message_id":"c6dae54630cd833cd0d7e73d0cbbdb33894a6a9b","unresolved":true,"context_lines":[{"line_number":476,"context_line":"  need to participate in port representor plugging and eSwitch programming"},{"line_number":477,"context_line":"  which is not necessarily specific to Nova or even OpenStack. Other"},{"line_number":478,"context_line":"  infrastructure projects may benefit from that as well so the larger effort"},{"line_number":479,"context_line":"  needs to concentrate on reusability. This is why possible a introduction of"},{"line_number":480,"context_line":"  transport node services specific to OpenStack needs to be avoided (i.e. it is"},{"line_number":481,"context_line":"  better to extend OVN to do that and handle VF plugging at the Nova side)."},{"line_number":482,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"5509a715_9d1e42a4","line":479,"range":{"start_line":479,"start_character":60,"end_line":479,"end_character":61},"updated":"2021-04-29 19:07:57.000000000","message":"an","commit_id":"2e3e13781872d593aaf4708f2fabfb048b1a55d0"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"fc85234cb23b464b96c15d0888dfcbf08e7951c2","unresolved":false,"context_lines":[{"line_number":476,"context_line":"  need to participate in port representor plugging and eSwitch programming"},{"line_number":477,"context_line":"  which is not necessarily specific to Nova or even OpenStack. Other"},{"line_number":478,"context_line":"  infrastructure projects may benefit from that as well so the larger effort"},{"line_number":479,"context_line":"  needs to concentrate on reusability. This is why possible a introduction of"},{"line_number":480,"context_line":"  transport node services specific to OpenStack needs to be avoided (i.e. it is"},{"line_number":481,"context_line":"  better to extend OVN to do that and handle VF plugging at the Nova side)."},{"line_number":482,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"2b684b99_7efe786f","line":479,"range":{"start_line":479,"start_character":60,"end_line":479,"end_character":61},"in_reply_to":"5509a715_9d1e42a4","updated":"2021-04-30 10:38:04.000000000","message":"Ack","commit_id":"2e3e13781872d593aaf4708f2fabfb048b1a55d0"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":5,"context_line":" http://creativecommons.org/licenses/by/3.0/legalcode"},{"line_number":6,"context_line":""},{"line_number":7,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":8,"context_line":"Introduce Transport Nodes"},{"line_number":9,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/nova/+spec/introduce-transport-nodes"}],"source_content_type":"text/x-rst","patch_set":4,"id":"d10d6325_79f92ad8","line":8,"range":{"start_line":8,"start_character":0,"end_line":8,"end_character":25},"updated":"2021-05-07 12:08:25.000000000","message":"Integration with remote networking agents.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":false,"context_lines":[{"line_number":5,"context_line":" http://creativecommons.org/licenses/by/3.0/legalcode"},{"line_number":6,"context_line":""},{"line_number":7,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":8,"context_line":"Introduce Transport Nodes"},{"line_number":9,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/nova/+spec/introduce-transport-nodes"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7948807c_9979951d","line":8,"range":{"start_line":8,"start_character":0,"end_line":8,"end_character":25},"in_reply_to":"d10d6325_79f92ad8","updated":"2021-05-11 13:17:45.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":8,"context_line":"Introduce Transport Nodes"},{"line_number":9,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/nova/+spec/introduce-transport-nodes"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"Off-path SmartNICs (or DPUs) introduce an architecture change where"},{"line_number":14,"context_line":"network agents responsible for eSwitch configuration and representor interface"}],"source_content_type":"text/x-rst","patch_set":4,"id":"8c7da485_552217f4","line":11,"range":{"start_line":11,"start_character":44,"end_line":11,"end_character":69},"updated":"2021-05-07 12:08:25.000000000","message":"when i first read the title there was a large disconnect between the term introduce-transport-nodes and the content of the spec. For me with a nova context my mind went to adding something like the nova conductor.\n\nas an operator i think i would find this title confusing","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":8,"context_line":"Introduce Transport Nodes"},{"line_number":9,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/nova/+spec/introduce-transport-nodes"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"Off-path SmartNICs (or DPUs) introduce an architecture change where"},{"line_number":14,"context_line":"network agents responsible for eSwitch configuration and representor interface"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5b9a19c3_f92d1b50","line":11,"range":{"start_line":11,"start_character":44,"end_line":11,"end_character":69},"in_reply_to":"62b2ba98_a123016e","updated":"2021-05-21 19:01:48.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":8,"context_line":"Introduce Transport Nodes"},{"line_number":9,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/nova/+spec/introduce-transport-nodes"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"Off-path SmartNICs (or DPUs) introduce an architecture change where"},{"line_number":14,"context_line":"network agents responsible for eSwitch configuration and representor interface"}],"source_content_type":"text/x-rst","patch_set":4,"id":"62b2ba98_a123016e","line":11,"range":{"start_line":11,"start_character":44,"end_line":11,"end_character":69},"in_reply_to":"8c7da485_552217f4","updated":"2021-05-11 13:17:45.000000000","message":"OK, one of the reasons I had for this particular naming was around providing a distinct name for an entity in the resource provider hierarchy. Since that is gone after the initial review, I agree that it should have a different name not revolving around the concept of the previous iteration.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":14,"context_line":"network agents responsible for eSwitch configuration and representor interface"},{"line_number":15,"context_line":"plugging run on a separate SoC with its own CPU, memory and that runs a"},{"line_number":16,"context_line":"separate OS kernel. The side-effect of that is that hypervisor hostnames no"},{"line_number":17,"context_line":"longer match SmartNIC (DPU) hostnames which are seen by OVS and OVN agents"},{"line_number":18,"context_line":"while the existing port binding code relies on that. The goal of this"},{"line_number":19,"context_line":"specification is to introduce changes necessary to extend the existing hardware"},{"line_number":20,"context_line":"offload code to cope with the hostname mismatch and related design challenges"}],"source_content_type":"text/x-rst","patch_set":4,"id":"96433585_7149083d","line":17,"range":{"start_line":17,"start_character":55,"end_line":17,"end_character":67},"updated":"2021-05-07 12:08:25.000000000","message":"do you intend to enabel both ml2/ovs and ml2/ovn this cycle.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":14,"context_line":"network agents responsible for eSwitch configuration and representor interface"},{"line_number":15,"context_line":"plugging run on a separate SoC with its own CPU, memory and that runs a"},{"line_number":16,"context_line":"separate OS kernel. The side-effect of that is that hypervisor hostnames no"},{"line_number":17,"context_line":"longer match SmartNIC (DPU) hostnames which are seen by OVS and OVN agents"},{"line_number":18,"context_line":"while the existing port binding code relies on that. The goal of this"},{"line_number":19,"context_line":"specification is to introduce changes necessary to extend the existing hardware"},{"line_number":20,"context_line":"offload code to cope with the hostname mismatch and related design challenges"}],"source_content_type":"text/x-rst","patch_set":4,"id":"e2958499_19d1c3bf","line":17,"range":{"start_line":17,"start_character":55,"end_line":17,"end_character":67},"in_reply_to":"075d9701_1941aa68","updated":"2021-05-21 19:01:48.000000000","message":"Done","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":14,"context_line":"network agents responsible for eSwitch configuration and representor interface"},{"line_number":15,"context_line":"plugging run on a separate SoC with its own CPU, memory and that runs a"},{"line_number":16,"context_line":"separate OS kernel. The side-effect of that is that hypervisor hostnames no"},{"line_number":17,"context_line":"longer match SmartNIC (DPU) hostnames which are seen by OVS and OVN agents"},{"line_number":18,"context_line":"while the existing port binding code relies on that. The goal of this"},{"line_number":19,"context_line":"specification is to introduce changes necessary to extend the existing hardware"},{"line_number":20,"context_line":"offload code to cope with the hostname mismatch and related design challenges"}],"source_content_type":"text/x-rst","patch_set":4,"id":"075d9701_1941aa68","line":17,"range":{"start_line":17,"start_character":55,"end_line":17,"end_character":67},"in_reply_to":"96433585_7149083d","updated":"2021-05-11 13:17:45.000000000","message":"This comment was mainly to say that the hostname stored in the ovsdb (open_vswitch table) and the chassis name in the southbound db will differ from the hypervisor hostname (not to refer to the neutron-openvswitch-agent).\n\nAs such, we planned to go for ML2/OVN only and leave ML2/OVS out of scope since most of the new developments in Neutron are around OVN.\n\nOne other reason was that other projects besides OpenStack could leverage OVN-side representor plugging (k8s comes to mind).","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":24,"context_line":"program the necessary flows and do the necessary port plugging. Additionally,"},{"line_number":25,"context_line":"more information is suggested to be passed in the \"binding:profile\" during"},{"line_number":26,"context_line":"a port update to facilitate representor port plugging."},{"line_number":27,"context_line":""},{"line_number":28,"context_line":""},{"line_number":29,"context_line":"Problem description"},{"line_number":30,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":4,"id":"bfcd8a34_fdaf22ee","line":27,"updated":"2021-05-07 12:08:25.000000000","message":"+1","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":false,"context_lines":[{"line_number":24,"context_line":"program the necessary flows and do the necessary port plugging. Additionally,"},{"line_number":25,"context_line":"more information is suggested to be passed in the \"binding:profile\" during"},{"line_number":26,"context_line":"a port update to facilitate representor port plugging."},{"line_number":27,"context_line":""},{"line_number":28,"context_line":""},{"line_number":29,"context_line":"Problem description"},{"line_number":30,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":4,"id":"eff88b5d_b54cbad8","line":27,"in_reply_to":"bfcd8a34_fdaf22ee","updated":"2021-05-11 13:17:45.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":32,"context_line":"Terminology"},{"line_number":33,"context_line":"-----------"},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"* Off-path SmartNIC architecture [1]_ [2]_ - an architecture where NIC cores"},{"line_number":36,"context_line":"  are responsible for programming an eSwitch and are bypassed when rules"},{"line_number":37,"context_line":"  programmed into the eSwitch are enough to make a decision on where to deliver"},{"line_number":38,"context_line":"  packets. Normally, NIC cores only participate in packet forwarding for the"},{"line_number":39,"context_line":"  \"slow path\" only and the \"fast path\" is handled via an ASIC;"},{"line_number":40,"context_line":"* On-path SmartNIC architecture [1]_ [2]_ - an architecture where NIC cores"},{"line_number":41,"context_line":"  participate in processing of every packet going through the NIC as a whole."},{"line_number":42,"context_line":"  In other words, NIC cores are always on the \"fast path\" of all packets. This"},{"line_number":43,"context_line":"  type of SmartNICs typically contains FPGAs;"},{"line_number":44,"context_line":"* Transport Node - a term used in OVN to refer collectively to hypervisors and"},{"line_number":45,"context_line":"  gateways per ovn-architecture(7) [3]_. Used in this specification to refer"},{"line_number":46,"context_line":"  more generically to SmartNICs or DPUs;"}],"source_content_type":"text/x-rst","patch_set":4,"id":"aa9aa4cd_eef6e9c9","line":43,"range":{"start_line":35,"start_character":0,"end_line":43,"end_character":45},"updated":"2021-05-07 12:08:25.000000000","message":"we can work with these definition but they are not how im used to thinking of this.\nfor me an off-path accelerator is one in which  the accelerator asic that runs in a look aside mode and may or may not have cpu cores. where as inline smart nics use an asic or fpga in the data plane.\n\na melonox connectx6-dx where teh control plane run on the host would be an inline smartnics it would alos be an on-path smartnic is VDPA mode since it actully cannot handel the virtio translation in hardware and useisn the onboard nic cores to do that in firmeware.\n\nthe point i wanted to make is on-path and off-path in my previous exprience has never been related to if there are cpu cores or not, it has been a reference to if there is an accelerator element inline with the nic phy though which all packets are processed(an inline/on-path smartnic) where the accelerator element work in a look aside mode in parallel to the main datastream(off-path).\n\nso on-path/off-path has generally been used to refer to data plane processing not contolplane execution.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":32,"context_line":"Terminology"},{"line_number":33,"context_line":"-----------"},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"* Off-path SmartNIC architecture [1]_ [2]_ - an architecture where NIC cores"},{"line_number":36,"context_line":"  are responsible for programming an eSwitch and are bypassed when rules"},{"line_number":37,"context_line":"  programmed into the eSwitch are enough to make a decision on where to deliver"},{"line_number":38,"context_line":"  packets. Normally, NIC cores only participate in packet forwarding for the"},{"line_number":39,"context_line":"  \"slow path\" only and the \"fast path\" is handled via an ASIC;"},{"line_number":40,"context_line":"* On-path SmartNIC architecture [1]_ [2]_ - an architecture where NIC cores"},{"line_number":41,"context_line":"  participate in processing of every packet going through the NIC as a whole."},{"line_number":42,"context_line":"  In other words, NIC cores are always on the \"fast path\" of all packets. This"},{"line_number":43,"context_line":"  type of SmartNICs typically contains FPGAs;"},{"line_number":44,"context_line":"* Transport Node - a term used in OVN to refer collectively to hypervisors and"},{"line_number":45,"context_line":"  gateways per ovn-architecture(7) [3]_. Used in this specification to refer"},{"line_number":46,"context_line":"  more generically to SmartNICs or DPUs;"}],"source_content_type":"text/x-rst","patch_set":4,"id":"19c6a920_5f46fd55","line":43,"range":{"start_line":35,"start_character":0,"end_line":43,"end_character":45},"in_reply_to":"387dc8ca_7e8a9959","updated":"2021-05-21 19:01:48.000000000","message":"Done","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":32,"context_line":"Terminology"},{"line_number":33,"context_line":"-----------"},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"* Off-path SmartNIC architecture [1]_ [2]_ - an architecture where NIC cores"},{"line_number":36,"context_line":"  are responsible for programming an eSwitch and are bypassed when rules"},{"line_number":37,"context_line":"  programmed into the eSwitch are enough to make a decision on where to deliver"},{"line_number":38,"context_line":"  packets. Normally, NIC cores only participate in packet forwarding for the"},{"line_number":39,"context_line":"  \"slow path\" only and the \"fast path\" is handled via an ASIC;"},{"line_number":40,"context_line":"* On-path SmartNIC architecture [1]_ [2]_ - an architecture where NIC cores"},{"line_number":41,"context_line":"  participate in processing of every packet going through the NIC as a whole."},{"line_number":42,"context_line":"  In other words, NIC cores are always on the \"fast path\" of all packets. This"},{"line_number":43,"context_line":"  type of SmartNICs typically contains FPGAs;"},{"line_number":44,"context_line":"* Transport Node - a term used in OVN to refer collectively to hypervisors and"},{"line_number":45,"context_line":"  gateways per ovn-architecture(7) [3]_. Used in this specification to refer"},{"line_number":46,"context_line":"  more generically to SmartNICs or DPUs;"}],"source_content_type":"text/x-rst","patch_set":4,"id":"387dc8ca_7e8a9959","line":43,"range":{"start_line":35,"start_character":0,"end_line":43,"end_character":45},"in_reply_to":"aa9aa4cd_eef6e9c9","updated":"2021-05-11 13:17:45.000000000","message":"I see your point. I\u0027ve seen the term \"intelligent NIC\" pushed around to refer to NICs like ConnectX ones without additional ARM cores.\n\nI was basing those on the definitions from the referenced paper and presentation which assume that a SmartNIC has an \"Application CPU\" or \"NIC cores\". The paper also calls them collectively as \"Multicore SoC SmartNICs\" trying to underline the presence of additional cores (whether in the off-path or on-path architecture).\n\nhttps://homes.cs.washington.edu/~arvind/papers/ipipe.pdf\n\"A Multicore SoC SmartNIC consists of four major parts (as shown in Figure 1-a): (1) computing units, including a general-purpose ARM/MIPS multicore processor, along with accelerators for packet processing (e.g., deep packet inspection, packet buffer management) and specialized functions (e.g., encryption/decryption, hashing, pattern matching, compression); (2) onboard memory, enclosing fast self-managed scratchpad and slower L2/DRAM; (3) traffic control module that transfers packets between TX/RX ports and the packet buffer, accompanied by an internal traffic manager or NIC switch that delivers packets to NIC cores; (4) DMA engines for communicating with the host.\"\n\n\"Based on ***how SmartNIC cores interact with traffic***, we further categorize SmartNICs into two types: on-path and off-path SmartNICs. The cores for on-path SmartNICs (Figure 1-b) are on the packet communication path and hold the capability to manipulate each incoming/outgoing packet\"\n\nhttps://netdevconf.info/0x14/pub/slides/39/Netdev%200x14%20--%20Taking%20Control%20of%20your%20SmartNIC%20v1.pdf\nSlide 7 \"Benefits of On-Path Architecture\": \"**NIC cores** have direct access to packet memory for low latency packet processing\"\nSlide 8 \"Benefits of Off-Path Architecture\": \"ASIC hardware allows packets to skip **NIC cores** and go directly to host cores\"\n\nNot saying those are a single source of truth at this particular point - standardization is definitely lacking in this area and I am trying to reuse terms others have come up with as much as possible.\n\nI propose two alternative sets of definitions to fix this:\n\n1)\n- Off-path multi-core SoC SmartNIC;\n- On-path multi-core SoC SmartNIC.\n\n2)\n- Off-path SmartNIC DPU;\n- On-path SmartNIC DPU.\n\nThe latter being subjectively better.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":41,"context_line":"  participate in processing of every packet going through the NIC as a whole."},{"line_number":42,"context_line":"  In other words, NIC cores are always on the \"fast path\" of all packets. This"},{"line_number":43,"context_line":"  type of SmartNICs typically contains FPGAs;"},{"line_number":44,"context_line":"* Transport Node - a term used in OVN to refer collectively to hypervisors and"},{"line_number":45,"context_line":"  gateways per ovn-architecture(7) [3]_. Used in this specification to refer"},{"line_number":46,"context_line":"  more generically to SmartNICs or DPUs;"},{"line_number":47,"context_line":"* eSwitch - a programmable embedded switch present in various types of NICs"},{"line_number":48,"context_line":"  (SR-IOV-capable NICs, off-path SmartNICs). Typically relies on ASICs for"},{"line_number":49,"context_line":"  packet processing;"}],"source_content_type":"text/x-rst","patch_set":4,"id":"9124cc19_ae80878d","line":46,"range":{"start_line":44,"start_character":0,"end_line":46,"end_character":40},"updated":"2021-05-07 12:08:25.000000000","message":"in general i think this is highly unintuitive term to use and i would personally avoid it in the nova spec and use SmartNic or DPU","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":41,"context_line":"  participate in processing of every packet going through the NIC as a whole."},{"line_number":42,"context_line":"  In other words, NIC cores are always on the \"fast path\" of all packets. This"},{"line_number":43,"context_line":"  type of SmartNICs typically contains FPGAs;"},{"line_number":44,"context_line":"* Transport Node - a term used in OVN to refer collectively to hypervisors and"},{"line_number":45,"context_line":"  gateways per ovn-architecture(7) [3]_. Used in this specification to refer"},{"line_number":46,"context_line":"  more generically to SmartNICs or DPUs;"},{"line_number":47,"context_line":"* eSwitch - a programmable embedded switch present in various types of NICs"},{"line_number":48,"context_line":"  (SR-IOV-capable NICs, off-path SmartNICs). Typically relies on ASICs for"},{"line_number":49,"context_line":"  packet processing;"}],"source_content_type":"text/x-rst","patch_set":4,"id":"d9d673ea_715bae36","line":46,"range":{"start_line":44,"start_character":0,"end_line":46,"end_character":40},"in_reply_to":"9124cc19_ae80878d","updated":"2021-05-11 13:17:45.000000000","message":"OK, based on my suggestion in the comment above I suggest using a combination: \"SmartNIC DPU\".","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1933e7abc61d9926e1fb5cba9bb19aac9ef7ff52","unresolved":false,"context_lines":[{"line_number":41,"context_line":"  participate in processing of every packet going through the NIC as a whole."},{"line_number":42,"context_line":"  In other words, NIC cores are always on the \"fast path\" of all packets. This"},{"line_number":43,"context_line":"  type of SmartNICs typically contains FPGAs;"},{"line_number":44,"context_line":"* Transport Node - a term used in OVN to refer collectively to hypervisors and"},{"line_number":45,"context_line":"  gateways per ovn-architecture(7) [3]_. Used in this specification to refer"},{"line_number":46,"context_line":"  more generically to SmartNICs or DPUs;"},{"line_number":47,"context_line":"* eSwitch - a programmable embedded switch present in various types of NICs"},{"line_number":48,"context_line":"  (SR-IOV-capable NICs, off-path SmartNICs). Typically relies on ASICs for"},{"line_number":49,"context_line":"  packet processing;"}],"source_content_type":"text/x-rst","patch_set":4,"id":"c1ceb5b3_da17ebde","line":46,"range":{"start_line":44,"start_character":0,"end_line":46,"end_character":40},"in_reply_to":"d9d673ea_715bae36","updated":"2021-05-18 12:45:55.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":44,"context_line":"* Transport Node - a term used in OVN to refer collectively to hypervisors and"},{"line_number":45,"context_line":"  gateways per ovn-architecture(7) [3]_. Used in this specification to refer"},{"line_number":46,"context_line":"  more generically to SmartNICs or DPUs;"},{"line_number":47,"context_line":"* eSwitch - a programmable embedded switch present in various types of NICs"},{"line_number":48,"context_line":"  (SR-IOV-capable NICs, off-path SmartNICs). Typically relies on ASICs for"},{"line_number":49,"context_line":"  packet processing;"},{"line_number":50,"context_line":"* switchdev [4]_ - in-kernel driver model for switch devices which offload the"}],"source_content_type":"text/x-rst","patch_set":4,"id":"e09632d2_041cc68f","line":47,"range":{"start_line":47,"start_character":2,"end_line":47,"end_character":9},"updated":"2021-05-11 13:17:45.000000000","message":"There was a request from a reviewer of the Neutron spec to change it to \"NIC Switch\" - going to do that here as well.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1933e7abc61d9926e1fb5cba9bb19aac9ef7ff52","unresolved":false,"context_lines":[{"line_number":44,"context_line":"* Transport Node - a term used in OVN to refer collectively to hypervisors and"},{"line_number":45,"context_line":"  gateways per ovn-architecture(7) [3]_. Used in this specification to refer"},{"line_number":46,"context_line":"  more generically to SmartNICs or DPUs;"},{"line_number":47,"context_line":"* eSwitch - a programmable embedded switch present in various types of NICs"},{"line_number":48,"context_line":"  (SR-IOV-capable NICs, off-path SmartNICs). Typically relies on ASICs for"},{"line_number":49,"context_line":"  packet processing;"},{"line_number":50,"context_line":"* switchdev [4]_ - in-kernel driver model for switch devices which offload the"}],"source_content_type":"text/x-rst","patch_set":4,"id":"e0398a76_6b3d64b2","line":47,"range":{"start_line":47,"start_character":2,"end_line":47,"end_character":9},"in_reply_to":"e09632d2_041cc68f","updated":"2021-05-18 12:45:55.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":68,"context_line":"switchdev-capable NICs. However, further work is needed in order to support"},{"line_number":69,"context_line":"off-path SmartNICs which also expose PCI(e) functions to the hypervisor hosts."},{"line_number":70,"context_line":""},{"line_number":71,"context_line":"When working with ports of type \"direct\", instance creation involves several"},{"line_number":72,"context_line":"key steps, including:"},{"line_number":73,"context_line":""},{"line_number":74,"context_line":"* Creating the necessary context based on a client request (including PCI"}],"source_content_type":"text/x-rst","patch_set":4,"id":"00a9a024_9d70dbbf","line":71,"range":{"start_line":71,"start_character":27,"end_line":71,"end_character":31},"updated":"2021-05-07 12:08:25.000000000","message":"vnic_type","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":68,"context_line":"switchdev-capable NICs. However, further work is needed in order to support"},{"line_number":69,"context_line":"off-path SmartNICs which also expose PCI(e) functions to the hypervisor hosts."},{"line_number":70,"context_line":""},{"line_number":71,"context_line":"When working with ports of type \"direct\", instance creation involves several"},{"line_number":72,"context_line":"key steps, including:"},{"line_number":73,"context_line":""},{"line_number":74,"context_line":"* Creating the necessary context based on a client request (including PCI"}],"source_content_type":"text/x-rst","patch_set":4,"id":"9f08616e_8ce3cdaf","line":71,"range":{"start_line":71,"start_character":27,"end_line":71,"end_character":31},"in_reply_to":"00a9a024_9d70dbbf","updated":"2021-05-11 13:17:45.000000000","message":"Ack, will change that.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":68,"context_line":"switchdev-capable NICs. However, further work is needed in order to support"},{"line_number":69,"context_line":"off-path SmartNICs which also expose PCI(e) functions to the hypervisor hosts."},{"line_number":70,"context_line":""},{"line_number":71,"context_line":"When working with ports of type \"direct\", instance creation involves several"},{"line_number":72,"context_line":"key steps, including:"},{"line_number":73,"context_line":""},{"line_number":74,"context_line":"* Creating the necessary context based on a client request (including PCI"}],"source_content_type":"text/x-rst","patch_set":4,"id":"4b508c66_37e85d6d","line":71,"range":{"start_line":71,"start_character":27,"end_line":71,"end_character":31},"in_reply_to":"9f08616e_8ce3cdaf","updated":"2021-05-21 19:01:48.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":135,"context_line":"(terms \"SmartNIC\" and \"DPU\" will be used interchangeably with the term"},{"line_number":136,"context_line":"\"transport node\" in this spec.)"},{"line_number":137,"context_line":""},{"line_number":138,"context_line":"With off-path SmartNICs/DPUs, if an eSwitch has the necessary flows programmed"},{"line_number":139,"context_line":"and an incoming packet matches those flows, it is delivered to the destination"},{"line_number":140,"context_line":"over the fast path bypassing the \"Application CPU\". Otherwise, the packet is"},{"line_number":141,"context_line":"processed in software at the Application CPU and then forwarded to the"},{"line_number":142,"context_line":"destination."},{"line_number":143,"context_line":""},{"line_number":144,"context_line":"There are more sophisticated scenarios as well:"},{"line_number":145,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"fd603660_77e7ebe2","line":142,"range":{"start_line":138,"start_character":0,"end_line":142,"end_character":12},"updated":"2021-05-07 12:08:25.000000000","message":"this is generally true of on path smartnics too by the way at least for\nthe definition im more familiar with.\n\nthat is how the existing hardware offloaed ovs works.\nthis proposals does not reduce the packet processing overhead on vs normal hardware offloaded ovs.\n\nthis proposal reduces the control plane over head e.g. the flow calculation done by the ovs-vswitchd or the ovn/neutron agent.\n\nwhere hardware offload are already processing the packets this is a very low overhead since only the first few packets are processed by the host cpu in software.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":135,"context_line":"(terms \"SmartNIC\" and \"DPU\" will be used interchangeably with the term"},{"line_number":136,"context_line":"\"transport node\" in this spec.)"},{"line_number":137,"context_line":""},{"line_number":138,"context_line":"With off-path SmartNICs/DPUs, if an eSwitch has the necessary flows programmed"},{"line_number":139,"context_line":"and an incoming packet matches those flows, it is delivered to the destination"},{"line_number":140,"context_line":"over the fast path bypassing the \"Application CPU\". Otherwise, the packet is"},{"line_number":141,"context_line":"processed in software at the Application CPU and then forwarded to the"},{"line_number":142,"context_line":"destination."},{"line_number":143,"context_line":""},{"line_number":144,"context_line":"There are more sophisticated scenarios as well:"},{"line_number":145,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"3c25904b_fdd296ff","line":142,"range":{"start_line":138,"start_character":0,"end_line":142,"end_character":12},"in_reply_to":"98027f25_99d465eb","updated":"2021-05-21 19:01:48.000000000","message":"Done","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":135,"context_line":"(terms \"SmartNIC\" and \"DPU\" will be used interchangeably with the term"},{"line_number":136,"context_line":"\"transport node\" in this spec.)"},{"line_number":137,"context_line":""},{"line_number":138,"context_line":"With off-path SmartNICs/DPUs, if an eSwitch has the necessary flows programmed"},{"line_number":139,"context_line":"and an incoming packet matches those flows, it is delivered to the destination"},{"line_number":140,"context_line":"over the fast path bypassing the \"Application CPU\". Otherwise, the packet is"},{"line_number":141,"context_line":"processed in software at the Application CPU and then forwarded to the"},{"line_number":142,"context_line":"destination."},{"line_number":143,"context_line":""},{"line_number":144,"context_line":"There are more sophisticated scenarios as well:"},{"line_number":145,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"98027f25_99d465eb","line":142,"range":{"start_line":138,"start_character":0,"end_line":142,"end_character":12},"in_reply_to":"fd603660_77e7ebe2","updated":"2021-05-11 13:17:45.000000000","message":"\u003e this proposals does not reduce the packet processing overhead on vs normal hardware offloaded ovs.\n\nAgreed, not intending to compare the on-path and off-path ones in this paragraph - just trying to briefly introduce the concept of off-path ones that have an application CPU as shown on the picture above.\n\nI think you are right that there should be more words said around the move of the control plane component which results in less overhead at the hypervisor host.\n\nHowever, this proposal is also about bringing isolation of the networking control plane from the hypervisor into the design.\n\nSubjectively, just addressing the control plane overhead is not a good enough reason to justify a significant change in the architecture.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":151,"context_line":"  * PCIe Shared IO [9]_."},{"line_number":152,"context_line":""},{"line_number":153,"context_line":"Networking agents such as ovs-vswitchd and ovn-controller are expected to run"},{"line_number":154,"context_line":"on the SmartNIC OS which will have a different hostname from the hypervisor"},{"line_number":155,"context_line":"OS which results in a mismatch during port binding (more specifically, the"},{"line_number":156,"context_line":"external_ids[\"hostname\"] field in the Open_vSwitch table differs from the"},{"line_number":157,"context_line":"hypervisor hostname). Likewise, representor plugging and flow programming"}],"source_content_type":"text/x-rst","patch_set":4,"id":"49194cd2_5fa705a4","line":154,"range":{"start_line":154,"start_character":7,"end_line":154,"end_character":15},"updated":"2021-05-07 12:08:25.000000000","message":"by the way are you talking about things like the BlueField-2 DPU product line rather then standard smartnics like connectx6-dx sicne while they may have cores interenlly they are not capable of runing ovs/ovn to my knowledge.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":151,"context_line":"  * PCIe Shared IO [9]_."},{"line_number":152,"context_line":""},{"line_number":153,"context_line":"Networking agents such as ovs-vswitchd and ovn-controller are expected to run"},{"line_number":154,"context_line":"on the SmartNIC OS which will have a different hostname from the hypervisor"},{"line_number":155,"context_line":"OS which results in a mismatch during port binding (more specifically, the"},{"line_number":156,"context_line":"external_ids[\"hostname\"] field in the Open_vSwitch table differs from the"},{"line_number":157,"context_line":"hypervisor hostname). Likewise, representor plugging and flow programming"}],"source_content_type":"text/x-rst","patch_set":4,"id":"c0d81950_56d77ca3","line":154,"range":{"start_line":154,"start_character":7,"end_line":154,"end_character":15},"in_reply_to":"49194cd2_5fa705a4","updated":"2021-05-11 13:17:45.000000000","message":"Yes, {NVIDIA/Mellanox Bluefield, Broadcom Stringray, Marvell Octeon DPUs, Netronome Agilio} etc.\n\nThere is some terminology confusion when it comes to naming those: I believe some refer to NICs like ConnectX 6 DX as \"intelligent NICs\" while referring to the ones that have a dedicated \"application CPU\" as \"SmartNICs\".\n\nThere are certainly internal CPUs there for things like PHY management, PXE, NC-SI and other things which run their own firmware on the drawing above I refer to this as a \"management CPU\" while things may be way more complex - this is largely a space where the particulars are likely  a confidential IP which vendors hide for competitive reasons. Kernel drivers and NVRAM options give hints about the functionality they implement though.\n\nYes, ConnectX 6\u0027s internal CPUs do other things and there is no way to run OVS/ovn-controller there.\n\nI avoided specific product references to keep it generic and applicable to a class of SmartNICs rather than a particular family only and keep it neutral such that different parties interested in it could provide feedback.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1933e7abc61d9926e1fb5cba9bb19aac9ef7ff52","unresolved":true,"context_lines":[{"line_number":151,"context_line":"  * PCIe Shared IO [9]_."},{"line_number":152,"context_line":""},{"line_number":153,"context_line":"Networking agents such as ovs-vswitchd and ovn-controller are expected to run"},{"line_number":154,"context_line":"on the SmartNIC OS which will have a different hostname from the hypervisor"},{"line_number":155,"context_line":"OS which results in a mismatch during port binding (more specifically, the"},{"line_number":156,"context_line":"external_ids[\"hostname\"] field in the Open_vSwitch table differs from the"},{"line_number":157,"context_line":"hypervisor hostname). Likewise, representor plugging and flow programming"}],"source_content_type":"text/x-rst","patch_set":4,"id":"c69a11cf_e8d63465","line":154,"range":{"start_line":154,"start_character":7,"end_line":154,"end_character":15},"in_reply_to":"c0d81950_56d77ca3","updated":"2021-05-18 12:45:55.000000000","message":"sure but it can also be useful to refence concrete examples to better define the classes of devices.\n\nyou dont have to include the product names in the spec its self as long as we know the set of capablites the deivce classes posesss.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":151,"context_line":"  * PCIe Shared IO [9]_."},{"line_number":152,"context_line":""},{"line_number":153,"context_line":"Networking agents such as ovs-vswitchd and ovn-controller are expected to run"},{"line_number":154,"context_line":"on the SmartNIC OS which will have a different hostname from the hypervisor"},{"line_number":155,"context_line":"OS which results in a mismatch during port binding (more specifically, the"},{"line_number":156,"context_line":"external_ids[\"hostname\"] field in the Open_vSwitch table differs from the"},{"line_number":157,"context_line":"hypervisor hostname). Likewise, representor plugging and flow programming"}],"source_content_type":"text/x-rst","patch_set":4,"id":"c15a9f7a_9ec3d606","line":154,"range":{"start_line":154,"start_character":7,"end_line":154,"end_character":15},"in_reply_to":"c69a11cf_e8d63465","updated":"2021-05-21 19:01:48.000000000","message":"Done","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":158,"context_line":"happens on the SmartNIC host, not on the hypervisor host. As a result, Nova"},{"line_number":159,"context_line":"(with the help of os-vif) can no longer be responsible for VIF plugging in the"},{"line_number":160,"context_line":"same way it is in the hardware offload scenario: OVS bridges and port"},{"line_number":161,"context_line":"representors are no longer exposed to the OS on the main board."},{"line_number":162,"context_line":""},{"line_number":163,"context_line":"Since Nova and networking agents run on different hosts, there needs to be a"},{"line_number":164,"context_line":"different set of interactions in order to:"}],"source_content_type":"text/x-rst","patch_set":4,"id":"9ffb7e6f_aa2b5c4a","line":161,"range":{"start_line":161,"start_character":52,"end_line":161,"end_character":62},"updated":"2021-05-07 12:08:25.000000000","message":"host server.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1933e7abc61d9926e1fb5cba9bb19aac9ef7ff52","unresolved":true,"context_lines":[{"line_number":158,"context_line":"happens on the SmartNIC host, not on the hypervisor host. As a result, Nova"},{"line_number":159,"context_line":"(with the help of os-vif) can no longer be responsible for VIF plugging in the"},{"line_number":160,"context_line":"same way it is in the hardware offload scenario: OVS bridges and port"},{"line_number":161,"context_line":"representors are no longer exposed to the OS on the main board."},{"line_number":162,"context_line":""},{"line_number":163,"context_line":"Since Nova and networking agents run on different hosts, there needs to be a"},{"line_number":164,"context_line":"different set of interactions in order to:"}],"source_content_type":"text/x-rst","patch_set":4,"id":"c61deb46_6c1a2a7b","line":161,"range":{"start_line":161,"start_character":52,"end_line":161,"end_character":62},"in_reply_to":"81b8e5c6_73d549a1","updated":"2021-05-18 12:45:55.000000000","message":"ack, i generally avoid using the term hypervior since it slightly nebulious if you mean a software component or not but \"hypervisor host\" works for me.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":158,"context_line":"happens on the SmartNIC host, not on the hypervisor host. As a result, Nova"},{"line_number":159,"context_line":"(with the help of os-vif) can no longer be responsible for VIF plugging in the"},{"line_number":160,"context_line":"same way it is in the hardware offload scenario: OVS bridges and port"},{"line_number":161,"context_line":"representors are no longer exposed to the OS on the main board."},{"line_number":162,"context_line":""},{"line_number":163,"context_line":"Since Nova and networking agents run on different hosts, there needs to be a"},{"line_number":164,"context_line":"different set of interactions in order to:"}],"source_content_type":"text/x-rst","patch_set":4,"id":"81b8e5c6_73d549a1","line":161,"range":{"start_line":161,"start_character":52,"end_line":161,"end_character":62},"in_reply_to":"9ffb7e6f_aa2b5c4a","updated":"2021-05-11 13:17:45.000000000","message":"I suggest using the term \"hypervisor host\" to replace this then. I deliberately try to avoid the term \"host\" as a standalone thing and \"server\" since it may refer to the hardware system as a whole.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":158,"context_line":"happens on the SmartNIC host, not on the hypervisor host. As a result, Nova"},{"line_number":159,"context_line":"(with the help of os-vif) can no longer be responsible for VIF plugging in the"},{"line_number":160,"context_line":"same way it is in the hardware offload scenario: OVS bridges and port"},{"line_number":161,"context_line":"representors are no longer exposed to the OS on the main board."},{"line_number":162,"context_line":""},{"line_number":163,"context_line":"Since Nova and networking agents run on different hosts, there needs to be a"},{"line_number":164,"context_line":"different set of interactions in order to:"}],"source_content_type":"text/x-rst","patch_set":4,"id":"1506d4d6_f8c43031","line":161,"range":{"start_line":161,"start_character":52,"end_line":161,"end_character":62},"in_reply_to":"c61deb46_6c1a2a7b","updated":"2021-05-21 19:01:48.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":165,"context_line":""},{"line_number":166,"context_line":"* Schedule an instance to a host where a VF with the necessary capability is"},{"line_number":167,"context_line":"  present;"},{"line_number":168,"context_line":"* Select a suitable VF at the main board host side and create a resource"},{"line_number":169,"context_line":"  request for it;"},{"line_number":170,"context_line":"* Bind a port to the correct mechanism driver;"},{"line_number":171,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"69b5b96e_499eb3d4","line":168,"range":{"start_line":168,"start_character":30,"end_line":168,"end_character":40},"updated":"2021-05-07 12:08:25.000000000","message":"delete","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1933e7abc61d9926e1fb5cba9bb19aac9ef7ff52","unresolved":true,"context_lines":[{"line_number":165,"context_line":""},{"line_number":166,"context_line":"* Schedule an instance to a host where a VF with the necessary capability is"},{"line_number":167,"context_line":"  present;"},{"line_number":168,"context_line":"* Select a suitable VF at the main board host side and create a resource"},{"line_number":169,"context_line":"  request for it;"},{"line_number":170,"context_line":"* Bind a port to the correct mechanism driver;"},{"line_number":171,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"e055974d_6e5ae3b9","line":168,"range":{"start_line":168,"start_character":30,"end_line":168,"end_character":40},"in_reply_to":"31711e11_38d0e3e9","updated":"2021-05-18 12:45:55.000000000","message":"yes that would be fine with me.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":165,"context_line":""},{"line_number":166,"context_line":"* Schedule an instance to a host where a VF with the necessary capability is"},{"line_number":167,"context_line":"  present;"},{"line_number":168,"context_line":"* Select a suitable VF at the main board host side and create a resource"},{"line_number":169,"context_line":"  request for it;"},{"line_number":170,"context_line":"* Bind a port to the correct mechanism driver;"},{"line_number":171,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"31711e11_38d0e3e9","line":168,"range":{"start_line":168,"start_character":30,"end_line":168,"end_character":40},"in_reply_to":"69b5b96e_499eb3d4","updated":"2021-05-11 13:17:45.000000000","message":"ack, could use the same approach as above though replacing it with \"hypervisor host\".","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":165,"context_line":""},{"line_number":166,"context_line":"* Schedule an instance to a host where a VF with the necessary capability is"},{"line_number":167,"context_line":"  present;"},{"line_number":168,"context_line":"* Select a suitable VF at the main board host side and create a resource"},{"line_number":169,"context_line":"  request for it;"},{"line_number":170,"context_line":"* Bind a port to the correct mechanism driver;"},{"line_number":171,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"598dec1a_25413efb","line":168,"range":{"start_line":168,"start_character":30,"end_line":168,"end_character":40},"in_reply_to":"e055974d_6e5ae3b9","updated":"2021-05-21 19:01:48.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":165,"context_line":""},{"line_number":166,"context_line":"* Schedule an instance to a host where a VF with the necessary capability is"},{"line_number":167,"context_line":"  present;"},{"line_number":168,"context_line":"* Select a suitable VF at the main board host side and create a resource"},{"line_number":169,"context_line":"  request for it;"},{"line_number":170,"context_line":"* Bind a port to the correct mechanism driver;"},{"line_number":171,"context_line":""},{"line_number":172,"context_line":"  * Discover that networking agents run on a transport node (differentiate"}],"source_content_type":"text/x-rst","patch_set":4,"id":"9c640a40_85f9f9dd","line":169,"range":{"start_line":168,"start_character":54,"end_line":169,"end_character":17},"updated":"2021-05-07 12:08:25.000000000","message":"technically the resource request is create prior to scheduling to have the scheduler select a host where the resource request can be be fulfilled.\n\ndo you mean claim the pci_device in the nova database.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1933e7abc61d9926e1fb5cba9bb19aac9ef7ff52","unresolved":true,"context_lines":[{"line_number":165,"context_line":""},{"line_number":166,"context_line":"* Schedule an instance to a host where a VF with the necessary capability is"},{"line_number":167,"context_line":"  present;"},{"line_number":168,"context_line":"* Select a suitable VF at the main board host side and create a resource"},{"line_number":169,"context_line":"  request for it;"},{"line_number":170,"context_line":"* Bind a port to the correct mechanism driver;"},{"line_number":171,"context_line":""},{"line_number":172,"context_line":"  * Discover that networking agents run on a transport node (differentiate"}],"source_content_type":"text/x-rst","patch_set":4,"id":"83f2e7d0_f6739503","line":169,"range":{"start_line":168,"start_character":54,"end_line":169,"end_character":17},"in_reply_to":"565fc711_449c1c56","updated":"2021-05-18 12:45:55.000000000","message":"+1","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":165,"context_line":""},{"line_number":166,"context_line":"* Schedule an instance to a host where a VF with the necessary capability is"},{"line_number":167,"context_line":"  present;"},{"line_number":168,"context_line":"* Select a suitable VF at the main board host side and create a resource"},{"line_number":169,"context_line":"  request for it;"},{"line_number":170,"context_line":"* Bind a port to the correct mechanism driver;"},{"line_number":171,"context_line":""},{"line_number":172,"context_line":"  * Discover that networking agents run on a transport node (differentiate"}],"source_content_type":"text/x-rst","patch_set":4,"id":"8a38620e_68e924ff","line":169,"range":{"start_line":168,"start_character":54,"end_line":169,"end_character":17},"in_reply_to":"83f2e7d0_f6739503","updated":"2021-05-21 19:01:48.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":165,"context_line":""},{"line_number":166,"context_line":"* Schedule an instance to a host where a VF with the necessary capability is"},{"line_number":167,"context_line":"  present;"},{"line_number":168,"context_line":"* Select a suitable VF at the main board host side and create a resource"},{"line_number":169,"context_line":"  request for it;"},{"line_number":170,"context_line":"* Bind a port to the correct mechanism driver;"},{"line_number":171,"context_line":""},{"line_number":172,"context_line":"  * Discover that networking agents run on a transport node (differentiate"}],"source_content_type":"text/x-rst","patch_set":4,"id":"565fc711_449c1c56","line":169,"range":{"start_line":168,"start_character":54,"end_line":169,"end_character":17},"in_reply_to":"9c640a40_85f9f9dd","updated":"2021-05-11 13:17:45.000000000","message":"Yes, you are right.\n\nIt should be something like: \"Select a suitable VF and create a PCI device claim for it\".","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":169,"context_line":"  request for it;"},{"line_number":170,"context_line":"* Bind a port to the correct mechanism driver;"},{"line_number":171,"context_line":""},{"line_number":172,"context_line":"  * Discover that networking agents run on a transport node (differentiate"},{"line_number":173,"context_line":"    with the regular hardware offload use-case);"},{"line_number":174,"context_line":"  * Discover the right transport node hostname with which to associate a port;"},{"line_number":175,"context_line":"    * There may be one or multiple transport nodes per host, not just one;"},{"line_number":176,"context_line":"* Initiate the functionality to plug and program the right representor at"}],"source_content_type":"text/x-rst","patch_set":4,"id":"d885c24a_c7c27f6d","line":173,"range":{"start_line":172,"start_character":2,"end_line":173,"end_character":48},"updated":"2021-05-07 12:08:25.000000000","message":"this is not done in nova.\n\nnova binds a port to a host in nuetorn but never know what the networking backend is.\n\nnova can infer some info form the vif_type but vif_type ovs is used by ml2/ovs, ml2/ovn, ml2/odl and some other SDN contolers","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1933e7abc61d9926e1fb5cba9bb19aac9ef7ff52","unresolved":true,"context_lines":[{"line_number":169,"context_line":"  request for it;"},{"line_number":170,"context_line":"* Bind a port to the correct mechanism driver;"},{"line_number":171,"context_line":""},{"line_number":172,"context_line":"  * Discover that networking agents run on a transport node (differentiate"},{"line_number":173,"context_line":"    with the regular hardware offload use-case);"},{"line_number":174,"context_line":"  * Discover the right transport node hostname with which to associate a port;"},{"line_number":175,"context_line":"    * There may be one or multiple transport nodes per host, not just one;"},{"line_number":176,"context_line":"* Initiate the functionality to plug and program the right representor at"}],"source_content_type":"text/x-rst","patch_set":4,"id":"3e7d2863_7e32a973","line":173,"range":{"start_line":172,"start_character":2,"end_line":173,"end_character":48},"in_reply_to":"2dd7d360_8752cbb8","updated":"2021-05-18 12:45:55.000000000","message":"i think refering to the neutorn spec might help.\nthere is a lot of content in this spec which is good but it will deter people form reviewing.\nif we can condense donwn the info that is most salient to the nova team while still provideing some content it will help. putting the details in the neutron specs for the neutron bits is one way to do that that still enable use to click through if its needed without expanding the spec too much.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":169,"context_line":"  request for it;"},{"line_number":170,"context_line":"* Bind a port to the correct mechanism driver;"},{"line_number":171,"context_line":""},{"line_number":172,"context_line":"  * Discover that networking agents run on a transport node (differentiate"},{"line_number":173,"context_line":"    with the regular hardware offload use-case);"},{"line_number":174,"context_line":"  * Discover the right transport node hostname with which to associate a port;"},{"line_number":175,"context_line":"    * There may be one or multiple transport nodes per host, not just one;"},{"line_number":176,"context_line":"* Initiate the functionality to plug and program the right representor at"}],"source_content_type":"text/x-rst","patch_set":4,"id":"4f9de20a_fd12c245","line":173,"range":{"start_line":172,"start_character":2,"end_line":173,"end_character":48},"in_reply_to":"3e7d2863_7e32a973","updated":"2021-05-21 19:01:48.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":169,"context_line":"  request for it;"},{"line_number":170,"context_line":"* Bind a port to the correct mechanism driver;"},{"line_number":171,"context_line":""},{"line_number":172,"context_line":"  * Discover that networking agents run on a transport node (differentiate"},{"line_number":173,"context_line":"    with the regular hardware offload use-case);"},{"line_number":174,"context_line":"  * Discover the right transport node hostname with which to associate a port;"},{"line_number":175,"context_line":"    * There may be one or multiple transport nodes per host, not just one;"},{"line_number":176,"context_line":"* Initiate the functionality to plug and program the right representor at"}],"source_content_type":"text/x-rst","patch_set":4,"id":"2dd7d360_8752cbb8","line":173,"range":{"start_line":172,"start_character":2,"end_line":173,"end_character":48},"in_reply_to":"d885c24a_c7c27f6d","updated":"2021-05-11 13:17:45.000000000","message":"Yes, I was describing it here as a set of cross-project interactions that need to happen and port binding to a mechanism driver happens at the Neutron side. Now that there is a Neutron spec, I could simply refer to it here to avoid confusion about which part of this is relevant for Nova.\n\nApologies, if the intention was not clear here.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":171,"context_line":""},{"line_number":172,"context_line":"  * Discover that networking agents run on a transport node (differentiate"},{"line_number":173,"context_line":"    with the regular hardware offload use-case);"},{"line_number":174,"context_line":"  * Discover the right transport node hostname with which to associate a port;"},{"line_number":175,"context_line":"    * There may be one or multiple transport nodes per host, not just one;"},{"line_number":176,"context_line":"* Initiate the functionality to plug and program the right representor at"},{"line_number":177,"context_line":"  the transport node side."},{"line_number":178,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"dc3c1a9e_23373649","line":175,"range":{"start_line":174,"start_character":3,"end_line":175,"end_character":74},"updated":"2021-05-07 12:08:25.000000000","message":"again this si part of what neutron does so its not really relevent to the nova specs.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1933e7abc61d9926e1fb5cba9bb19aac9ef7ff52","unresolved":false,"context_lines":[{"line_number":171,"context_line":""},{"line_number":172,"context_line":"  * Discover that networking agents run on a transport node (differentiate"},{"line_number":173,"context_line":"    with the regular hardware offload use-case);"},{"line_number":174,"context_line":"  * Discover the right transport node hostname with which to associate a port;"},{"line_number":175,"context_line":"    * There may be one or multiple transport nodes per host, not just one;"},{"line_number":176,"context_line":"* Initiate the functionality to plug and program the right representor at"},{"line_number":177,"context_line":"  the transport node side."},{"line_number":178,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"e7993999_11dd0335","line":175,"range":{"start_line":174,"start_character":3,"end_line":175,"end_character":74},"in_reply_to":"b77c72d1_11af5104","updated":"2021-05-18 12:45:55.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":171,"context_line":""},{"line_number":172,"context_line":"  * Discover that networking agents run on a transport node (differentiate"},{"line_number":173,"context_line":"    with the regular hardware offload use-case);"},{"line_number":174,"context_line":"  * Discover the right transport node hostname with which to associate a port;"},{"line_number":175,"context_line":"    * There may be one or multiple transport nodes per host, not just one;"},{"line_number":176,"context_line":"* Initiate the functionality to plug and program the right representor at"},{"line_number":177,"context_line":"  the transport node side."},{"line_number":178,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"b77c72d1_11af5104","line":175,"range":{"start_line":174,"start_character":3,"end_line":175,"end_character":74},"in_reply_to":"dc3c1a9e_23373649","updated":"2021-05-11 13:17:45.000000000","message":"Ack, same as above.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":173,"context_line":"    with the regular hardware offload use-case);"},{"line_number":174,"context_line":"  * Discover the right transport node hostname with which to associate a port;"},{"line_number":175,"context_line":"    * There may be one or multiple transport nodes per host, not just one;"},{"line_number":176,"context_line":"* Initiate the functionality to plug and program the right representor at"},{"line_number":177,"context_line":"  the transport node side."},{"line_number":178,"context_line":""},{"line_number":179,"context_line":"The transport node selection in particular becomes an issue to address due to"},{"line_number":180,"context_line":"the following:"}],"source_content_type":"text/x-rst","patch_set":4,"id":"87ab4280_b26c3237","line":177,"range":{"start_line":176,"start_character":2,"end_line":177,"end_character":26},"updated":"2021-05-07 12:08:25.000000000","message":"this will be done by nova using os-vif to plug the interface in the ovs db over tcp?\n\nneutron should not be the entity that plugs interfaces.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":179,"context_line":"The transport node selection in particular becomes an issue to address due to"},{"line_number":180,"context_line":"the following:"},{"line_number":181,"context_line":""},{"line_number":182,"context_line":"* PF and VF mac addresses can be reprogrammed so they cannot be used as"},{"line_number":183,"context_line":"  reliable persistent identifiers for transport nodes;"},{"line_number":184,"context_line":"* PCI(e) add-in cards themselves do not have entries in sysfs but PCI(e)"},{"line_number":185,"context_line":"  endpoints do;"}],"source_content_type":"text/x-rst","patch_set":4,"id":"a6d062e4_302b485a","line":182,"range":{"start_line":182,"start_character":2,"end_line":182,"end_character":4},"updated":"2021-05-07 12:08:25.000000000","message":"actuylly PF macs cannot be set sicne they are reset when detach form the host kernel but we should still not rely on them.\n\nfor vnic_type driect-physical we actully set the neutorn port to the PF mac instead of seeing the PF to the neutron port mac like we normally do.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":179,"context_line":"The transport node selection in particular becomes an issue to address due to"},{"line_number":180,"context_line":"the following:"},{"line_number":181,"context_line":""},{"line_number":182,"context_line":"* PF and VF mac addresses can be reprogrammed so they cannot be used as"},{"line_number":183,"context_line":"  reliable persistent identifiers for transport nodes;"},{"line_number":184,"context_line":"* PCI(e) add-in cards themselves do not have entries in sysfs but PCI(e)"},{"line_number":185,"context_line":"  endpoints do;"}],"source_content_type":"text/x-rst","patch_set":4,"id":"d957f3bc_cdd780c3","line":182,"range":{"start_line":182,"start_character":2,"end_line":182,"end_character":4},"in_reply_to":"9359641a_b2c29524","updated":"2021-05-21 19:01:48.000000000","message":"I agree, we discarded VF MACs from the beginning because of that. PF logical numbers (that we can deduce from PCI function numbers) are relative to controllers in general and we don\u0027t see them from the hypervisor host - that was a very unfortunate conclusion from looking at the actual hardware as using {PF logical number, VF logical number} combination was very compelling.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":179,"context_line":"The transport node selection in particular becomes an issue to address due to"},{"line_number":180,"context_line":"the following:"},{"line_number":181,"context_line":""},{"line_number":182,"context_line":"* PF and VF mac addresses can be reprogrammed so they cannot be used as"},{"line_number":183,"context_line":"  reliable persistent identifiers for transport nodes;"},{"line_number":184,"context_line":"* PCI(e) add-in cards themselves do not have entries in sysfs but PCI(e)"},{"line_number":185,"context_line":"  endpoints do;"}],"source_content_type":"text/x-rst","patch_set":4,"id":"dc2170dc_9b734d4a","line":182,"range":{"start_line":182,"start_character":2,"end_line":182,"end_character":4},"in_reply_to":"a6d062e4_302b485a","updated":"2021-05-11 13:17:45.000000000","message":"What I mean here is that storing a map of PF or VF MAC addresses to SmartNIC hostnames for lookup purposes is not feasible and that a better way is to use a serial number from VPD.\n\nDefault PF and VF MAC addresses can be programmed from the SmartNIC host side,\n\nhttps://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id\u003da1e8ae907c8d67f57432190bb742802a76516b00 (upstream API)\nhttps://github.com/torvalds/linux/blob/v5.12/net/core/devlink.c#L1057-L1089\n\nhttps://docs.mellanox.com/display/BlueFieldSWv21010924/Controlling%20Host%20PF%20and%20VF%20Parameters (Bluefield-specific)\n\nhowever, the hypervisor host can change them (at least in its own kernel) and so they will no longer be usable for lookup purposes anymore.\n\nThe scenario where only VFs are used is a bit different:\n\n* PFs MACs are not touched;\n* VF MACs are.\n\nTherefore, default PF MACs set from the SmartNIC host can be used for consistent lookup of PFs at both sides.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1933e7abc61d9926e1fb5cba9bb19aac9ef7ff52","unresolved":true,"context_lines":[{"line_number":179,"context_line":"The transport node selection in particular becomes an issue to address due to"},{"line_number":180,"context_line":"the following:"},{"line_number":181,"context_line":""},{"line_number":182,"context_line":"* PF and VF mac addresses can be reprogrammed so they cannot be used as"},{"line_number":183,"context_line":"  reliable persistent identifiers for transport nodes;"},{"line_number":184,"context_line":"* PCI(e) add-in cards themselves do not have entries in sysfs but PCI(e)"},{"line_number":185,"context_line":"  endpoints do;"}],"source_content_type":"text/x-rst","patch_set":4,"id":"9359641a_b2c29524","line":182,"range":{"start_line":182,"start_character":2,"end_line":182,"end_character":4},"in_reply_to":"dc2170dc_9b734d4a","updated":"2021-05-18 12:45:55.000000000","message":"they could be yes but we have normally avoid using the mac.\nif the pf default mac is the only thing we can use then that is fine but experince has made us cauious of trusting VF macs in partcalar becasue they often have different behavior for differen vendor.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":180,"context_line":"the following:"},{"line_number":181,"context_line":""},{"line_number":182,"context_line":"* PF and VF mac addresses can be reprogrammed so they cannot be used as"},{"line_number":183,"context_line":"  reliable persistent identifiers for transport nodes;"},{"line_number":184,"context_line":"* PCI(e) add-in cards themselves do not have entries in sysfs but PCI(e)"},{"line_number":185,"context_line":"  endpoints do;"},{"line_number":186,"context_line":"* Hypervisor hosts and transport node hosts do not see the same set of PCIe"}],"source_content_type":"text/x-rst","patch_set":4,"id":"0aebc49c_86bfac6b","line":183,"updated":"2021-05-07 12:08:25.000000000","message":"+1","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":false,"context_lines":[{"line_number":180,"context_line":"the following:"},{"line_number":181,"context_line":""},{"line_number":182,"context_line":"* PF and VF mac addresses can be reprogrammed so they cannot be used as"},{"line_number":183,"context_line":"  reliable persistent identifiers for transport nodes;"},{"line_number":184,"context_line":"* PCI(e) add-in cards themselves do not have entries in sysfs but PCI(e)"},{"line_number":185,"context_line":"  endpoints do;"},{"line_number":186,"context_line":"* Hypervisor hosts and transport node hosts do not see the same set of PCIe"}],"source_content_type":"text/x-rst","patch_set":4,"id":"935d978d_fad464e8","line":183,"in_reply_to":"0aebc49c_86bfac6b","updated":"2021-05-11 13:17:45.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":183,"context_line":"  reliable persistent identifiers for transport nodes;"},{"line_number":184,"context_line":"* PCI(e) add-in cards themselves do not have entries in sysfs but PCI(e)"},{"line_number":185,"context_line":"  endpoints do;"},{"line_number":186,"context_line":"* Hypervisor hosts and transport node hosts do not see the same set of PCIe"},{"line_number":187,"context_line":"  functions as they see **isolated PCIe topologies**. Each host enumerates the"},{"line_number":188,"context_line":"  PCIe topology it is able to observe. While the same NIC is exposed to both"},{"line_number":189,"context_line":"  topologies, the set of functions and config spaces observed by hosts differs."}],"source_content_type":"text/x-rst","patch_set":4,"id":"9f627ec7_bb3f2633","line":186,"range":{"start_line":186,"start_character":2,"end_line":186,"end_character":43},"updated":"2021-05-07 12:08:25.000000000","message":"The host server OS and smartnic OS","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":183,"context_line":"  reliable persistent identifiers for transport nodes;"},{"line_number":184,"context_line":"* PCI(e) add-in cards themselves do not have entries in sysfs but PCI(e)"},{"line_number":185,"context_line":"  endpoints do;"},{"line_number":186,"context_line":"* Hypervisor hosts and transport node hosts do not see the same set of PCIe"},{"line_number":187,"context_line":"  functions as they see **isolated PCIe topologies**. Each host enumerates the"},{"line_number":188,"context_line":"  PCIe topology it is able to observe. While the same NIC is exposed to both"},{"line_number":189,"context_line":"  topologies, the set of functions and config spaces observed by hosts differs."}],"source_content_type":"text/x-rst","patch_set":4,"id":"9a825278_bd3a91e5","line":186,"range":{"start_line":186,"start_character":2,"end_line":186,"end_character":43},"in_reply_to":"5afb6762_681a10d2","updated":"2021-05-21 19:01:48.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":183,"context_line":"  reliable persistent identifiers for transport nodes;"},{"line_number":184,"context_line":"* PCI(e) add-in cards themselves do not have entries in sysfs but PCI(e)"},{"line_number":185,"context_line":"  endpoints do;"},{"line_number":186,"context_line":"* Hypervisor hosts and transport node hosts do not see the same set of PCIe"},{"line_number":187,"context_line":"  functions as they see **isolated PCIe topologies**. Each host enumerates the"},{"line_number":188,"context_line":"  PCIe topology it is able to observe. While the same NIC is exposed to both"},{"line_number":189,"context_line":"  topologies, the set of functions and config spaces observed by hosts differs."}],"source_content_type":"text/x-rst","patch_set":4,"id":"5afb6762_681a10d2","line":186,"range":{"start_line":186,"start_character":2,"end_line":186,"end_character":43},"in_reply_to":"9f627ec7_bb3f2633","updated":"2021-05-11 13:17:45.000000000","message":"I think \"hypervisor host\" is explicit enough - i.e. \"the host where Nova runs\".\n\nI can change it if you prefer it that way but I find the term \"hypervisor host\" less ambiguous.\n\nWill change the \"transport node host\" as you suggest.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":184,"context_line":"* PCI(e) add-in cards themselves do not have entries in sysfs but PCI(e)"},{"line_number":185,"context_line":"  endpoints do;"},{"line_number":186,"context_line":"* Hypervisor hosts and transport node hosts do not see the same set of PCIe"},{"line_number":187,"context_line":"  functions as they see **isolated PCIe topologies**. Each host enumerates the"},{"line_number":188,"context_line":"  PCIe topology it is able to observe. While the same NIC is exposed to both"},{"line_number":189,"context_line":"  topologies, the set of functions and config spaces observed by hosts differs."},{"line_number":190,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"07ba97ca_10ea4685","line":187,"range":{"start_line":187,"start_character":24,"end_line":187,"end_character":53},"updated":"2021-05-07 12:08:25.000000000","message":"that actully is assuming there is even a pci toplogy connected to the cpu cores on the smartnic.\n\nthe last time i worked with a smartnic that ran linux on cores with ovs it did not have a pcie bus connect to the arm cpus, it comunicated over a differnt bus.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":184,"context_line":"* PCI(e) add-in cards themselves do not have entries in sysfs but PCI(e)"},{"line_number":185,"context_line":"  endpoints do;"},{"line_number":186,"context_line":"* Hypervisor hosts and transport node hosts do not see the same set of PCIe"},{"line_number":187,"context_line":"  functions as they see **isolated PCIe topologies**. Each host enumerates the"},{"line_number":188,"context_line":"  PCIe topology it is able to observe. While the same NIC is exposed to both"},{"line_number":189,"context_line":"  topologies, the set of functions and config spaces observed by hosts differs."},{"line_number":190,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"83a9a465_b18041db","line":187,"range":{"start_line":187,"start_character":24,"end_line":187,"end_character":53},"in_reply_to":"07ba97ca_10ea4685","updated":"2021-05-11 13:17:45.000000000","message":"This is a good point. Even if the hypervisor host can see PCIe endpoints it does not mean that the SmartNIC host will talk to PCIe endpoints at its side.\n\nEven the switchdev docs use PCIe as an example (not as a mandatory implementation detail): https://www.kernel.org/doc/Documentation/networking/switchdev.txt\n\nIt could well be platform devices or some other IO bus.\n\nMy aim here was to illustrate that if PCIe is used at both sides, those topologies are isolated and that relying on PCI addresses is not an option.\n\nSome SmartNIC DPU examples where a PCIe topology is present (for reference, ARM host side):\nhttps://paste.ubuntu.com/p/QT98R7ZMzp/ (Bluefield, Stringray)\nhttps://doc.dpdk.org/guides/cryptodevs/octeontx2.html#initialization (Octeon TX2)\n\n\nI can rephrase it such that it is clear that it may be the case that PCIe is used at both sides and, if so, PCI addresses should not be relied upon.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":184,"context_line":"* PCI(e) add-in cards themselves do not have entries in sysfs but PCI(e)"},{"line_number":185,"context_line":"  endpoints do;"},{"line_number":186,"context_line":"* Hypervisor hosts and transport node hosts do not see the same set of PCIe"},{"line_number":187,"context_line":"  functions as they see **isolated PCIe topologies**. Each host enumerates the"},{"line_number":188,"context_line":"  PCIe topology it is able to observe. While the same NIC is exposed to both"},{"line_number":189,"context_line":"  topologies, the set of functions and config spaces observed by hosts differs."},{"line_number":190,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"d01ee005_0ca758f2","line":187,"range":{"start_line":187,"start_character":24,"end_line":187,"end_character":53},"in_reply_to":"614857ed_fea34855","updated":"2021-05-21 19:01:48.000000000","message":"Done","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1933e7abc61d9926e1fb5cba9bb19aac9ef7ff52","unresolved":true,"context_lines":[{"line_number":184,"context_line":"* PCI(e) add-in cards themselves do not have entries in sysfs but PCI(e)"},{"line_number":185,"context_line":"  endpoints do;"},{"line_number":186,"context_line":"* Hypervisor hosts and transport node hosts do not see the same set of PCIe"},{"line_number":187,"context_line":"  functions as they see **isolated PCIe topologies**. Each host enumerates the"},{"line_number":188,"context_line":"  PCIe topology it is able to observe. While the same NIC is exposed to both"},{"line_number":189,"context_line":"  topologies, the set of functions and config spaces observed by hosts differs."},{"line_number":190,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"614857ed_fea34855","line":187,"range":{"start_line":187,"start_character":24,"end_line":187,"end_character":53},"in_reply_to":"83a9a465_b18041db","updated":"2021-05-18 12:45:55.000000000","message":"yes i would just make the statement stronger that even if both sides can see the same pci address which is unlikely we still cannot rely on that since in general it would not be true.\n\nand yes i have seen smartnics that use pcie too. the spefcic one i was dealing with did not but that was a requslt of the soc that was being used as a prototype not supporting it and pcie on arm just beeing less common then it is on other plathforms. the idea of both sharing a pcie bus is not in question :)","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":186,"context_line":"* Hypervisor hosts and transport node hosts do not see the same set of PCIe"},{"line_number":187,"context_line":"  functions as they see **isolated PCIe topologies**. Each host enumerates the"},{"line_number":188,"context_line":"  PCIe topology it is able to observe. While the same NIC is exposed to both"},{"line_number":189,"context_line":"  topologies, the set of functions and config spaces observed by hosts differs."},{"line_number":190,"context_line":""},{"line_number":191,"context_line":"In order to track transport nodes and associations of PFs/VFs with them, there"},{"line_number":192,"context_line":"is a need for a unique and persistent identifier that is discoverable from both"}],"source_content_type":"text/x-rst","patch_set":4,"id":"4167f329_11f7ebe3","line":189,"updated":"2021-05-07 12:08:25.000000000","message":"yes where it exits we cannot assume its the same.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":false,"context_lines":[{"line_number":186,"context_line":"* Hypervisor hosts and transport node hosts do not see the same set of PCIe"},{"line_number":187,"context_line":"  functions as they see **isolated PCIe topologies**. Each host enumerates the"},{"line_number":188,"context_line":"  PCIe topology it is able to observe. While the same NIC is exposed to both"},{"line_number":189,"context_line":"  topologies, the set of functions and config spaces observed by hosts differs."},{"line_number":190,"context_line":""},{"line_number":191,"context_line":"In order to track transport nodes and associations of PFs/VFs with them, there"},{"line_number":192,"context_line":"is a need for a unique and persistent identifier that is discoverable from both"}],"source_content_type":"text/x-rst","patch_set":4,"id":"26cc164f_bc20b72e","line":189,"in_reply_to":"4167f329_11f7ebe3","updated":"2021-05-11 13:17:45.000000000","message":"Ack, the benefit we get though is that the same VPD is exposed by PCIe functions on different PCIe hierarchies.\n\nThe aim is to use devlink first to try to get a broad_serial using devlink-info (that would make it possible to work with different buses, not just PCIe) and then try to use PCIe VPD directly if that\u0027s not available and PCIe is used at both sides.\n\nhttps://www.kernel.org/doc/html/latest/networking/devlink/devlink-info.html#devlink-info","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":204,"context_line":"devlink-info - there are no ties to a particular IO standard such as PCI(e) so"},{"line_number":205,"context_line":"other types of devices (e.g. platform devices) could leverage this as well."},{"line_number":206,"context_line":""},{"line_number":207,"context_line":"For the PCI(e) use-case specifically, there is a need to distinguish the"},{"line_number":208,"context_line":"PFs/VFs that simply expose a VPD from the ones that also need to be associated"},{"line_number":209,"context_line":"with transport nodes which calls for an approach similar to the one provided"},{"line_number":210,"context_line":"with the ``pci_passthrough_whitelist`` to avoid hard-coding a database of"},{"line_number":211,"context_line":"devices that need to be treated as transport nodes."},{"line_number":212,"context_line":""},{"line_number":213,"context_line":"Transport nodes are also a form of a resource provider since eSwitch hardware"},{"line_number":214,"context_line":"resources are limited:"}],"source_content_type":"text/x-rst","patch_set":4,"id":"3802db27_3ac0781c","line":211,"range":{"start_line":207,"start_character":0,"end_line":211,"end_character":51},"updated":"2021-05-07 12:08:25.000000000","message":"i don\u0027t think this is required.\n\nwe should just store the VPD info for all pci device that are listed in the pci_passthrough_whitelist\n\nwe do not need to extend it or add any other filtering or modify the exciting pci_passthrough_whitelist mechanism.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":204,"context_line":"devlink-info - there are no ties to a particular IO standard such as PCI(e) so"},{"line_number":205,"context_line":"other types of devices (e.g. platform devices) could leverage this as well."},{"line_number":206,"context_line":""},{"line_number":207,"context_line":"For the PCI(e) use-case specifically, there is a need to distinguish the"},{"line_number":208,"context_line":"PFs/VFs that simply expose a VPD from the ones that also need to be associated"},{"line_number":209,"context_line":"with transport nodes which calls for an approach similar to the one provided"},{"line_number":210,"context_line":"with the ``pci_passthrough_whitelist`` to avoid hard-coding a database of"},{"line_number":211,"context_line":"devices that need to be treated as transport nodes."},{"line_number":212,"context_line":""},{"line_number":213,"context_line":"Transport nodes are also a form of a resource provider since eSwitch hardware"},{"line_number":214,"context_line":"resources are limited:"}],"source_content_type":"text/x-rst","patch_set":4,"id":"f123b814_2ba0bd95","line":211,"range":{"start_line":207,"start_character":0,"end_line":211,"end_character":51},"in_reply_to":"3802db27_3ac0781c","updated":"2021-05-11 13:17:45.000000000","message":"Could gather VPD info for all devices.\n\nThe aim here was to multiplex between the hardware offload case with NICs like ConnectX 5/6 and the SmartNIC DPU case (e.g. Bluefield).\n\nI can rephrase this particular paragraph but we need to address the multiplexing between different hardware architectures in a different place in this spec then.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":204,"context_line":"devlink-info - there are no ties to a particular IO standard such as PCI(e) so"},{"line_number":205,"context_line":"other types of devices (e.g. platform devices) could leverage this as well."},{"line_number":206,"context_line":""},{"line_number":207,"context_line":"For the PCI(e) use-case specifically, there is a need to distinguish the"},{"line_number":208,"context_line":"PFs/VFs that simply expose a VPD from the ones that also need to be associated"},{"line_number":209,"context_line":"with transport nodes which calls for an approach similar to the one provided"},{"line_number":210,"context_line":"with the ``pci_passthrough_whitelist`` to avoid hard-coding a database of"},{"line_number":211,"context_line":"devices that need to be treated as transport nodes."},{"line_number":212,"context_line":""},{"line_number":213,"context_line":"Transport nodes are also a form of a resource provider since eSwitch hardware"},{"line_number":214,"context_line":"resources are limited:"}],"source_content_type":"text/x-rst","patch_set":4,"id":"ae1d9341_d4fa30a3","line":211,"range":{"start_line":207,"start_character":0,"end_line":211,"end_character":51},"in_reply_to":"f123b814_2ba0bd95","updated":"2021-05-21 19:01:48.000000000","message":"Done","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":210,"context_line":"with the ``pci_passthrough_whitelist`` to avoid hard-coding a database of"},{"line_number":211,"context_line":"devices that need to be treated as transport nodes."},{"line_number":212,"context_line":""},{"line_number":213,"context_line":"Transport nodes are also a form of a resource provider since eSwitch hardware"},{"line_number":214,"context_line":"resources are limited:"},{"line_number":215,"context_line":""},{"line_number":216,"context_line":"* the number of functions that can be exposed (PFs, VFs);"},{"line_number":217,"context_line":"* the maximum number of flows that can be programmed per function;"}],"source_content_type":"text/x-rst","patch_set":4,"id":"15e72fdc_65ff9447","line":214,"range":{"start_line":213,"start_character":0,"end_line":214,"end_character":22},"updated":"2021-05-07 12:08:25.000000000","message":"i don\u0027t think we shoudl be modelling transport nodes as new RPs.\n\nwe shoudl just model them using the existing neutron RPs if we need to","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":false,"context_lines":[{"line_number":210,"context_line":"with the ``pci_passthrough_whitelist`` to avoid hard-coding a database of"},{"line_number":211,"context_line":"devices that need to be treated as transport nodes."},{"line_number":212,"context_line":""},{"line_number":213,"context_line":"Transport nodes are also a form of a resource provider since eSwitch hardware"},{"line_number":214,"context_line":"resources are limited:"},{"line_number":215,"context_line":""},{"line_number":216,"context_line":"* the number of functions that can be exposed (PFs, VFs);"},{"line_number":217,"context_line":"* the maximum number of flows that can be programmed per function;"}],"source_content_type":"text/x-rst","patch_set":4,"id":"ff392648_f8c33567","line":214,"range":{"start_line":213,"start_character":0,"end_line":214,"end_character":22},"in_reply_to":"15e72fdc_65ff9447","updated":"2021-05-11 13:17:45.000000000","message":"OK, I think this paragraph can be dropped as a whole since we agreed to avoid introducing new RPs for the purposes of this spec.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":213,"context_line":"Transport nodes are also a form of a resource provider since eSwitch hardware"},{"line_number":214,"context_line":"resources are limited:"},{"line_number":215,"context_line":""},{"line_number":216,"context_line":"* the number of functions that can be exposed (PFs, VFs);"},{"line_number":217,"context_line":"* the maximum number of flows that can be programmed per function;"},{"line_number":218,"context_line":"* the maximum bandwidth of uplink ports."},{"line_number":219,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"d5ca3989_b150d6e4","line":216,"range":{"start_line":216,"start_character":0,"end_line":216,"end_character":57},"updated":"2021-05-07 12:08:25.000000000","message":"this will be modeled by nova generically eventually so we shoudl not expose this infomation in placment as part of this spec","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":false,"context_lines":[{"line_number":213,"context_line":"Transport nodes are also a form of a resource provider since eSwitch hardware"},{"line_number":214,"context_line":"resources are limited:"},{"line_number":215,"context_line":""},{"line_number":216,"context_line":"* the number of functions that can be exposed (PFs, VFs);"},{"line_number":217,"context_line":"* the maximum number of flows that can be programmed per function;"},{"line_number":218,"context_line":"* the maximum bandwidth of uplink ports."},{"line_number":219,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"e909caf2_28154185","line":216,"range":{"start_line":216,"start_character":0,"end_line":216,"end_character":57},"in_reply_to":"d5ca3989_b150d6e4","updated":"2021-05-11 13:17:45.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":214,"context_line":"resources are limited:"},{"line_number":215,"context_line":""},{"line_number":216,"context_line":"* the number of functions that can be exposed (PFs, VFs);"},{"line_number":217,"context_line":"* the maximum number of flows that can be programmed per function;"},{"line_number":218,"context_line":"* the maximum bandwidth of uplink ports."},{"line_number":219,"context_line":""},{"line_number":220,"context_line":"Since IOV sharing can be supported by a hardware platform, those resources"}],"source_content_type":"text/x-rst","patch_set":4,"id":"ec107eeb_2c78bb81","line":217,"range":{"start_line":217,"start_character":2,"end_line":217,"end_character":66},"updated":"2021-05-07 12:08:25.000000000","message":"i dont think this will be a consumable resouce as that depend on the the traffic that will be sent by the guest and is dynmic","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":false,"context_lines":[{"line_number":214,"context_line":"resources are limited:"},{"line_number":215,"context_line":""},{"line_number":216,"context_line":"* the number of functions that can be exposed (PFs, VFs);"},{"line_number":217,"context_line":"* the maximum number of flows that can be programmed per function;"},{"line_number":218,"context_line":"* the maximum bandwidth of uplink ports."},{"line_number":219,"context_line":""},{"line_number":220,"context_line":"Since IOV sharing can be supported by a hardware platform, those resources"}],"source_content_type":"text/x-rst","patch_set":4,"id":"2f06ba1d_4a248b18","line":217,"range":{"start_line":217,"start_character":2,"end_line":217,"end_character":66},"in_reply_to":"ec107eeb_2c78bb81","updated":"2021-05-11 13:17:45.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":215,"context_line":""},{"line_number":216,"context_line":"* the number of functions that can be exposed (PFs, VFs);"},{"line_number":217,"context_line":"* the maximum number of flows that can be programmed per function;"},{"line_number":218,"context_line":"* the maximum bandwidth of uplink ports."},{"line_number":219,"context_line":""},{"line_number":220,"context_line":"Since IOV sharing can be supported by a hardware platform, those resources"},{"line_number":221,"context_line":"can also be shared across Nova hypervisors which has an effect on how resource"}],"source_content_type":"text/x-rst","patch_set":4,"id":"20b17dec_8a54d6d5","line":218,"range":{"start_line":218,"start_character":2,"end_line":218,"end_character":40},"updated":"2021-05-07 12:08:25.000000000","message":"this should be modelled using the existing guaranteed minimum bandwidth feature.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":false,"context_lines":[{"line_number":215,"context_line":""},{"line_number":216,"context_line":"* the number of functions that can be exposed (PFs, VFs);"},{"line_number":217,"context_line":"* the maximum number of flows that can be programmed per function;"},{"line_number":218,"context_line":"* the maximum bandwidth of uplink ports."},{"line_number":219,"context_line":""},{"line_number":220,"context_line":"Since IOV sharing can be supported by a hardware platform, those resources"},{"line_number":221,"context_line":"can also be shared across Nova hypervisors which has an effect on how resource"}],"source_content_type":"text/x-rst","patch_set":4,"id":"2c771929_dc33c6a0","line":218,"range":{"start_line":218,"start_character":2,"end_line":218,"end_character":40},"in_reply_to":"20b17dec_8a54d6d5","updated":"2021-05-11 13:17:45.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":221,"context_line":"can also be shared across Nova hypervisors which has an effect on how resource"},{"line_number":222,"context_line":"providers need to be tracked. This particular problem was excluded from this"},{"line_number":223,"context_line":"specification after the initial specification review."},{"line_number":224,"context_line":""},{"line_number":225,"context_line":"From the scheduling point of view, the reliance on the \"switchdev\" capability"},{"line_number":226,"context_line":"(persisted into the extra_info column of pci_devices table) is also problematic"},{"line_number":227,"context_line":"since the PFs exposed to a hypervisor host by the NIC on a SmartNIC/DPU board"}],"source_content_type":"text/x-rst","patch_set":4,"id":"e777a7e8_d71f0a3b","line":224,"updated":"2021-05-07 12:08:25.000000000","message":"+1 multi host smartnic i think should not be in scope of nova to support directly.\nif that is to be enabeld in the futrue it shoudl be done via cyborg leveraging\nhttps://review.opendev.org/c/openstack/project-config/+/787523/2","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":false,"context_lines":[{"line_number":221,"context_line":"can also be shared across Nova hypervisors which has an effect on how resource"},{"line_number":222,"context_line":"providers need to be tracked. This particular problem was excluded from this"},{"line_number":223,"context_line":"specification after the initial specification review."},{"line_number":224,"context_line":""},{"line_number":225,"context_line":"From the scheduling point of view, the reliance on the \"switchdev\" capability"},{"line_number":226,"context_line":"(persisted into the extra_info column of pci_devices table) is also problematic"},{"line_number":227,"context_line":"since the PFs exposed to a hypervisor host by the NIC on a SmartNIC/DPU board"}],"source_content_type":"text/x-rst","patch_set":4,"id":"502e017d_3887e8e1","line":224,"in_reply_to":"e777a7e8_d71f0a3b","updated":"2021-05-11 13:17:45.000000000","message":"Ack, let\u0027s focus on other things first.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":222,"context_line":"providers need to be tracked. This particular problem was excluded from this"},{"line_number":223,"context_line":"specification after the initial specification review."},{"line_number":224,"context_line":""},{"line_number":225,"context_line":"From the scheduling point of view, the reliance on the \"switchdev\" capability"},{"line_number":226,"context_line":"(persisted into the extra_info column of pci_devices table) is also problematic"},{"line_number":227,"context_line":"since the PFs exposed to a hypervisor host by the NIC on a SmartNIC/DPU board"},{"line_number":228,"context_line":"do not provide access to the eSwitch - it is not possible to query whether the"},{"line_number":229,"context_line":"eSwitch is in the \"legacy\" or \"switchdev\" mode from the hypervisor side. This"},{"line_number":230,"context_line":"has to do with NIC internals and the way the same NIC is exposed to PCIe"},{"line_number":231,"context_line":"hierarchies of the main board CPUs and the \"application CPU\" on the add-in"},{"line_number":232,"context_line":"card. Devlink documentation in the kernel provides an example of that: [9]_"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7bf78d5b_c4cd73b5","line":229,"range":{"start_line":225,"start_character":0,"end_line":229,"end_character":72},"updated":"2021-05-07 12:08:25.000000000","message":"well since we do not support PF wiht hardware offloads today im not sure this is a concern\nwe should scope this effort to only VFs\n\nin terms of querying if a smartnic is in switchdev mode or not i think we have to intoduce a requriement that the nic is reported as swtichdev capable to close this security bug  https://bugs.launchpad.net/nova/+bug/1915282 we dont need to assert if its in swtichdev mode but to be safe we should assert its swtichdev capable.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1933e7abc61d9926e1fb5cba9bb19aac9ef7ff52","unresolved":true,"context_lines":[{"line_number":222,"context_line":"providers need to be tracked. This particular problem was excluded from this"},{"line_number":223,"context_line":"specification after the initial specification review."},{"line_number":224,"context_line":""},{"line_number":225,"context_line":"From the scheduling point of view, the reliance on the \"switchdev\" capability"},{"line_number":226,"context_line":"(persisted into the extra_info column of pci_devices table) is also problematic"},{"line_number":227,"context_line":"since the PFs exposed to a hypervisor host by the NIC on a SmartNIC/DPU board"},{"line_number":228,"context_line":"do not provide access to the eSwitch - it is not possible to query whether the"},{"line_number":229,"context_line":"eSwitch is in the \"legacy\" or \"switchdev\" mode from the hypervisor side. This"},{"line_number":230,"context_line":"has to do with NIC internals and the way the same NIC is exposed to PCIe"},{"line_number":231,"context_line":"hierarchies of the main board CPUs and the \"application CPU\" on the add-in"},{"line_number":232,"context_line":"card. Devlink documentation in the kernel provides an example of that: [9]_"}],"source_content_type":"text/x-rst","patch_set":4,"id":"ce462f2d_969a1000","line":229,"range":{"start_line":225,"start_character":0,"end_line":229,"end_character":72},"in_reply_to":"7a35f172_c62800d1","updated":"2021-05-18 12:45:55.000000000","message":"thats good to see. i have some connect x 6 nics but i have not currently got a deployment of openstack on those host so i did not have a way to check quickly what we put in the db.\n\n\nopenstack port create --network private --vnic-type\u003ddirect --binding-profile \u0027{\"capabilities\": [\"switchdev\"]}\u0027 direct_port2\n\nwill not have any effect today.\nwe do not inclue the capabilities request in the lookup and i dont think we actully want to proceed with supporting that. intended to support that in the past but we did not complete the implementation because we decided to wait for nested resource providers in placement.\nso that part of hardware offloads was not implemented as ferenced in \nhttps://docs.openstack.org/neutron/latest/admin/config-ovs-offload.html#validate-open-vswitch-hardware-offloading\n\nthe binding profile is actully admin only also so normal users cannot set values in it.\n\nif we need to add a flag directly to the VF for the case where the eswitch is only exposed to the i think we either want to have a new vnic_type for this case or we want to use a generic traits request.\n\ni have a work in progress draft of my generic pci devices in placement spec published for early review\nhttps://review.opendev.org/c/openstack/nova-specs/+/791047\n\nas part of that im suggesting adding a generic placement traits tag to the pci whitelist\n\nhttps://review.opendev.org/c/openstack/nova-specs/+/791047/1/specs/xena/approved/pci-device-tracking-in-placement.rst#118\n\nwe could allow neutron port to have traits requests too like im proposing for the pci alias.\nthat can be part of the port_resouce_requests featrue. if we did that then all we would have to do is\n\nopenstack port create --network private --vnic-type\u003ddirect --traits { required:[\u0027DPU\u0027], forbiden:[]}\n\nif we had new vnic types the ml2 driver could just automatically populate the traits request.\n\ni think that would be a clean apparoch in the long run since it would allow the existing hardware offload to ask for \"switchdev\" traits. in the future this would also allow other aspect of the nic to be modeled abstratly like \"ipsec_offload\" or \"physnet1\"","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":222,"context_line":"providers need to be tracked. This particular problem was excluded from this"},{"line_number":223,"context_line":"specification after the initial specification review."},{"line_number":224,"context_line":""},{"line_number":225,"context_line":"From the scheduling point of view, the reliance on the \"switchdev\" capability"},{"line_number":226,"context_line":"(persisted into the extra_info column of pci_devices table) is also problematic"},{"line_number":227,"context_line":"since the PFs exposed to a hypervisor host by the NIC on a SmartNIC/DPU board"},{"line_number":228,"context_line":"do not provide access to the eSwitch - it is not possible to query whether the"},{"line_number":229,"context_line":"eSwitch is in the \"legacy\" or \"switchdev\" mode from the hypervisor side. This"},{"line_number":230,"context_line":"has to do with NIC internals and the way the same NIC is exposed to PCIe"},{"line_number":231,"context_line":"hierarchies of the main board CPUs and the \"application CPU\" on the add-in"},{"line_number":232,"context_line":"card. Devlink documentation in the kernel provides an example of that: [9]_"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a35f172_c62800d1","line":229,"range":{"start_line":225,"start_character":0,"end_line":229,"end_character":72},"in_reply_to":"7bf78d5b_c4cd73b5","updated":"2021-05-11 13:17:45.000000000","message":"\u003e we have to intoduce a requriement that the nic is reported as swtichdev capable \n\nDo you mean at the kernel driver level or something else?\n\n\u003e we should scope this effort to only VFs\n\nThis was written mostly with VFs in mind. PFs are relied on in the non-SmartNIC DPU case to retrieve the \"switchdev\" capability that will be associated with its VFs as well.\n\nThis is then used to specify a requirement for this capability in the binding:profile attribute:\nhttps://docs.openstack.org/neutron/latest/admin/config-ovs-offload.html#validate-open-vswitch-hardware-offloading\n# openstack port create --network private --vnic-type\u003ddirect --binding-profile \u0027{\"capabilities\": [\"switchdev\"]}\u0027 direct_port2\n\n\nHere is an example of a deployment state:\nhttps://paste.ubuntu.com/p/yMmdjtMQQv/ (database table dumps for a deployment with OVS hardware offload. Note that the \"switchdev\" capability is present on both PFs and VFs)\n\n\nThe problem is that the NIC Switch is not exposed to the hypervisor host in the SmartNIC DPU case and for that we need to flag VFs explicitly with a certain capability to make the mechanism above work in the same way as with hardware offload (a different capability name can be used for multiplexing).","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":222,"context_line":"providers need to be tracked. This particular problem was excluded from this"},{"line_number":223,"context_line":"specification after the initial specification review."},{"line_number":224,"context_line":""},{"line_number":225,"context_line":"From the scheduling point of view, the reliance on the \"switchdev\" capability"},{"line_number":226,"context_line":"(persisted into the extra_info column of pci_devices table) is also problematic"},{"line_number":227,"context_line":"since the PFs exposed to a hypervisor host by the NIC on a SmartNIC/DPU board"},{"line_number":228,"context_line":"do not provide access to the eSwitch - it is not possible to query whether the"},{"line_number":229,"context_line":"eSwitch is in the \"legacy\" or \"switchdev\" mode from the hypervisor side. This"},{"line_number":230,"context_line":"has to do with NIC internals and the way the same NIC is exposed to PCIe"},{"line_number":231,"context_line":"hierarchies of the main board CPUs and the \"application CPU\" on the add-in"},{"line_number":232,"context_line":"card. Devlink documentation in the kernel provides an example of that: [9]_"}],"source_content_type":"text/x-rst","patch_set":4,"id":"87734505_683d64c1","line":229,"range":{"start_line":225,"start_character":0,"end_line":229,"end_character":72},"in_reply_to":"ce462f2d_969a1000","updated":"2021-05-21 19:01:48.000000000","message":"I like the generic traits direction.\n\nFor now, I will change the spec to refer to an already existing VNIC type (VNIC_SMARTNIC), however, it would be good to implement this with a solution that is good in the long term. Depending on timing, maybe that\u0027s what will end up being used in the end - happy to discuss ways to approach it further.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":235,"context_line":""},{"line_number":236,"context_line":"* Managed PCIe switches (MR-SR-IOV) present on an add-in card with synthetic"},{"line_number":237,"context_line":"  sub-hierarchies presented to the main board CPUs and the application CPU"},{"line_number":238,"context_line":"  [8]_, [9]_;"},{"line_number":239,"context_line":"* Multiple PCIe controllers per NIC with functions exposed to different"},{"line_number":240,"context_line":"  topologies [10]_."},{"line_number":241,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"2ef66e39_01c357cc","line":238,"range":{"start_line":238,"start_character":2,"end_line":238,"end_character":7},"updated":"2021-05-07 12:08:25.000000000","message":"i think its a bad idea to use patents as references.\n\ni would remove the entire possible hardware implementation alternatives include:\"\nsection and the associated references form this spec.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":false,"context_lines":[{"line_number":235,"context_line":""},{"line_number":236,"context_line":"* Managed PCIe switches (MR-SR-IOV) present on an add-in card with synthetic"},{"line_number":237,"context_line":"  sub-hierarchies presented to the main board CPUs and the application CPU"},{"line_number":238,"context_line":"  [8]_, [9]_;"},{"line_number":239,"context_line":"* Multiple PCIe controllers per NIC with functions exposed to different"},{"line_number":240,"context_line":"  topologies [10]_."},{"line_number":241,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"b265a30a_da1ee4f9","line":238,"range":{"start_line":238,"start_character":2,"end_line":238,"end_character":7},"in_reply_to":"2ef66e39_01c357cc","updated":"2021-05-11 13:17:45.000000000","message":"OK, will do.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":244,"context_line":""},{"line_number":245,"context_line":"The main use-case is the support for programming flows into **off-path**"},{"line_number":246,"context_line":"SmartNICs/DPUs in a manner similar to the hardware offload scenarios currently"},{"line_number":247,"context_line":"supported in conjunction with ML2 OVS and OVN mechanism drivers. The following"},{"line_number":248,"context_line":"points summarize the desired outcome:"},{"line_number":249,"context_line":""},{"line_number":250,"context_line":"* **Off-path** SmartNICs/DPUs from various vendors;"},{"line_number":251,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"481b32ef_0658c988","line":248,"range":{"start_line":247,"start_character":65,"end_line":248,"end_character":36},"updated":"2021-05-11 13:17:45.000000000","message":"Will move this to a separate section since this describes the desired outcome rather than a distinct use-case.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":244,"context_line":""},{"line_number":245,"context_line":"The main use-case is the support for programming flows into **off-path**"},{"line_number":246,"context_line":"SmartNICs/DPUs in a manner similar to the hardware offload scenarios currently"},{"line_number":247,"context_line":"supported in conjunction with ML2 OVS and OVN mechanism drivers. The following"},{"line_number":248,"context_line":"points summarize the desired outcome:"},{"line_number":249,"context_line":""},{"line_number":250,"context_line":"* **Off-path** SmartNICs/DPUs from various vendors;"},{"line_number":251,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"2a5b18fc_266e41af","line":248,"range":{"start_line":247,"start_character":65,"end_line":248,"end_character":36},"in_reply_to":"481b32ef_0658c988","updated":"2021-05-21 19:01:48.000000000","message":"Done","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":249,"context_line":""},{"line_number":250,"context_line":"* **Off-path** SmartNICs/DPUs from various vendors;"},{"line_number":251,"context_line":""},{"line_number":252,"context_line":"  * **On-path** FPGA-based SmartNICs are intentionally not considered because"},{"line_number":253,"context_line":"    of a different hardware architecture;"},{"line_number":254,"context_line":"* \"direct\" vnic type;"},{"line_number":255,"context_line":"* A new capability called \"transportnode\" used during scheduling (via a"},{"line_number":256,"context_line":"  binding:profile port attribute containing this capability);"}],"source_content_type":"text/x-rst","patch_set":4,"id":"10a50042_aa28692f","line":253,"range":{"start_line":252,"start_character":16,"end_line":253,"end_character":40},"updated":"2021-05-07 12:08:25.000000000","message":"use of an FPGA has nothing to do with if the contol plane is executed on or of the Smartnic.\n\nthat is why i find your current definiton of on vs off path unhelpful.\n\nthe smart nics i have worked with that implement this architecture used an asic to provide the base functionality with an FPGA that can probably be enable inline or in a look aside mode and had arm core where linux and ovs ran.\n\nThe presence of an fpga or not in the data path has noting to do with if the control plane executes on the smartnic or not.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":249,"context_line":""},{"line_number":250,"context_line":"* **Off-path** SmartNICs/DPUs from various vendors;"},{"line_number":251,"context_line":""},{"line_number":252,"context_line":"  * **On-path** FPGA-based SmartNICs are intentionally not considered because"},{"line_number":253,"context_line":"    of a different hardware architecture;"},{"line_number":254,"context_line":"* \"direct\" vnic type;"},{"line_number":255,"context_line":"* A new capability called \"transportnode\" used during scheduling (via a"},{"line_number":256,"context_line":"  binding:profile port attribute containing this capability);"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5a3128b8_95025c83","line":253,"range":{"start_line":252,"start_character":16,"end_line":253,"end_character":40},"in_reply_to":"10a50042_aa28692f","updated":"2021-05-11 13:17:45.000000000","message":"Agreed that the reference to FPGA should be dropped here.\n\nI will rephrase this to focus more on SmartNIC DPUs as a whole (to note that an app CPU needs to be present) and on the off-path architecture and presence of a switchdev implementation for a particular NIC driver initially.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1933e7abc61d9926e1fb5cba9bb19aac9ef7ff52","unresolved":false,"context_lines":[{"line_number":249,"context_line":""},{"line_number":250,"context_line":"* **Off-path** SmartNICs/DPUs from various vendors;"},{"line_number":251,"context_line":""},{"line_number":252,"context_line":"  * **On-path** FPGA-based SmartNICs are intentionally not considered because"},{"line_number":253,"context_line":"    of a different hardware architecture;"},{"line_number":254,"context_line":"* \"direct\" vnic type;"},{"line_number":255,"context_line":"* A new capability called \"transportnode\" used during scheduling (via a"},{"line_number":256,"context_line":"  binding:profile port attribute containing this capability);"}],"source_content_type":"text/x-rst","patch_set":4,"id":"11e28256_5e4bf26c","line":253,"range":{"start_line":252,"start_character":16,"end_line":253,"end_character":40},"in_reply_to":"5a3128b8_95025c83","updated":"2021-05-18 12:45:55.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":252,"context_line":"  * **On-path** FPGA-based SmartNICs are intentionally not considered because"},{"line_number":253,"context_line":"    of a different hardware architecture;"},{"line_number":254,"context_line":"* \"direct\" vnic type;"},{"line_number":255,"context_line":"* A new capability called \"transportnode\" used during scheduling (via a"},{"line_number":256,"context_line":"  binding:profile port attribute containing this capability);"},{"line_number":257,"context_line":""},{"line_number":258,"context_line":"  * The \"switchdev\" capability cannot be reused since a hypervisor does not"}],"source_content_type":"text/x-rst","patch_set":4,"id":"cdeb9582_56b4be8c","line":255,"updated":"2021-05-07 12:08:25.000000000","message":"this is not a usecase so this should not be in this section","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1933e7abc61d9926e1fb5cba9bb19aac9ef7ff52","unresolved":false,"context_lines":[{"line_number":252,"context_line":"  * **On-path** FPGA-based SmartNICs are intentionally not considered because"},{"line_number":253,"context_line":"    of a different hardware architecture;"},{"line_number":254,"context_line":"* \"direct\" vnic type;"},{"line_number":255,"context_line":"* A new capability called \"transportnode\" used during scheduling (via a"},{"line_number":256,"context_line":"  binding:profile port attribute containing this capability);"},{"line_number":257,"context_line":""},{"line_number":258,"context_line":"  * The \"switchdev\" capability cannot be reused since a hypervisor does not"}],"source_content_type":"text/x-rst","patch_set":4,"id":"cc2f4a51_3eda2941","line":255,"in_reply_to":"c4ecfa94_f096e008","updated":"2021-05-18 12:45:55.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":252,"context_line":"  * **On-path** FPGA-based SmartNICs are intentionally not considered because"},{"line_number":253,"context_line":"    of a different hardware architecture;"},{"line_number":254,"context_line":"* \"direct\" vnic type;"},{"line_number":255,"context_line":"* A new capability called \"transportnode\" used during scheduling (via a"},{"line_number":256,"context_line":"  binding:profile port attribute containing this capability);"},{"line_number":257,"context_line":""},{"line_number":258,"context_line":"  * The \"switchdev\" capability cannot be reused since a hypervisor does not"}],"source_content_type":"text/x-rst","patch_set":4,"id":"c4ecfa94_f096e008","line":255,"in_reply_to":"cdeb9582_56b4be8c","updated":"2021-05-11 13:17:45.000000000","message":"Will move this to a separate section since this describes the desired outcome rather than a distinct use-case.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":253,"context_line":"    of a different hardware architecture;"},{"line_number":254,"context_line":"* \"direct\" vnic type;"},{"line_number":255,"context_line":"* A new capability called \"transportnode\" used during scheduling (via a"},{"line_number":256,"context_line":"  binding:profile port attribute containing this capability);"},{"line_number":257,"context_line":""},{"line_number":258,"context_line":"  * The \"switchdev\" capability cannot be reused since a hypervisor does not"},{"line_number":259,"context_line":"    see the eswitch via PFs exposed to it by a SmartNIC/DPU;"}],"source_content_type":"text/x-rst","patch_set":4,"id":"eb77a6b5_8668f36f","line":256,"range":{"start_line":256,"start_character":2,"end_line":256,"end_character":17},"updated":"2021-05-07 12:08:25.000000000","message":"the binding:profile is owned by nova and should not be populated by neutron.\nits also an admin only filed so we shoudl not assume the end user can set something in it.\nso in generally im not sure how the binding:profile is relevant since its not populated until after scheduling complete in general.\n\nwe should not be trying to emulate how the trusted vf feature works as that is a hack that should have been done using a neutron extension.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":253,"context_line":"    of a different hardware architecture;"},{"line_number":254,"context_line":"* \"direct\" vnic type;"},{"line_number":255,"context_line":"* A new capability called \"transportnode\" used during scheduling (via a"},{"line_number":256,"context_line":"  binding:profile port attribute containing this capability);"},{"line_number":257,"context_line":""},{"line_number":258,"context_line":"  * The \"switchdev\" capability cannot be reused since a hypervisor does not"},{"line_number":259,"context_line":"    see the eswitch via PFs exposed to it by a SmartNIC/DPU;"}],"source_content_type":"text/x-rst","patch_set":4,"id":"a9e8fb7f_1e5a3c72","line":256,"range":{"start_line":256,"start_character":2,"end_line":256,"end_character":17},"in_reply_to":"eb77a6b5_8668f36f","updated":"2021-05-21 19:01:48.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":255,"context_line":"* A new capability called \"transportnode\" used during scheduling (via a"},{"line_number":256,"context_line":"  binding:profile port attribute containing this capability);"},{"line_number":257,"context_line":""},{"line_number":258,"context_line":"  * The \"switchdev\" capability cannot be reused since a hypervisor does not"},{"line_number":259,"context_line":"    see the eswitch via PFs exposed to it by a SmartNIC/DPU;"},{"line_number":260,"context_line":"* 1 or N SmartNIC/DPUs per host;"},{"line_number":261,"context_line":"* No expectation that the hypervisor host will be responsible for placing an"},{"line_number":262,"context_line":"  image onto a SmartNIC/DPU directly;"}],"source_content_type":"text/x-rst","patch_set":4,"id":"86826c73_30428265","line":259,"range":{"start_line":258,"start_character":0,"end_line":259,"end_character":60},"updated":"2021-05-07 12:08:25.000000000","message":"that is not improant provided the VFs do.\nwill the VFs still have a swtichdev capablity.\n\nthis is also not a usecase","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":255,"context_line":"* A new capability called \"transportnode\" used during scheduling (via a"},{"line_number":256,"context_line":"  binding:profile port attribute containing this capability);"},{"line_number":257,"context_line":""},{"line_number":258,"context_line":"  * The \"switchdev\" capability cannot be reused since a hypervisor does not"},{"line_number":259,"context_line":"    see the eswitch via PFs exposed to it by a SmartNIC/DPU;"},{"line_number":260,"context_line":"* 1 or N SmartNIC/DPUs per host;"},{"line_number":261,"context_line":"* No expectation that the hypervisor host will be responsible for placing an"},{"line_number":262,"context_line":"  image onto a SmartNIC/DPU directly;"}],"source_content_type":"text/x-rst","patch_set":4,"id":"c054af3d_1f8ea620","line":259,"range":{"start_line":258,"start_character":0,"end_line":259,"end_character":60},"in_reply_to":"0a8f8acc_5c03b781","updated":"2021-05-21 19:01:48.000000000","message":"Ack, I think the tagging in the pci_passthrough_whitelist approach you mentioned will work combined with a separate VNIC type.\n\nWill change the spec to use it instead.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1933e7abc61d9926e1fb5cba9bb19aac9ef7ff52","unresolved":true,"context_lines":[{"line_number":255,"context_line":"* A new capability called \"transportnode\" used during scheduling (via a"},{"line_number":256,"context_line":"  binding:profile port attribute containing this capability);"},{"line_number":257,"context_line":""},{"line_number":258,"context_line":"  * The \"switchdev\" capability cannot be reused since a hypervisor does not"},{"line_number":259,"context_line":"    see the eswitch via PFs exposed to it by a SmartNIC/DPU;"},{"line_number":260,"context_line":"* 1 or N SmartNIC/DPUs per host;"},{"line_number":261,"context_line":"* No expectation that the hypervisor host will be responsible for placing an"},{"line_number":262,"context_line":"  image onto a SmartNIC/DPU directly;"}],"source_content_type":"text/x-rst","patch_set":4,"id":"0a8f8acc_5c03b781","line":259,"range":{"start_line":258,"start_character":0,"end_line":259,"end_character":60},"in_reply_to":"3133ea80_844f5aad","updated":"2021-05-18 12:45:55.000000000","message":"yep my team implemented both the libvirt support using ethtool ioctls and the support in nova to get that info form libvirt as part of https://specs.openstack.org/openstack/nova-specs/specs/rocky/approved/enable-sriov-nic-features.html\n\nso in the bluefild case that capablity will not be present ok then yes we need a way to expose it.\n\ni mentioned in a previes comment tha ti think the way to do this is to extend the pci whitelist with a traits key/tag where we can allow you to expose traits in plamcent for it.\n\ni think we can use a new standard trait HW_NET_DPU or similar to model the capablitie for your usecase.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":255,"context_line":"* A new capability called \"transportnode\" used during scheduling (via a"},{"line_number":256,"context_line":"  binding:profile port attribute containing this capability);"},{"line_number":257,"context_line":""},{"line_number":258,"context_line":"  * The \"switchdev\" capability cannot be reused since a hypervisor does not"},{"line_number":259,"context_line":"    see the eswitch via PFs exposed to it by a SmartNIC/DPU;"},{"line_number":260,"context_line":"* 1 or N SmartNIC/DPUs per host;"},{"line_number":261,"context_line":"* No expectation that the hypervisor host will be responsible for placing an"},{"line_number":262,"context_line":"  image onto a SmartNIC/DPU directly;"}],"source_content_type":"text/x-rst","patch_set":4,"id":"3133ea80_844f5aad","line":259,"range":{"start_line":258,"start_character":0,"end_line":259,"end_character":60},"in_reply_to":"86826c73_30428265","updated":"2021-05-11 13:17:45.000000000","message":"Will move this to a separate section since this describes the desired outcome rather than a distinct use-case.\n\n\u003e will the VFs still have a swtichdev capablity.\n\nNo, Nova gets this information from libvirt and libvirt exposes it as a feature in the \"network\" capability if a NIC switch is exposed to the host on a PF (VFs also get it via a PF which is the reason why I am talking about it here in the first place):\n\nhttps://github.com/libvirt/libvirt/blob/16c69e7aaed4cabfd8e8c19cc326854d4c480437/src/util/virnetdev.c#L3166-L3252\n\nthis is how it looks like at the hypervisor side on a Bluefield device:\n\nvirsh #  nodedev-dumpxml net_enp130s0f0np0v0_a2_ba_68_d2_3a_53\n\u003cdevice\u003e\n  \u003cname\u003enet_enp130s0f0np0v0_a2_ba_68_d2_3a_53\u003c/name\u003e\n  \u003cpath\u003e/sys/devices/pci0000:80/0000:80:03.0/0000:82:00.3/net/enp130s0f0np0v0\u003c/path\u003e\n  \u003cparent\u003epci_0000_82_00_3\u003c/parent\u003e\n  \u003ccapability type\u003d\u0027net\u0027\u003e\n    \u003cinterface\u003eenp130s0f0np0v0\u003c/interface\u003e\n    \u003caddress\u003ea2:ba:68:d2:3a:53\u003c/address\u003e\n    \u003clink state\u003d\u0027down\u0027/\u003e\n    \u003cfeature name\u003d\u0027rx\u0027/\u003e\n    \u003cfeature name\u003d\u0027tx\u0027/\u003e\n    \u003cfeature name\u003d\u0027sg\u0027/\u003e\n    \u003cfeature name\u003d\u0027tso\u0027/\u003e\n    \u003cfeature name\u003d\u0027gso\u0027/\u003e\n    \u003cfeature name\u003d\u0027gro\u0027/\u003e\n    \u003cfeature name\u003d\u0027rxvlan\u0027/\u003e\n    \u003cfeature name\u003d\u0027txvlan\u0027/\u003e\n    \u003cfeature name\u003d\u0027rxhash\u0027/\u003e\n    \u003cfeature name\u003d\u0027rdma\u0027/\u003e\n    \u003ccapability type\u003d\u002780203\u0027/\u003e\n  \u003c/capability\u003e\n\u003c/device\u003e\n\nvs how it should look like for a device with a NIC switch exposed:\n\nhttps://github.com/libvirt/libvirt/blob/96e99e49483de4fb293ce26691cbce19110fb3a9/tests/nodedevschemadata/net_00_15_58_2f_e9_55.xml\n\n\n# trying to get eswitch info via devlink at the hypervisor host side.\n$ sudo devlink dev eswitch show pci/0000:82:00.0\ndevlink answers: Operation not supported","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":257,"context_line":""},{"line_number":258,"context_line":"  * The \"switchdev\" capability cannot be reused since a hypervisor does not"},{"line_number":259,"context_line":"    see the eswitch via PFs exposed to it by a SmartNIC/DPU;"},{"line_number":260,"context_line":"* 1 or N SmartNIC/DPUs per host;"},{"line_number":261,"context_line":"* No expectation that the hypervisor host will be responsible for placing an"},{"line_number":262,"context_line":"  image onto a SmartNIC/DPU directly;"},{"line_number":263,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"8f17532b_1d27d1e1","line":260,"range":{"start_line":260,"start_character":2,"end_line":260,"end_character":32},"updated":"2021-05-07 12:08:25.000000000","message":"as an opperator i want to support multiple 1+ SmartNIC/DPUs per host;","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":257,"context_line":""},{"line_number":258,"context_line":"  * The \"switchdev\" capability cannot be reused since a hypervisor does not"},{"line_number":259,"context_line":"    see the eswitch via PFs exposed to it by a SmartNIC/DPU;"},{"line_number":260,"context_line":"* 1 or N SmartNIC/DPUs per host;"},{"line_number":261,"context_line":"* No expectation that the hypervisor host will be responsible for placing an"},{"line_number":262,"context_line":"  image onto a SmartNIC/DPU directly;"},{"line_number":263,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"cf2e97f4_67246c87","line":260,"range":{"start_line":260,"start_character":2,"end_line":260,"end_character":32},"in_reply_to":"8f17532b_1d27d1e1","updated":"2021-05-11 13:17:45.000000000","message":"OK, will rephrase it.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1933e7abc61d9926e1fb5cba9bb19aac9ef7ff52","unresolved":false,"context_lines":[{"line_number":257,"context_line":""},{"line_number":258,"context_line":"  * The \"switchdev\" capability cannot be reused since a hypervisor does not"},{"line_number":259,"context_line":"    see the eswitch via PFs exposed to it by a SmartNIC/DPU;"},{"line_number":260,"context_line":"* 1 or N SmartNIC/DPUs per host;"},{"line_number":261,"context_line":"* No expectation that the hypervisor host will be responsible for placing an"},{"line_number":262,"context_line":"  image onto a SmartNIC/DPU directly;"},{"line_number":263,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"2719a42a_4570aaca","line":260,"range":{"start_line":260,"start_character":2,"end_line":260,"end_character":32},"in_reply_to":"cf2e97f4_67246c87","updated":"2021-05-18 12:45:55.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":258,"context_line":"  * The \"switchdev\" capability cannot be reused since a hypervisor does not"},{"line_number":259,"context_line":"    see the eswitch via PFs exposed to it by a SmartNIC/DPU;"},{"line_number":260,"context_line":"* 1 or N SmartNIC/DPUs per host;"},{"line_number":261,"context_line":"* No expectation that the hypervisor host will be responsible for placing an"},{"line_number":262,"context_line":"  image onto a SmartNIC/DPU directly;"},{"line_number":263,"context_line":""},{"line_number":264,"context_line":"  * a security boundary is assumed between the main board host and the"},{"line_number":265,"context_line":"    SmartNIC/DPU;"},{"line_number":266,"context_line":"  * Any communication between Nova and neutron agents is assumed to be indirect"},{"line_number":267,"context_line":"    and happening via Neutron API;"},{"line_number":268,"context_line":"* Focus on the libvirt virt driver for any related changes initially but make"},{"line_number":269,"context_line":"  the design generic for other virt drivers to follow;"},{"line_number":270,"context_line":"* Primarily target OVN mechanism driver integration at the Neutron side"},{"line_number":271,"context_line":"  (ML2/OVS is also possible but is not a priority) without breaking the"},{"line_number":272,"context_line":"  existing functionality;"},{"line_number":273,"context_line":"* PCI(e) support initially while keeping the design extensible for other types"},{"line_number":274,"context_line":"  of devices (platform devices, CXL 2.0 etc.);"},{"line_number":275,"context_line":"* PF/VFs with vendor-specific control and data plane with future extensions to"},{"line_number":276,"context_line":"  support vDPA to use the virtio data plane."},{"line_number":277,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"aa017d52_b0d53c1f","line":274,"range":{"start_line":261,"start_character":2,"end_line":274,"end_character":46},"updated":"2021-05-07 12:08:25.000000000","message":"i think we can remove this.\nthese are not really usecases and dont really add info that is revent to the nova changes.\ni think there is already too much context in this spec making it hard to review so we shoudl try to slim it down to the point most relevent to the nova team.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":258,"context_line":"  * The \"switchdev\" capability cannot be reused since a hypervisor does not"},{"line_number":259,"context_line":"    see the eswitch via PFs exposed to it by a SmartNIC/DPU;"},{"line_number":260,"context_line":"* 1 or N SmartNIC/DPUs per host;"},{"line_number":261,"context_line":"* No expectation that the hypervisor host will be responsible for placing an"},{"line_number":262,"context_line":"  image onto a SmartNIC/DPU directly;"},{"line_number":263,"context_line":""},{"line_number":264,"context_line":"  * a security boundary is assumed between the main board host and the"},{"line_number":265,"context_line":"    SmartNIC/DPU;"},{"line_number":266,"context_line":"  * Any communication between Nova and neutron agents is assumed to be indirect"},{"line_number":267,"context_line":"    and happening via Neutron API;"},{"line_number":268,"context_line":"* Focus on the libvirt virt driver for any related changes initially but make"},{"line_number":269,"context_line":"  the design generic for other virt drivers to follow;"},{"line_number":270,"context_line":"* Primarily target OVN mechanism driver integration at the Neutron side"},{"line_number":271,"context_line":"  (ML2/OVS is also possible but is not a priority) without breaking the"},{"line_number":272,"context_line":"  existing functionality;"},{"line_number":273,"context_line":"* PCI(e) support initially while keeping the design extensible for other types"},{"line_number":274,"context_line":"  of devices (platform devices, CXL 2.0 etc.);"},{"line_number":275,"context_line":"* PF/VFs with vendor-specific control and data plane with future extensions to"},{"line_number":276,"context_line":"  support vDPA to use the virtio data plane."},{"line_number":277,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"5a319e8e_0ef70f7f","line":274,"range":{"start_line":261,"start_character":2,"end_line":274,"end_character":46},"in_reply_to":"aa017d52_b0d53c1f","updated":"2021-05-21 19:01:48.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":272,"context_line":"  existing functionality;"},{"line_number":273,"context_line":"* PCI(e) support initially while keeping the design extensible for other types"},{"line_number":274,"context_line":"  of devices (platform devices, CXL 2.0 etc.);"},{"line_number":275,"context_line":"* PF/VFs with vendor-specific control and data plane with future extensions to"},{"line_number":276,"context_line":"  support vDPA to use the virtio data plane."},{"line_number":277,"context_line":""},{"line_number":278,"context_line":"The SmartNIC/DPU provisioning is outside of the scope of this specification,"},{"line_number":279,"context_line":"however, the expectation is that it should happen prior to the provisioning"}],"source_content_type":"text/x-rst","patch_set":4,"id":"742a5d1b_708a75c2","line":276,"range":{"start_line":275,"start_character":2,"end_line":276,"end_character":44},"updated":"2021-05-07 12:08:25.000000000","message":"so are you planning to support VDPA initially or not?\nthis is not clear.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":272,"context_line":"  existing functionality;"},{"line_number":273,"context_line":"* PCI(e) support initially while keeping the design extensible for other types"},{"line_number":274,"context_line":"  of devices (platform devices, CXL 2.0 etc.);"},{"line_number":275,"context_line":"* PF/VFs with vendor-specific control and data plane with future extensions to"},{"line_number":276,"context_line":"  support vDPA to use the virtio data plane."},{"line_number":277,"context_line":""},{"line_number":278,"context_line":"The SmartNIC/DPU provisioning is outside of the scope of this specification,"},{"line_number":279,"context_line":"however, the expectation is that it should happen prior to the provisioning"}],"source_content_type":"text/x-rst","patch_set":4,"id":"a84c64d5_94bf5d6c","line":276,"range":{"start_line":275,"start_character":2,"end_line":276,"end_character":44},"in_reply_to":"742a5d1b_708a75c2","updated":"2021-05-11 13:17:45.000000000","message":"OK, I can rephrase it.\n\nThe aim was to say:\n\n* no vDPA initially;\n* the design will aim to allow for vDPA to be supported in the future as much as possible.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1933e7abc61d9926e1fb5cba9bb19aac9ef7ff52","unresolved":false,"context_lines":[{"line_number":272,"context_line":"  existing functionality;"},{"line_number":273,"context_line":"* PCI(e) support initially while keeping the design extensible for other types"},{"line_number":274,"context_line":"  of devices (platform devices, CXL 2.0 etc.);"},{"line_number":275,"context_line":"* PF/VFs with vendor-specific control and data plane with future extensions to"},{"line_number":276,"context_line":"  support vDPA to use the virtio data plane."},{"line_number":277,"context_line":""},{"line_number":278,"context_line":"The SmartNIC/DPU provisioning is outside of the scope of this specification,"},{"line_number":279,"context_line":"however, the expectation is that it should happen prior to the provisioning"}],"source_content_type":"text/x-rst","patch_set":4,"id":"0929359e_5223b811","line":276,"range":{"start_line":275,"start_character":2,"end_line":276,"end_character":44},"in_reply_to":"a84c64d5_94bf5d6c","updated":"2021-05-18 12:45:55.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":275,"context_line":"* PF/VFs with vendor-specific control and data plane with future extensions to"},{"line_number":276,"context_line":"  support vDPA to use the virtio data plane."},{"line_number":277,"context_line":""},{"line_number":278,"context_line":"The SmartNIC/DPU provisioning is outside of the scope of this specification,"},{"line_number":279,"context_line":"however, the expectation is that it should happen prior to the provisioning"},{"line_number":280,"context_line":"of the hypervisor nodes (an image may be provisioned at a factory,"},{"line_number":281,"context_line":"over the network via PXE boot, by loading an image from a BMC over USB if BMC"},{"line_number":282,"context_line":"firmware supports that). Software installation is also out of scope and it"},{"line_number":283,"context_line":"will be up to the deployment automation to do it."},{"line_number":284,"context_line":""},{"line_number":285,"context_line":"From the deployer point of view, besides understanding the provisioning"},{"line_number":286,"context_line":"challenges, it is also expected that a list of devices to be considered"},{"line_number":287,"context_line":"transport nodes will be formed out of PCI device IDs and vendor IDs of PCI"},{"line_number":288,"context_line":"endpoints known to be provided by SmartNICs/DPUs."},{"line_number":289,"context_line":""},{"line_number":290,"context_line":"For an end user, the workflow will be similar to the one with the hardware"},{"line_number":291,"context_line":"offload functionality except for additional multiplexing between the"},{"line_number":292,"context_line":"SmartNIC/DPU use-case and the plain hardware offload case at the port creation"},{"line_number":293,"context_line":"and instance scheduling time."},{"line_number":294,"context_line":""},{"line_number":295,"context_line":"Uplink interface bonding is expected to be handled at the SmartNIC/DPU OS level"},{"line_number":296,"context_line":"and be transparent to the consumers (main board host processes and instances)."},{"line_number":297,"context_line":""},{"line_number":298,"context_line":"Proposed change"},{"line_number":299,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":4,"id":"831b61d0_2064eb2a","line":296,"range":{"start_line":278,"start_character":0,"end_line":296,"end_character":78},"updated":"2021-05-07 12:08:25.000000000","message":"while i know why you added this is not relevent tot the usecasue section.\n\ni would delete it and just state the following in the problem statement section.\n\n\"Configuration and deployment of the smartnic and its controlplane is outside the scope of this spec.\"","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":275,"context_line":"* PF/VFs with vendor-specific control and data plane with future extensions to"},{"line_number":276,"context_line":"  support vDPA to use the virtio data plane."},{"line_number":277,"context_line":""},{"line_number":278,"context_line":"The SmartNIC/DPU provisioning is outside of the scope of this specification,"},{"line_number":279,"context_line":"however, the expectation is that it should happen prior to the provisioning"},{"line_number":280,"context_line":"of the hypervisor nodes (an image may be provisioned at a factory,"},{"line_number":281,"context_line":"over the network via PXE boot, by loading an image from a BMC over USB if BMC"},{"line_number":282,"context_line":"firmware supports that). Software installation is also out of scope and it"},{"line_number":283,"context_line":"will be up to the deployment automation to do it."},{"line_number":284,"context_line":""},{"line_number":285,"context_line":"From the deployer point of view, besides understanding the provisioning"},{"line_number":286,"context_line":"challenges, it is also expected that a list of devices to be considered"},{"line_number":287,"context_line":"transport nodes will be formed out of PCI device IDs and vendor IDs of PCI"},{"line_number":288,"context_line":"endpoints known to be provided by SmartNICs/DPUs."},{"line_number":289,"context_line":""},{"line_number":290,"context_line":"For an end user, the workflow will be similar to the one with the hardware"},{"line_number":291,"context_line":"offload functionality except for additional multiplexing between the"},{"line_number":292,"context_line":"SmartNIC/DPU use-case and the plain hardware offload case at the port creation"},{"line_number":293,"context_line":"and instance scheduling time."},{"line_number":294,"context_line":""},{"line_number":295,"context_line":"Uplink interface bonding is expected to be handled at the SmartNIC/DPU OS level"},{"line_number":296,"context_line":"and be transparent to the consumers (main board host processes and instances)."},{"line_number":297,"context_line":""},{"line_number":298,"context_line":"Proposed change"},{"line_number":299,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":4,"id":"bc4a09ce_a97fd43d","line":296,"range":{"start_line":278,"start_character":0,"end_line":296,"end_character":78},"in_reply_to":"831b61d0_2064eb2a","updated":"2021-05-11 13:17:45.000000000","message":"OK, works for me.\n\nIf it comes up later, it can be documented elsewhere.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":275,"context_line":"* PF/VFs with vendor-specific control and data plane with future extensions to"},{"line_number":276,"context_line":"  support vDPA to use the virtio data plane."},{"line_number":277,"context_line":""},{"line_number":278,"context_line":"The SmartNIC/DPU provisioning is outside of the scope of this specification,"},{"line_number":279,"context_line":"however, the expectation is that it should happen prior to the provisioning"},{"line_number":280,"context_line":"of the hypervisor nodes (an image may be provisioned at a factory,"},{"line_number":281,"context_line":"over the network via PXE boot, by loading an image from a BMC over USB if BMC"},{"line_number":282,"context_line":"firmware supports that). Software installation is also out of scope and it"},{"line_number":283,"context_line":"will be up to the deployment automation to do it."},{"line_number":284,"context_line":""},{"line_number":285,"context_line":"From the deployer point of view, besides understanding the provisioning"},{"line_number":286,"context_line":"challenges, it is also expected that a list of devices to be considered"},{"line_number":287,"context_line":"transport nodes will be formed out of PCI device IDs and vendor IDs of PCI"},{"line_number":288,"context_line":"endpoints known to be provided by SmartNICs/DPUs."},{"line_number":289,"context_line":""},{"line_number":290,"context_line":"For an end user, the workflow will be similar to the one with the hardware"},{"line_number":291,"context_line":"offload functionality except for additional multiplexing between the"},{"line_number":292,"context_line":"SmartNIC/DPU use-case and the plain hardware offload case at the port creation"},{"line_number":293,"context_line":"and instance scheduling time."},{"line_number":294,"context_line":""},{"line_number":295,"context_line":"Uplink interface bonding is expected to be handled at the SmartNIC/DPU OS level"},{"line_number":296,"context_line":"and be transparent to the consumers (main board host processes and instances)."},{"line_number":297,"context_line":""},{"line_number":298,"context_line":"Proposed change"},{"line_number":299,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":4,"id":"17d5a820_10e9657a","line":296,"range":{"start_line":278,"start_character":0,"end_line":296,"end_character":78},"in_reply_to":"bc4a09ce_a97fd43d","updated":"2021-05-21 19:01:48.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":299,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":300,"context_line":""},{"line_number":301,"context_line":"The scope of this change is in Nova but it is a part of a larger effort that"},{"line_number":302,"context_line":"involves OVN and Neutron."},{"line_number":303,"context_line":""},{"line_number":304,"context_line":"Largely, the goal is to gather the information necessary for representor"},{"line_number":305,"context_line":"plugging via Nova and pass it to Neutron during a port update."}],"source_content_type":"text/x-rst","patch_set":4,"id":"f180c4e4_536b64ee","line":302,"updated":"2021-05-07 12:08:25.000000000","message":"ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":false,"context_lines":[{"line_number":299,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":300,"context_line":""},{"line_number":301,"context_line":"The scope of this change is in Nova but it is a part of a larger effort that"},{"line_number":302,"context_line":"involves OVN and Neutron."},{"line_number":303,"context_line":""},{"line_number":304,"context_line":"Largely, the goal is to gather the information necessary for representor"},{"line_number":305,"context_line":"plugging via Nova and pass it to Neutron during a port update."}],"source_content_type":"text/x-rst","patch_set":4,"id":"2c674836_f3ee27bc","line":302,"in_reply_to":"f180c4e4_536b64ee","updated":"2021-05-11 13:17:45.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":304,"context_line":"Largely, the goal is to gather the information necessary for representor"},{"line_number":305,"context_line":"plugging via Nova and pass it to Neutron during a port update."},{"line_number":306,"context_line":""},{"line_number":307,"context_line":"Both the hypervisor host and the transport node host that belong to the same"},{"line_number":308,"context_line":"physical machine can see PCI(e) functions exposed by the controllers on the"},{"line_number":309,"context_line":"same card, therefore, they can see the same unique add-in-card serial number."},{"line_number":310,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"2393ab70_1d49bc0e","line":307,"range":{"start_line":307,"start_character":33,"end_line":307,"end_character":52},"updated":"2021-05-07 12:08:25.000000000","message":"smartnic/dpu","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":false,"context_lines":[{"line_number":304,"context_line":"Largely, the goal is to gather the information necessary for representor"},{"line_number":305,"context_line":"plugging via Nova and pass it to Neutron during a port update."},{"line_number":306,"context_line":""},{"line_number":307,"context_line":"Both the hypervisor host and the transport node host that belong to the same"},{"line_number":308,"context_line":"physical machine can see PCI(e) functions exposed by the controllers on the"},{"line_number":309,"context_line":"same card, therefore, they can see the same unique add-in-card serial number."},{"line_number":310,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"30a58dfc_fcd523b4","line":307,"range":{"start_line":307,"start_character":33,"end_line":307,"end_character":52},"in_reply_to":"2393ab70_1d49bc0e","updated":"2021-05-11 13:17:45.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":305,"context_line":"plugging via Nova and pass it to Neutron during a port update."},{"line_number":306,"context_line":""},{"line_number":307,"context_line":"Both the hypervisor host and the transport node host that belong to the same"},{"line_number":308,"context_line":"physical machine can see PCI(e) functions exposed by the controllers on the"},{"line_number":309,"context_line":"same card, therefore, they can see the same unique add-in-card serial number."},{"line_number":310,"context_line":""},{"line_number":311,"context_line":"Nova can store the information about the observed cards and use it later during"}],"source_content_type":"text/x-rst","patch_set":4,"id":"60fed223_82a530a9","line":308,"range":{"start_line":308,"start_character":17,"end_line":308,"end_character":20},"updated":"2021-05-07 12:08:25.000000000","message":"may","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":305,"context_line":"plugging via Nova and pass it to Neutron during a port update."},{"line_number":306,"context_line":""},{"line_number":307,"context_line":"Both the hypervisor host and the transport node host that belong to the same"},{"line_number":308,"context_line":"physical machine can see PCI(e) functions exposed by the controllers on the"},{"line_number":309,"context_line":"same card, therefore, they can see the same unique add-in-card serial number."},{"line_number":310,"context_line":""},{"line_number":311,"context_line":"Nova can store the information about the observed cards and use it later during"}],"source_content_type":"text/x-rst","patch_set":4,"id":"166dfd38_c1ffbeea","line":308,"range":{"start_line":308,"start_character":57,"end_line":308,"end_character":68},"updated":"2021-05-07 12:08:25.000000000","message":"pcie complex","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":false,"context_lines":[{"line_number":305,"context_line":"plugging via Nova and pass it to Neutron during a port update."},{"line_number":306,"context_line":""},{"line_number":307,"context_line":"Both the hypervisor host and the transport node host that belong to the same"},{"line_number":308,"context_line":"physical machine can see PCI(e) functions exposed by the controllers on the"},{"line_number":309,"context_line":"same card, therefore, they can see the same unique add-in-card serial number."},{"line_number":310,"context_line":""},{"line_number":311,"context_line":"Nova can store the information about the observed cards and use it later during"}],"source_content_type":"text/x-rst","patch_set":4,"id":"a8e592d6_ed4581dc","line":308,"range":{"start_line":308,"start_character":57,"end_line":308,"end_character":68},"in_reply_to":"166dfd38_c1ffbeea","updated":"2021-05-11 13:17:45.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":false,"context_lines":[{"line_number":305,"context_line":"plugging via Nova and pass it to Neutron during a port update."},{"line_number":306,"context_line":""},{"line_number":307,"context_line":"Both the hypervisor host and the transport node host that belong to the same"},{"line_number":308,"context_line":"physical machine can see PCI(e) functions exposed by the controllers on the"},{"line_number":309,"context_line":"same card, therefore, they can see the same unique add-in-card serial number."},{"line_number":310,"context_line":""},{"line_number":311,"context_line":"Nova can store the information about the observed cards and use it later during"}],"source_content_type":"text/x-rst","patch_set":4,"id":"cb628c92_1958934a","line":308,"range":{"start_line":308,"start_character":17,"end_line":308,"end_character":20},"in_reply_to":"60fed223_82a530a9","updated":"2021-05-11 13:17:45.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":311,"context_line":"Nova can store the information about the observed cards and use it later during"},{"line_number":312,"context_line":"the port update process to affect the selection of a transport node that will"},{"line_number":313,"context_line":"do the representor plugging."},{"line_number":314,"context_line":""},{"line_number":315,"context_line":"A config value in the [pci] section declaring which PCI vendor and device IDs"},{"line_number":316,"context_line":"to treat as indicators that a function belongs to a transport node will tell"},{"line_number":317,"context_line":"Nova for which devices to collect the serial numbers from the PCI VPD"}],"source_content_type":"text/x-rst","patch_set":4,"id":"ef7deff3_c7545a2e","line":314,"updated":"2021-05-07 12:08:25.000000000","message":"+1","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":false,"context_lines":[{"line_number":311,"context_line":"Nova can store the information about the observed cards and use it later during"},{"line_number":312,"context_line":"the port update process to affect the selection of a transport node that will"},{"line_number":313,"context_line":"do the representor plugging."},{"line_number":314,"context_line":""},{"line_number":315,"context_line":"A config value in the [pci] section declaring which PCI vendor and device IDs"},{"line_number":316,"context_line":"to treat as indicators that a function belongs to a transport node will tell"},{"line_number":317,"context_line":"Nova for which devices to collect the serial numbers from the PCI VPD"}],"source_content_type":"text/x-rst","patch_set":4,"id":"c6b67c5c_e555894f","line":314,"in_reply_to":"ef7deff3_c7545a2e","updated":"2021-05-11 13:17:45.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":315,"context_line":"A config value in the [pci] section declaring which PCI vendor and device IDs"},{"line_number":316,"context_line":"to treat as indicators that a function belongs to a transport node will tell"},{"line_number":317,"context_line":"Nova for which devices to collect the serial numbers from the PCI VPD"},{"line_number":318,"context_line":"capability."},{"line_number":319,"context_line":""},{"line_number":320,"context_line":"* Store VPD info from the PCI(e) capability for each PciDevice"},{"line_number":321,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"77868123_6f586465","line":318,"updated":"2021-05-07 12:08:25.000000000","message":"we could do this but i dont think its required.\nwhy would it be harmful to collect it for all devices?\nthis is somehting we only need to do once on comptue agent start.\nwe do not update the extra_info in the db perodicaly so this seam like a premature optimisation to me\nso im not sure its worth the extra complxity.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1933e7abc61d9926e1fb5cba9bb19aac9ef7ff52","unresolved":true,"context_lines":[{"line_number":315,"context_line":"A config value in the [pci] section declaring which PCI vendor and device IDs"},{"line_number":316,"context_line":"to treat as indicators that a function belongs to a transport node will tell"},{"line_number":317,"context_line":"Nova for which devices to collect the serial numbers from the PCI VPD"},{"line_number":318,"context_line":"capability."},{"line_number":319,"context_line":""},{"line_number":320,"context_line":"* Store VPD info from the PCI(e) capability for each PciDevice"},{"line_number":321,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"67b89323_2de3061c","line":318,"in_reply_to":"53af8f2a_27b8ff4e","updated":"2021-05-18 12:45:55.000000000","message":"yes if we are just doing this  at agent start i think that is likely acceptable as we would be restarting the agents as part of upgrade anyway. you are correct though that we potentially would have to udpate at least a portion of exising records. not all nic today actully implent the VPD extenition so only a subset fo pci records will update if they have the new info.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":315,"context_line":"A config value in the [pci] section declaring which PCI vendor and device IDs"},{"line_number":316,"context_line":"to treat as indicators that a function belongs to a transport node will tell"},{"line_number":317,"context_line":"Nova for which devices to collect the serial numbers from the PCI VPD"},{"line_number":318,"context_line":"capability."},{"line_number":319,"context_line":""},{"line_number":320,"context_line":"* Store VPD info from the PCI(e) capability for each PciDevice"},{"line_number":321,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"d9006431_caf8631a","line":318,"in_reply_to":"67b89323_2de3061c","updated":"2021-05-21 19:01:48.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":315,"context_line":"A config value in the [pci] section declaring which PCI vendor and device IDs"},{"line_number":316,"context_line":"to treat as indicators that a function belongs to a transport node will tell"},{"line_number":317,"context_line":"Nova for which devices to collect the serial numbers from the PCI VPD"},{"line_number":318,"context_line":"capability."},{"line_number":319,"context_line":""},{"line_number":320,"context_line":"* Store VPD info from the PCI(e) capability for each PciDevice"},{"line_number":321,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"53af8f2a_27b8ff4e","line":318,"in_reply_to":"77868123_6f586465","updated":"2021-05-11 13:17:45.000000000","message":"Agreed, this is probably more relevant for dealing with the lack of the switchdev capability exposure to the hypervisor host.\n\nWe would need to re-iterate over the existing PCI devices during an upgrade in order to collect this information for all devices then.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":321,"context_line":""},{"line_number":322,"context_line":"  * card_serial_number - a string of up to 255 bytes since PCI and PCIe specs"},{"line_number":323,"context_line":"    use a 1-byte length field for the SN;"},{"line_number":324,"context_line":"  * ``extra_info: \u0027{\"capabilities\": \"vpd\": {\"card_serial_number\": \"\u003csn\u003e\"}]\u0027}``;"},{"line_number":325,"context_line":"* Implement modules to extract the card serial numbers:"},{"line_number":326,"context_line":""},{"line_number":327,"context_line":"  * using devlink-info in a PCIe-independent way: if drivers and firmware"}],"source_content_type":"text/x-rst","patch_set":4,"id":"61f4bb03_336687ac","line":324,"range":{"start_line":324,"start_character":5,"end_line":324,"end_character":79},"updated":"2021-05-07 12:08:25.000000000","message":"the alternative is to do \n\n`extra_info: \u0027{\"capabilities\": ..., \"vpd\": {\"card_serial_number\": \"\u003csn\u003e\"}\u0027}``;\n\nwe do not need to nest this inside the capabilities dictionary.\nwe can but the intention was always to allow additional top level keys in extra info.\nthis is a pci capability so im ok with what you proposed but just wanted to make this point incase you had not considered that.\n\nreusing capablities is only useful if we wanted to also resue the existing code to","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":321,"context_line":""},{"line_number":322,"context_line":"  * card_serial_number - a string of up to 255 bytes since PCI and PCIe specs"},{"line_number":323,"context_line":"    use a 1-byte length field for the SN;"},{"line_number":324,"context_line":"  * ``extra_info: \u0027{\"capabilities\": \"vpd\": {\"card_serial_number\": \"\u003csn\u003e\"}]\u0027}``;"},{"line_number":325,"context_line":"* Implement modules to extract the card serial numbers:"},{"line_number":326,"context_line":""},{"line_number":327,"context_line":"  * using devlink-info in a PCIe-independent way: if drivers and firmware"}],"source_content_type":"text/x-rst","patch_set":4,"id":"f88e45c3_7df9d087","line":324,"range":{"start_line":324,"start_character":5,"end_line":324,"end_character":79},"in_reply_to":"61f4bb03_336687ac","updated":"2021-05-11 13:17:45.000000000","message":"Yes, as you say, I thought it would make sense to put it alongside PCI capabilities since VPD is a capability in the PCI/PCIe specs.\n\nI don\u0027t have a particular preference - just felt like reusing the existing structure.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":321,"context_line":""},{"line_number":322,"context_line":"  * card_serial_number - a string of up to 255 bytes since PCI and PCIe specs"},{"line_number":323,"context_line":"    use a 1-byte length field for the SN;"},{"line_number":324,"context_line":"  * ``extra_info: \u0027{\"capabilities\": \"vpd\": {\"card_serial_number\": \"\u003csn\u003e\"}]\u0027}``;"},{"line_number":325,"context_line":"* Implement modules to extract the card serial numbers:"},{"line_number":326,"context_line":""},{"line_number":327,"context_line":"  * using devlink-info in a PCIe-independent way: if drivers and firmware"}],"source_content_type":"text/x-rst","patch_set":4,"id":"db735acf_4b613df8","line":324,"range":{"start_line":324,"start_character":5,"end_line":324,"end_character":79},"in_reply_to":"9a444c00_73981c52","updated":"2021-05-21 19:01:48.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1933e7abc61d9926e1fb5cba9bb19aac9ef7ff52","unresolved":true,"context_lines":[{"line_number":321,"context_line":""},{"line_number":322,"context_line":"  * card_serial_number - a string of up to 255 bytes since PCI and PCIe specs"},{"line_number":323,"context_line":"    use a 1-byte length field for the SN;"},{"line_number":324,"context_line":"  * ``extra_info: \u0027{\"capabilities\": \"vpd\": {\"card_serial_number\": \"\u003csn\u003e\"}]\u0027}``;"},{"line_number":325,"context_line":"* Implement modules to extract the card serial numbers:"},{"line_number":326,"context_line":""},{"line_number":327,"context_line":"  * using devlink-info in a PCIe-independent way: if drivers and firmware"}],"source_content_type":"text/x-rst","patch_set":4,"id":"9a444c00_73981c52","line":324,"range":{"start_line":324,"start_character":5,"end_line":324,"end_character":79},"in_reply_to":"f88e45c3_7df9d087","updated":"2021-05-18 12:45:55.000000000","message":"i don\u0027t have a strong preference either. when i looked at adding vdpa path info to this i was going to make it a top level key because it was not a capablies it was just metadata about the vdpa device.\n\njust wanted to make sure you did not feel forced to use the same structure.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":324,"context_line":"  * ``extra_info: \u0027{\"capabilities\": \"vpd\": {\"card_serial_number\": \"\u003csn\u003e\"}]\u0027}``;"},{"line_number":325,"context_line":"* Implement modules to extract the card serial numbers:"},{"line_number":326,"context_line":""},{"line_number":327,"context_line":"  * using devlink-info in a PCIe-independent way: if drivers and firmware"},{"line_number":328,"context_line":"    support getting a card serial number and the kernel version supports"},{"line_number":329,"context_line":"    devlink-info;"},{"line_number":330,"context_line":""},{"line_number":331,"context_line":"    * Relevant kernel and iproute versions:"},{"line_number":332,"context_line":""},{"line_number":333,"context_line":"      * 4.10 - devlink infrastructure itself;"},{"line_number":334,"context_line":"      * 5.1 - devlink-info API;"},{"line_number":335,"context_line":"      * 5.9 - \"board.serial_number\" added to devlink-info (subject to NIC"},{"line_number":336,"context_line":"        driver support);"},{"line_number":337,"context_line":"  * using PCIe VPD data exposed by PCI(e) endpoints: as a fallback if devlink"},{"line_number":338,"context_line":"    is not usable for some reason (old kernel, no driver support etc.);"},{"line_number":339,"context_line":"* Extend the libvirt virt driver code to gather card serial numbers via PFs"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fc294b88_b7e8ffb7","line":336,"range":{"start_line":327,"start_character":3,"end_line":336,"end_character":24},"updated":"2021-05-07 12:08:25.000000000","message":"if we can do that via pyroute2 then this might be a good option although it\ndoes limit this to nics.\n\nin general i think we would prefer if libvirt did this on our behalf and included the serial number in the nodedev xml.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":324,"context_line":"  * ``extra_info: \u0027{\"capabilities\": \"vpd\": {\"card_serial_number\": \"\u003csn\u003e\"}]\u0027}``;"},{"line_number":325,"context_line":"* Implement modules to extract the card serial numbers:"},{"line_number":326,"context_line":""},{"line_number":327,"context_line":"  * using devlink-info in a PCIe-independent way: if drivers and firmware"},{"line_number":328,"context_line":"    support getting a card serial number and the kernel version supports"},{"line_number":329,"context_line":"    devlink-info;"},{"line_number":330,"context_line":""},{"line_number":331,"context_line":"    * Relevant kernel and iproute versions:"},{"line_number":332,"context_line":""},{"line_number":333,"context_line":"      * 4.10 - devlink infrastructure itself;"},{"line_number":334,"context_line":"      * 5.1 - devlink-info API;"},{"line_number":335,"context_line":"      * 5.9 - \"board.serial_number\" added to devlink-info (subject to NIC"},{"line_number":336,"context_line":"        driver support);"},{"line_number":337,"context_line":"  * using PCIe VPD data exposed by PCI(e) endpoints: as a fallback if devlink"},{"line_number":338,"context_line":"    is not usable for some reason (old kernel, no driver support etc.);"},{"line_number":339,"context_line":"* Extend the libvirt virt driver code to gather card serial numbers via PFs"}],"source_content_type":"text/x-rst","patch_set":4,"id":"1786f176_c222097c","line":336,"range":{"start_line":327,"start_character":3,"end_line":336,"end_character":24},"in_reply_to":"0b37edae_f2756a72","updated":"2021-05-21 19:01:48.000000000","message":"Will attempt to use pyroute2. This is an implementation detail in the spec so we can just say that devlink-info will be used one way or the other.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":324,"context_line":"  * ``extra_info: \u0027{\"capabilities\": \"vpd\": {\"card_serial_number\": \"\u003csn\u003e\"}]\u0027}``;"},{"line_number":325,"context_line":"* Implement modules to extract the card serial numbers:"},{"line_number":326,"context_line":""},{"line_number":327,"context_line":"  * using devlink-info in a PCIe-independent way: if drivers and firmware"},{"line_number":328,"context_line":"    support getting a card serial number and the kernel version supports"},{"line_number":329,"context_line":"    devlink-info;"},{"line_number":330,"context_line":""},{"line_number":331,"context_line":"    * Relevant kernel and iproute versions:"},{"line_number":332,"context_line":""},{"line_number":333,"context_line":"      * 4.10 - devlink infrastructure itself;"},{"line_number":334,"context_line":"      * 5.1 - devlink-info API;"},{"line_number":335,"context_line":"      * 5.9 - \"board.serial_number\" added to devlink-info (subject to NIC"},{"line_number":336,"context_line":"        driver support);"},{"line_number":337,"context_line":"  * using PCIe VPD data exposed by PCI(e) endpoints: as a fallback if devlink"},{"line_number":338,"context_line":"    is not usable for some reason (old kernel, no driver support etc.);"},{"line_number":339,"context_line":"* Extend the libvirt virt driver code to gather card serial numbers via PFs"}],"source_content_type":"text/x-rst","patch_set":4,"id":"0b37edae_f2756a72","line":336,"range":{"start_line":327,"start_character":3,"end_line":336,"end_character":24},"in_reply_to":"fc294b88_b7e8ffb7","updated":"2021-05-11 13:17:45.000000000","message":"devlink-info support for that is very recent. \"devlink\" is also a separate binary in iproute2 so I need to check if pyroute2 has anything like this.\n\nJust calling the devlink binary seems like the fastest way to get it to work. The plan had been to use direct devlink calls first and then replace it with native libvirt functionality later when it appears.\n\nLibvirt is definitely the right place to do it long term - just makes the Nova implementation dependent on a particular version of it and on upstreaming of this in libvirt.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":334,"context_line":"      * 5.1 - devlink-info API;"},{"line_number":335,"context_line":"      * 5.9 - \"board.serial_number\" added to devlink-info (subject to NIC"},{"line_number":336,"context_line":"        driver support);"},{"line_number":337,"context_line":"  * using PCIe VPD data exposed by PCI(e) endpoints: as a fallback if devlink"},{"line_number":338,"context_line":"    is not usable for some reason (old kernel, no driver support etc.);"},{"line_number":339,"context_line":"* Extend the libvirt virt driver code to gather card serial numbers via PFs"},{"line_number":340,"context_line":"  and VFs;"},{"line_number":341,"context_line":"* Modify the PciDevTracker to store the card serial number information (if"}],"source_content_type":"text/x-rst","patch_set":4,"id":"865c26cc_2f0b8e9b","line":338,"range":{"start_line":337,"start_character":2,"end_line":338,"end_character":71},"updated":"2021-05-07 12:08:25.000000000","message":"depending on the complexity im not sure we want to have 2 implemation of this in nova.\n\nin general we try to avoid directly interacting with sysfs or hardware form nova and in the libvirt virt driver in partcalar we want to leverage it to do this lookup where possible so if libvirt cannot already provide this infomation we likely shoudl request that they added.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":334,"context_line":"      * 5.1 - devlink-info API;"},{"line_number":335,"context_line":"      * 5.9 - \"board.serial_number\" added to devlink-info (subject to NIC"},{"line_number":336,"context_line":"        driver support);"},{"line_number":337,"context_line":"  * using PCIe VPD data exposed by PCI(e) endpoints: as a fallback if devlink"},{"line_number":338,"context_line":"    is not usable for some reason (old kernel, no driver support etc.);"},{"line_number":339,"context_line":"* Extend the libvirt virt driver code to gather card serial numbers via PFs"},{"line_number":340,"context_line":"  and VFs;"},{"line_number":341,"context_line":"* Modify the PciDevTracker to store the card serial number information (if"}],"source_content_type":"text/x-rst","patch_set":4,"id":"9789679b_e679d219","line":338,"range":{"start_line":337,"start_character":2,"end_line":338,"end_character":71},"in_reply_to":"865c26cc_2f0b8e9b","updated":"2021-05-11 13:17:45.000000000","message":"It\u0027s not that difficult, I have some prototype code which I can share here that does both.\n\nThe plan is to do it in Nova to enable the initial functionality and later switch to using libvirt depending on how fast the upstreaming process will go.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":334,"context_line":"      * 5.1 - devlink-info API;"},{"line_number":335,"context_line":"      * 5.9 - \"board.serial_number\" added to devlink-info (subject to NIC"},{"line_number":336,"context_line":"        driver support);"},{"line_number":337,"context_line":"  * using PCIe VPD data exposed by PCI(e) endpoints: as a fallback if devlink"},{"line_number":338,"context_line":"    is not usable for some reason (old kernel, no driver support etc.);"},{"line_number":339,"context_line":"* Extend the libvirt virt driver code to gather card serial numbers via PFs"},{"line_number":340,"context_line":"  and VFs;"},{"line_number":341,"context_line":"* Modify the PciDevTracker to store the card serial number information (if"}],"source_content_type":"text/x-rst","patch_set":4,"id":"d8394425_e2fd7593","line":338,"range":{"start_line":337,"start_character":2,"end_line":338,"end_character":71},"in_reply_to":"9789679b_e679d219","updated":"2021-05-21 19:01:48.000000000","message":"Will send an e-mail to the libvirt development mailing list as discussed in a different comment.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":338,"context_line":"    is not usable for some reason (old kernel, no driver support etc.);"},{"line_number":339,"context_line":"* Extend the libvirt virt driver code to gather card serial numbers via PFs"},{"line_number":340,"context_line":"  and VFs;"},{"line_number":341,"context_line":"* Modify the PciDevTracker to store the card serial number information (if"},{"line_number":342,"context_line":"  present) in the PciDevice extra_info column under the \"vpd\" capability;"},{"line_number":343,"context_line":"* Modify the PciDevTracker code to add the ``transportnode`` capability to the"},{"line_number":344,"context_line":"  network capabilities dict of a device so that it gets persisted to extra_info"}],"source_content_type":"text/x-rst","patch_set":4,"id":"6aacd503_1e7c0e1f","line":341,"range":{"start_line":341,"start_character":13,"end_line":341,"end_character":26},"updated":"2021-05-07 12:08:25.000000000","message":"i dont think we actully need to modify it since the data stored in the pci tracker is provide by the virt driver so i think this can be largely be transparent to the pci module depending on where we choose to retive the info..","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":338,"context_line":"    is not usable for some reason (old kernel, no driver support etc.);"},{"line_number":339,"context_line":"* Extend the libvirt virt driver code to gather card serial numbers via PFs"},{"line_number":340,"context_line":"  and VFs;"},{"line_number":341,"context_line":"* Modify the PciDevTracker to store the card serial number information (if"},{"line_number":342,"context_line":"  present) in the PciDevice extra_info column under the \"vpd\" capability;"},{"line_number":343,"context_line":"* Modify the PciDevTracker code to add the ``transportnode`` capability to the"},{"line_number":344,"context_line":"  network capabilities dict of a device so that it gets persisted to extra_info"}],"source_content_type":"text/x-rst","patch_set":4,"id":"aa43fb75_48b1fecd","line":341,"range":{"start_line":341,"start_character":13,"end_line":341,"end_character":26},"in_reply_to":"6aacd503_1e7c0e1f","updated":"2021-05-11 13:17:45.000000000","message":"OK, I\u0027ll have another look at the code and comment here.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":338,"context_line":"    is not usable for some reason (old kernel, no driver support etc.);"},{"line_number":339,"context_line":"* Extend the libvirt virt driver code to gather card serial numbers via PFs"},{"line_number":340,"context_line":"  and VFs;"},{"line_number":341,"context_line":"* Modify the PciDevTracker to store the card serial number information (if"},{"line_number":342,"context_line":"  present) in the PciDevice extra_info column under the \"vpd\" capability;"},{"line_number":343,"context_line":"* Modify the PciDevTracker code to add the ``transportnode`` capability to the"},{"line_number":344,"context_line":"  network capabilities dict of a device so that it gets persisted to extra_info"}],"source_content_type":"text/x-rst","patch_set":4,"id":"b4ff923b_67bb9f7b","line":341,"range":{"start_line":341,"start_character":13,"end_line":341,"end_character":26},"in_reply_to":"aa43fb75_48b1fecd","updated":"2021-05-21 19:01:48.000000000","message":"Done","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":340,"context_line":"  and VFs;"},{"line_number":341,"context_line":"* Modify the PciDevTracker to store the card serial number information (if"},{"line_number":342,"context_line":"  present) in the PciDevice extra_info column under the \"vpd\" capability;"},{"line_number":343,"context_line":"* Modify the PciDevTracker code to add the ``transportnode`` capability to the"},{"line_number":344,"context_line":"  network capabilities dict of a device so that it gets persisted to extra_info"},{"line_number":345,"context_line":"  of a PciDevice as follows:"},{"line_number":346,"context_line":"  ``extra_info: \u0027{\"capabilities\": \"network\": [\"transportnode\"]\u0027}``"},{"line_number":347,"context_line":"* For each function added to an instance, collect a PF MAC and VF logical"},{"line_number":348,"context_line":"  number as seen by the hypervisor host and pass them to Neutron along with"},{"line_number":349,"context_line":"  the card serial number during port update requests that happen during"}],"source_content_type":"text/x-rst","patch_set":4,"id":"b97ded4d_948d52cc","line":346,"range":{"start_line":343,"start_character":2,"end_line":346,"end_character":66},"updated":"2021-05-07 12:08:25.000000000","message":"this i am not convince is needed.\n\nif we were to do this is would be a change to the pci whitelist to add a new tag and then it would be stored as part of the tags.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1933e7abc61d9926e1fb5cba9bb19aac9ef7ff52","unresolved":true,"context_lines":[{"line_number":340,"context_line":"  and VFs;"},{"line_number":341,"context_line":"* Modify the PciDevTracker to store the card serial number information (if"},{"line_number":342,"context_line":"  present) in the PciDevice extra_info column under the \"vpd\" capability;"},{"line_number":343,"context_line":"* Modify the PciDevTracker code to add the ``transportnode`` capability to the"},{"line_number":344,"context_line":"  network capabilities dict of a device so that it gets persisted to extra_info"},{"line_number":345,"context_line":"  of a PciDevice as follows:"},{"line_number":346,"context_line":"  ``extra_info: \u0027{\"capabilities\": \"network\": [\"transportnode\"]\u0027}``"},{"line_number":347,"context_line":"* For each function added to an instance, collect a PF MAC and VF logical"},{"line_number":348,"context_line":"  number as seen by the hypervisor host and pass them to Neutron along with"},{"line_number":349,"context_line":"  the card serial number during port update requests that happen during"}],"source_content_type":"text/x-rst","patch_set":4,"id":"d2bb1267_73960ce1","line":346,"range":{"start_line":343,"start_character":2,"end_line":346,"end_character":66},"in_reply_to":"b97ded4d_948d52cc","updated":"2021-05-18 12:45:55.000000000","message":"so reading your previous comments something like this is needed since we cannot detect the swtichdev capablity in all cases from the. host.\n\ni think this can be done more genericaly as a placment trait but yes we will need to modify the livrt dirver to popluate this info.\n\nthe data stored in the pci tacker is generated by _get_pcidev_info\nhttps://github.com/openstack/nova/blob/82141b12c356fe374822038bdf0a6f0aaa32047b/nova/virt/libvirt/host.py#L1228\n\nand the capablities are generated from the inner _get_device_capabilities function\n\nhttps://github.com/openstack/nova/blob/82141b12c356fe374822038bdf0a6f0aaa32047b/nova/virt/libvirt/host.py#L1283-L1300\n\n\nsothis will need to be modified but the PCIDevTracker shoudl not need to be.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":340,"context_line":"  and VFs;"},{"line_number":341,"context_line":"* Modify the PciDevTracker to store the card serial number information (if"},{"line_number":342,"context_line":"  present) in the PciDevice extra_info column under the \"vpd\" capability;"},{"line_number":343,"context_line":"* Modify the PciDevTracker code to add the ``transportnode`` capability to the"},{"line_number":344,"context_line":"  network capabilities dict of a device so that it gets persisted to extra_info"},{"line_number":345,"context_line":"  of a PciDevice as follows:"},{"line_number":346,"context_line":"  ``extra_info: \u0027{\"capabilities\": \"network\": [\"transportnode\"]\u0027}``"},{"line_number":347,"context_line":"* For each function added to an instance, collect a PF MAC and VF logical"},{"line_number":348,"context_line":"  number as seen by the hypervisor host and pass them to Neutron along with"},{"line_number":349,"context_line":"  the card serial number during port update requests that happen during"}],"source_content_type":"text/x-rst","patch_set":4,"id":"969cf068_a7231a81","line":346,"range":{"start_line":343,"start_character":2,"end_line":346,"end_character":66},"in_reply_to":"d2bb1267_73960ce1","updated":"2021-05-21 19:01:48.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":344,"context_line":"  network capabilities dict of a device so that it gets persisted to extra_info"},{"line_number":345,"context_line":"  of a PciDevice as follows:"},{"line_number":346,"context_line":"  ``extra_info: \u0027{\"capabilities\": \"network\": [\"transportnode\"]\u0027}``"},{"line_number":347,"context_line":"* For each function added to an instance, collect a PF MAC and VF logical"},{"line_number":348,"context_line":"  number as seen by the hypervisor host and pass them to Neutron along with"},{"line_number":349,"context_line":"  the card serial number during port update requests that happen during"},{"line_number":350,"context_line":"  instance creation (see the relevant section below for more details);"}],"source_content_type":"text/x-rst","patch_set":4,"id":"47cbfd31_d1bcb0b9","line":347,"range":{"start_line":347,"start_character":52,"end_line":347,"end_character":59},"updated":"2021-05-07 12:08:25.000000000","message":"PF mac is not one of the atibute we shoudl rely on.\n\nnova does not track the mac adress  fo pci devices today and im not sure we should start relying on it.\ndoes the PF have a serial number we can use or a logical port number like the VF?\nperhaps we can use the switchdev id?","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":344,"context_line":"  network capabilities dict of a device so that it gets persisted to extra_info"},{"line_number":345,"context_line":"  of a PciDevice as follows:"},{"line_number":346,"context_line":"  ``extra_info: \u0027{\"capabilities\": \"network\": [\"transportnode\"]\u0027}``"},{"line_number":347,"context_line":"* For each function added to an instance, collect a PF MAC and VF logical"},{"line_number":348,"context_line":"  number as seen by the hypervisor host and pass them to Neutron along with"},{"line_number":349,"context_line":"  the card serial number during port update requests that happen during"},{"line_number":350,"context_line":"  instance creation (see the relevant section below for more details);"}],"source_content_type":"text/x-rst","patch_set":4,"id":"2e366082_6f2acff0","line":347,"range":{"start_line":347,"start_character":42,"end_line":347,"end_character":51},"updated":"2021-05-07 12:08:25.000000000","message":"the collection is actully done in init_host","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1933e7abc61d9926e1fb5cba9bb19aac9ef7ff52","unresolved":true,"context_lines":[{"line_number":344,"context_line":"  network capabilities dict of a device so that it gets persisted to extra_info"},{"line_number":345,"context_line":"  of a PciDevice as follows:"},{"line_number":346,"context_line":"  ``extra_info: \u0027{\"capabilities\": \"network\": [\"transportnode\"]\u0027}``"},{"line_number":347,"context_line":"* For each function added to an instance, collect a PF MAC and VF logical"},{"line_number":348,"context_line":"  number as seen by the hypervisor host and pass them to Neutron along with"},{"line_number":349,"context_line":"  the card serial number during port update requests that happen during"},{"line_number":350,"context_line":"  instance creation (see the relevant section below for more details);"}],"source_content_type":"text/x-rst","patch_set":4,"id":"dbebf84d_6ac4ac22","line":347,"range":{"start_line":347,"start_character":52,"end_line":347,"end_character":59},"in_reply_to":"0fe9ac0e_e476efcf","updated":"2021-05-18 12:45:55.000000000","message":"ok i see.\ni guess we are assume that PF will be bound to a network driver? and not vfio-pci or similar.\n\nto use the mac i think we need to state that as a requirement as if you do not bind the PF to a networkign driver on the host it will not have a netdev and we wont easially be able to get its mac.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":344,"context_line":"  network capabilities dict of a device so that it gets persisted to extra_info"},{"line_number":345,"context_line":"  of a PciDevice as follows:"},{"line_number":346,"context_line":"  ``extra_info: \u0027{\"capabilities\": \"network\": [\"transportnode\"]\u0027}``"},{"line_number":347,"context_line":"* For each function added to an instance, collect a PF MAC and VF logical"},{"line_number":348,"context_line":"  number as seen by the hypervisor host and pass them to Neutron along with"},{"line_number":349,"context_line":"  the card serial number during port update requests that happen during"},{"line_number":350,"context_line":"  instance creation (see the relevant section below for more details);"}],"source_content_type":"text/x-rst","patch_set":4,"id":"0fe9ac0e_e476efcf","line":347,"range":{"start_line":347,"start_character":52,"end_line":347,"end_character":59},"in_reply_to":"47cbfd31_d1bcb0b9","updated":"2021-05-11 13:17:45.000000000","message":"Unfortunately, we cannot rely on the switchdev ID since it is not visible from the hypervisor host.\n\n$ sudo devlink dev eswitch show pci/0000:82:00.0\ndevlink answers: Operation not supported\n\nsudo cat /sys/bus/pci/devices/0000\\:82\\:00.0/net/enp3s0f0/phys_switch_id \ncat: \u0027/sys/bus/pci/devices/0000:82:00.0/net/enp3s0f0/phys_switch_id\u0027: Operation not supported\n\n\nsudo lspci -v | grep -i mella\n82:00.0 Ethernet controller: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller (rev 01)\n\tSubsystem: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller\n82:00.1 Ethernet controller: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller (rev 01)\n\tSubsystem: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller\n82:00.2 DMA controller: Mellanox Technologies MT42822 BlueField-2 SoC Management Interface (rev 01) (prog-if 00 [8237])\n\tSubsystem: Mellanox Technologies MT42822 BlueField-2 SoC Management Interface\n82:00.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)\n\tSubsystem: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function\n82:00.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)\n\tSubsystem: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function\n82:00.5 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)\n\tSubsystem: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function\n82:00.6 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)\n\tSubsystem: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function\n\nsudo cat /sys/bus/pci/devices/0000\\:82\\:00.0/net/enp3s0f0/phys_switch_id \ncat: \u0027/sys/bus/pci/devices/0000:82:00.0/net/enp3s0f0/phys_switch_id\u0027: Operation not supported\n\nsudo cat /sys/bus/pci/devices/0000\\:82\\:00.3/net/enp130s0f0np0v0/phys_switch_id \ncat: \u0027/sys/bus/pci/devices/0000:82:00.3/net/enp130s0f0np0v0/phys_switch_id\u0027: Operation not supported\n\n\nPF numbers are scoped to controllers at the SmartNIC DPU host side so PF numbers themselves are not enough to identify a PF.\n\nhttps://github.com/torvalds/linux/blob/v5.12/net/core/devlink.c#L8649-L8650\n\n\nIf we are using VFs only and PF MAC is set from the SmartNIC DPU host it could be a way to refer to a PF from both sides.\n\nUnless I don\u0027t see an obvious way to do it reliably.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":344,"context_line":"  network capabilities dict of a device so that it gets persisted to extra_info"},{"line_number":345,"context_line":"  of a PciDevice as follows:"},{"line_number":346,"context_line":"  ``extra_info: \u0027{\"capabilities\": \"network\": [\"transportnode\"]\u0027}``"},{"line_number":347,"context_line":"* For each function added to an instance, collect a PF MAC and VF logical"},{"line_number":348,"context_line":"  number as seen by the hypervisor host and pass them to Neutron along with"},{"line_number":349,"context_line":"  the card serial number during port update requests that happen during"},{"line_number":350,"context_line":"  instance creation (see the relevant section below for more details);"}],"source_content_type":"text/x-rst","patch_set":4,"id":"354e9125_b2f8fac2","line":347,"range":{"start_line":347,"start_character":52,"end_line":347,"end_character":59},"in_reply_to":"dbebf84d_6ac4ac22","updated":"2021-05-21 19:01:48.000000000","message":"Ack, I think requiring that vfio-pci is used only for VFs and that an associated PF will be utilized for hypervisor host purposes (e.g. control plane comms) is reasonable. At least that\u0027s a working assumption we had due to the potential lack of LoM interfaces on a given host.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":344,"context_line":"  network capabilities dict of a device so that it gets persisted to extra_info"},{"line_number":345,"context_line":"  of a PciDevice as follows:"},{"line_number":346,"context_line":"  ``extra_info: \u0027{\"capabilities\": \"network\": [\"transportnode\"]\u0027}``"},{"line_number":347,"context_line":"* For each function added to an instance, collect a PF MAC and VF logical"},{"line_number":348,"context_line":"  number as seen by the hypervisor host and pass them to Neutron along with"},{"line_number":349,"context_line":"  the card serial number during port update requests that happen during"},{"line_number":350,"context_line":"  instance creation (see the relevant section below for more details);"},{"line_number":351,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"1ea1d846_3beeac07","line":348,"range":{"start_line":347,"start_character":63,"end_line":348,"end_character":8},"updated":"2021-05-07 12:08:25.000000000","message":"the logical vf number should be sorted in the extra_info table i think although we could look it up.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":344,"context_line":"  network capabilities dict of a device so that it gets persisted to extra_info"},{"line_number":345,"context_line":"  of a PciDevice as follows:"},{"line_number":346,"context_line":"  ``extra_info: \u0027{\"capabilities\": \"network\": [\"transportnode\"]\u0027}``"},{"line_number":347,"context_line":"* For each function added to an instance, collect a PF MAC and VF logical"},{"line_number":348,"context_line":"  number as seen by the hypervisor host and pass them to Neutron along with"},{"line_number":349,"context_line":"  the card serial number during port update requests that happen during"},{"line_number":350,"context_line":"  instance creation (see the relevant section below for more details);"},{"line_number":351,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"7b4e69a8_9d8063a0","line":348,"range":{"start_line":347,"start_character":63,"end_line":348,"end_character":8},"in_reply_to":"1ea1d846_3beeac07","updated":"2021-05-11 13:17:45.000000000","message":"If it stays persistent across reboots, maybe we could.\n\nThe numbering is tied to a PF and starts from 0 for each PF.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":344,"context_line":"  network capabilities dict of a device so that it gets persisted to extra_info"},{"line_number":345,"context_line":"  of a PciDevice as follows:"},{"line_number":346,"context_line":"  ``extra_info: \u0027{\"capabilities\": \"network\": [\"transportnode\"]\u0027}``"},{"line_number":347,"context_line":"* For each function added to an instance, collect a PF MAC and VF logical"},{"line_number":348,"context_line":"  number as seen by the hypervisor host and pass them to Neutron along with"},{"line_number":349,"context_line":"  the card serial number during port update requests that happen during"},{"line_number":350,"context_line":"  instance creation (see the relevant section below for more details);"},{"line_number":351,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"e059cdd1_41b7414c","line":348,"range":{"start_line":347,"start_character":63,"end_line":348,"end_character":8},"in_reply_to":"7b4e69a8_9d8063a0","updated":"2021-05-21 19:01:48.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":358,"context_line":"proposed to be sent during a port update:"},{"line_number":359,"context_line":""},{"line_number":360,"context_line":"* card serial number;"},{"line_number":361,"context_line":"* PF mac address (seen both by the hypervisor and the transport node);"},{"line_number":362,"context_line":"* VF logical number."},{"line_number":363,"context_line":""},{"line_number":364,"context_line":"This is needed to do the following multiplexing decisions:"}],"source_content_type":"text/x-rst","patch_set":4,"id":"40bb77fe_998e7d6e","line":361,"range":{"start_line":361,"start_character":1,"end_line":361,"end_character":70},"updated":"2021-05-07 12:08:25.000000000","message":"if we can avoid the mac i would prefer too but if this is the only identifer we can use i guess its ok.\n\nthe mac for vms is not reliable which is why we do not trust it in general.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1933e7abc61d9926e1fb5cba9bb19aac9ef7ff52","unresolved":false,"context_lines":[{"line_number":358,"context_line":"proposed to be sent during a port update:"},{"line_number":359,"context_line":""},{"line_number":360,"context_line":"* card serial number;"},{"line_number":361,"context_line":"* PF mac address (seen both by the hypervisor and the transport node);"},{"line_number":362,"context_line":"* VF logical number."},{"line_number":363,"context_line":""},{"line_number":364,"context_line":"This is needed to do the following multiplexing decisions:"}],"source_content_type":"text/x-rst","patch_set":4,"id":"eb9d8bad_1b2b3643","line":361,"range":{"start_line":361,"start_character":1,"end_line":361,"end_character":70},"in_reply_to":"2f5dc6b4_2decdfbc","updated":"2021-05-18 12:45:55.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":358,"context_line":"proposed to be sent during a port update:"},{"line_number":359,"context_line":""},{"line_number":360,"context_line":"* card serial number;"},{"line_number":361,"context_line":"* PF mac address (seen both by the hypervisor and the transport node);"},{"line_number":362,"context_line":"* VF logical number."},{"line_number":363,"context_line":""},{"line_number":364,"context_line":"This is needed to do the following multiplexing decisions:"}],"source_content_type":"text/x-rst","patch_set":4,"id":"2f5dc6b4_2decdfbc","line":361,"range":{"start_line":361,"start_character":1,"end_line":361,"end_character":70},"in_reply_to":"40bb77fe_998e7d6e","updated":"2021-05-11 13:17:45.000000000","message":"This is for the case where VFs are enabled and we don\u0027t use PF passthrough. PF MAC should remain untouched in this case but it\u0027s needed to identify a correct representor.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":363,"context_line":""},{"line_number":364,"context_line":"This is needed to do the following multiplexing decisions:"},{"line_number":365,"context_line":""},{"line_number":366,"context_line":"* Determining the right transport node hostname which needs to handle a"},{"line_number":367,"context_line":"  representor plugging - there may be multiple transport nodes per physical"},{"line_number":368,"context_line":"  host. This can be done by associating a card serial number with a transport"},{"line_number":369,"context_line":"  node hostname at the Neutron \u0026 OVN side (Nova just needs to pass it with"}],"source_content_type":"text/x-rst","patch_set":4,"id":"2c2678db_7498ab76","line":366,"range":{"start_line":366,"start_character":24,"end_line":366,"end_character":38},"updated":"2021-05-07 12:08:25.000000000","message":"smartnic","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":false,"context_lines":[{"line_number":363,"context_line":""},{"line_number":364,"context_line":"This is needed to do the following multiplexing decisions:"},{"line_number":365,"context_line":""},{"line_number":366,"context_line":"* Determining the right transport node hostname which needs to handle a"},{"line_number":367,"context_line":"  representor plugging - there may be multiple transport nodes per physical"},{"line_number":368,"context_line":"  host. This can be done by associating a card serial number with a transport"},{"line_number":369,"context_line":"  node hostname at the Neutron \u0026 OVN side (Nova just needs to pass it with"}],"source_content_type":"text/x-rst","patch_set":4,"id":"61674227_0fd65674","line":366,"range":{"start_line":366,"start_character":24,"end_line":366,"end_character":38},"in_reply_to":"2c2678db_7498ab76","updated":"2021-05-11 13:17:45.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":364,"context_line":"This is needed to do the following multiplexing decisions:"},{"line_number":365,"context_line":""},{"line_number":366,"context_line":"* Determining the right transport node hostname which needs to handle a"},{"line_number":367,"context_line":"  representor plugging - there may be multiple transport nodes per physical"},{"line_number":368,"context_line":"  host. This can be done by associating a card serial number with a transport"},{"line_number":369,"context_line":"  node hostname at the Neutron \u0026 OVN side (Nova just needs to pass it with"},{"line_number":370,"context_line":"  a port update);"}],"source_content_type":"text/x-rst","patch_set":4,"id":"d4c5225b_19e28b44","line":367,"range":{"start_line":367,"start_character":47,"end_line":367,"end_character":62},"updated":"2021-05-07 12:08:25.000000000","message":"smartnics","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":false,"context_lines":[{"line_number":364,"context_line":"This is needed to do the following multiplexing decisions:"},{"line_number":365,"context_line":""},{"line_number":366,"context_line":"* Determining the right transport node hostname which needs to handle a"},{"line_number":367,"context_line":"  representor plugging - there may be multiple transport nodes per physical"},{"line_number":368,"context_line":"  host. This can be done by associating a card serial number with a transport"},{"line_number":369,"context_line":"  node hostname at the Neutron \u0026 OVN side (Nova just needs to pass it with"},{"line_number":370,"context_line":"  a port update);"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7911fe29_62837bb8","line":367,"range":{"start_line":367,"start_character":47,"end_line":367,"end_character":62},"in_reply_to":"d4c5225b_19e28b44","updated":"2021-05-11 13:17:45.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":365,"context_line":""},{"line_number":366,"context_line":"* Determining the right transport node hostname which needs to handle a"},{"line_number":367,"context_line":"  representor plugging - there may be multiple transport nodes per physical"},{"line_number":368,"context_line":"  host. This can be done by associating a card serial number with a transport"},{"line_number":369,"context_line":"  node hostname at the Neutron \u0026 OVN side (Nova just needs to pass it with"},{"line_number":370,"context_line":"  a port update);"},{"line_number":371,"context_line":"* Picking the right eSwitch at the transport node side. PF logical numbers"},{"line_number":372,"context_line":"  are tied to controllers [10]_ [12]_. Typically there is a single NIC and"}],"source_content_type":"text/x-rst","patch_set":4,"id":"a0b47876_265b9ef7","line":369,"range":{"start_line":368,"start_character":68,"end_line":369,"end_character":6},"updated":"2021-05-07 12:08:25.000000000","message":"smartnic","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":false,"context_lines":[{"line_number":365,"context_line":""},{"line_number":366,"context_line":"* Determining the right transport node hostname which needs to handle a"},{"line_number":367,"context_line":"  representor plugging - there may be multiple transport nodes per physical"},{"line_number":368,"context_line":"  host. This can be done by associating a card serial number with a transport"},{"line_number":369,"context_line":"  node hostname at the Neutron \u0026 OVN side (Nova just needs to pass it with"},{"line_number":370,"context_line":"  a port update);"},{"line_number":371,"context_line":"* Picking the right eSwitch at the transport node side. PF logical numbers"},{"line_number":372,"context_line":"  are tied to controllers [10]_ [12]_. Typically there is a single NIC and"}],"source_content_type":"text/x-rst","patch_set":4,"id":"80f93e23_e5b93a54","line":369,"range":{"start_line":368,"start_character":68,"end_line":369,"end_character":6},"in_reply_to":"a0b47876_265b9ef7","updated":"2021-05-11 13:17:45.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":368,"context_line":"  host. This can be done by associating a card serial number with a transport"},{"line_number":369,"context_line":"  node hostname at the Neutron \u0026 OVN side (Nova just needs to pass it with"},{"line_number":370,"context_line":"  a port update);"},{"line_number":371,"context_line":"* Picking the right eSwitch at the transport node side. PF logical numbers"},{"line_number":372,"context_line":"  are tied to controllers [10]_ [12]_. Typically there is a single NIC and"},{"line_number":373,"context_line":"  eSwitch in a SmartNIC but there is no guarantee that there will not be a"},{"line_number":374,"context_line":"  device with multiple of those. As a result, just passing a PF logical number"}],"source_content_type":"text/x-rst","patch_set":4,"id":"0c4899f9_113e7229","line":371,"range":{"start_line":371,"start_character":20,"end_line":371,"end_character":27},"updated":"2021-05-07 12:08:25.000000000","message":"so the eswitch has an id can we use that instead of the mac to identify the eswitch?\n\nhttps://github.com/openstack/os-vif/blob/d8af3568b8b92748f61029a96c46fd513b6795c2/vif_plug_ovs/linux_net.py#L411-L423\n\ndef _get_phys_switch_id(ifname):\n    \"\"\"Get the interface name and return its phys_switch_id\n    :param ifname: The interface name\n    :return: The phys_switch_id of the given ifname\n    \"\"\"\n    phys_port_name_path \u003d \"/sys/class/net/%s/phys_switch_id\" % ifname\n\n    if not os.path.isfile(phys_port_name_path):\n        return None\n\n    with open(phys_port_name_path, \u0027r\u0027) as fd:\n        return fd.readline().strip()","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":368,"context_line":"  host. This can be done by associating a card serial number with a transport"},{"line_number":369,"context_line":"  node hostname at the Neutron \u0026 OVN side (Nova just needs to pass it with"},{"line_number":370,"context_line":"  a port update);"},{"line_number":371,"context_line":"* Picking the right eSwitch at the transport node side. PF logical numbers"},{"line_number":372,"context_line":"  are tied to controllers [10]_ [12]_. Typically there is a single NIC and"},{"line_number":373,"context_line":"  eSwitch in a SmartNIC but there is no guarantee that there will not be a"},{"line_number":374,"context_line":"  device with multiple of those. As a result, just passing a PF logical number"}],"source_content_type":"text/x-rst","patch_set":4,"id":"bce0dbe1_8845e975","line":371,"range":{"start_line":371,"start_character":20,"end_line":371,"end_character":27},"in_reply_to":"0c4899f9_113e7229","updated":"2021-05-11 13:17:45.000000000","message":"(copied from the comment above)\n\nUnfortunately, we cannot rely on the switchdev ID since it is not visible from the hypervisor host.\n\n$ sudo devlink dev eswitch show pci/0000:82:00.0\ndevlink answers: Operation not supported\n\nsudo cat /sys/bus/pci/devices/0000\\:82\\:00.0/net/enp3s0f0/phys_switch_id \ncat: \u0027/sys/bus/pci/devices/0000:82:00.0/net/enp3s0f0/phys_switch_id\u0027: Operation not supported\n\n\nsudo lspci -v | grep -i mella\n82:00.0 Ethernet controller: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller (rev 01)\n\n\tSubsystem: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller\n82:00.1 Ethernet controller: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller (rev 01)\n\tSubsystem: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller\n82:00.2 DMA controller: Mellanox Technologies MT42822 BlueField-2 SoC Management Interface (rev 01) (prog-if 00 [8237])\n\tSubsystem: Mellanox Technologies MT42822 BlueField-2 SoC Management Interface\n82:00.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)\n\tSubsystem: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function\n82:00.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)\n\tSubsystem: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function\n82:00.5 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)\n\tSubsystem: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function\n82:00.6 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)\n\tSubsystem: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function\nsudo cat /sys/bus/pci/devices/0000\\:82\\:00.0/net/enp3s0f0/phys_switch_id \ncat: \u0027/sys/bus/pci/devices/0000:82:00.0/net/enp3s0f0/phys_switch_id\u0027: Operation not supported\n\nsudo cat /sys/bus/pci/devices/0000\\:82\\:00.3/net/enp130s0f0np0v0/phys_switch_id \ncat: \u0027/sys/bus/pci/devices/0000:82:00.3/net/enp130s0f0np0v0/phys_switch_id\u0027: Operation not supported\n\n\nPF numbers are scoped to controllers at the SmartNIC DPU host side so PF numbers themselves are not enough to identify a PF.\n\nhttps://github.com/torvalds/linux/blob/v5.12/net/core/devlink.c#L8649-L8650","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1933e7abc61d9926e1fb5cba9bb19aac9ef7ff52","unresolved":false,"context_lines":[{"line_number":368,"context_line":"  host. This can be done by associating a card serial number with a transport"},{"line_number":369,"context_line":"  node hostname at the Neutron \u0026 OVN side (Nova just needs to pass it with"},{"line_number":370,"context_line":"  a port update);"},{"line_number":371,"context_line":"* Picking the right eSwitch at the transport node side. PF logical numbers"},{"line_number":372,"context_line":"  are tied to controllers [10]_ [12]_. Typically there is a single NIC and"},{"line_number":373,"context_line":"  eSwitch in a SmartNIC but there is no guarantee that there will not be a"},{"line_number":374,"context_line":"  device with multiple of those. As a result, just passing a PF logical number"}],"source_content_type":"text/x-rst","patch_set":4,"id":"61b20a5e_66185e9f","line":371,"range":{"start_line":371,"start_character":20,"end_line":371,"end_character":27},"in_reply_to":"bce0dbe1_8845e975","updated":"2021-05-18 12:45:55.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":378,"context_line":"* Choosing the right VF representor - a VF logical number tied to a particular"},{"line_number":379,"context_line":"  PF."},{"line_number":380,"context_line":""},{"line_number":381,"context_line":"PF and controller numbers seen by the transport node are not visible from the"},{"line_number":382,"context_line":"hypervisor host since it does not see the eSwitch. To further expand on this,"},{"line_number":383,"context_line":"the devlink [11]_ infrastructure in the kernel supports different port flavors"},{"line_number":384,"context_line":"(quoted descriptions originate from linux/uapi/linux/devlink.h [13]_):"},{"line_number":385,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"c585e709_9ee67d8a","line":382,"range":{"start_line":381,"start_character":0,"end_line":382,"end_character":50},"updated":"2021-05-07 12:08:25.000000000","message":"if this is really the case i keep wonder ign if we shoudl have a sperate vnic_type for this instead of direct.\n\ngiven i think we shoudl start requriedign the setich dev capablitry whenever vnic_type direct is used in combindation with vif_type ovs for the security isssues related to ovn today","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1933e7abc61d9926e1fb5cba9bb19aac9ef7ff52","unresolved":true,"context_lines":[{"line_number":378,"context_line":"* Choosing the right VF representor - a VF logical number tied to a particular"},{"line_number":379,"context_line":"  PF."},{"line_number":380,"context_line":""},{"line_number":381,"context_line":"PF and controller numbers seen by the transport node are not visible from the"},{"line_number":382,"context_line":"hypervisor host since it does not see the eSwitch. To further expand on this,"},{"line_number":383,"context_line":"the devlink [11]_ infrastructure in the kernel supports different port flavors"},{"line_number":384,"context_line":"(quoted descriptions originate from linux/uapi/linux/devlink.h [13]_):"},{"line_number":385,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"8b9bfcdf_85dcd9e5","line":382,"range":{"start_line":381,"start_character":0,"end_line":382,"end_character":50},"in_reply_to":"56abe2f8_986f2ef7","updated":"2021-05-18 12:45:55.000000000","message":"right we cant use the binding profile to multiplex like that today and since its admin only i dont think that is a good idea in general.\n\nwhat i suggested in an eilar comment was either the new vnic type or a generic traits extension ot neutron that would allow you to ask for arbitrary traits.\ni think we need to think about this carfully and see if there is a generic solution that we can use.\n\nthe binding profile is intended to pass inform from nova to the netwroking backend \nhttps://github.com/openstack/neutron-lib/blob/master/neutron_lib/api/definitions/portbindings.py#L31-L34\nits really not intended to be used by an end user. it can be used to enable trusted_vfs to day which are generally a security risk so we can relax the admin_only default policy for the binding profile without creating a security issue.\n\ni will see if i can incoperate a way to pass traits request into my generic pci in placment spec that you can reuse for this usecase if you have time to review that.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":378,"context_line":"* Choosing the right VF representor - a VF logical number tied to a particular"},{"line_number":379,"context_line":"  PF."},{"line_number":380,"context_line":""},{"line_number":381,"context_line":"PF and controller numbers seen by the transport node are not visible from the"},{"line_number":382,"context_line":"hypervisor host since it does not see the eSwitch. To further expand on this,"},{"line_number":383,"context_line":"the devlink [11]_ infrastructure in the kernel supports different port flavors"},{"line_number":384,"context_line":"(quoted descriptions originate from linux/uapi/linux/devlink.h [13]_):"},{"line_number":385,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"951665ff_6d87718a","line":382,"range":{"start_line":381,"start_character":0,"end_line":382,"end_character":50},"in_reply_to":"8b9bfcdf_85dcd9e5","updated":"2021-05-21 19:01:48.000000000","message":"Will change the spec to use a different vnic type (VNIC_SMARTNIC). Meanwhile, will look into the generic traits specification because it looks like a good approach (also noted the same in one of the comments above).","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":378,"context_line":"* Choosing the right VF representor - a VF logical number tied to a particular"},{"line_number":379,"context_line":"  PF."},{"line_number":380,"context_line":""},{"line_number":381,"context_line":"PF and controller numbers seen by the transport node are not visible from the"},{"line_number":382,"context_line":"hypervisor host since it does not see the eSwitch. To further expand on this,"},{"line_number":383,"context_line":"the devlink [11]_ infrastructure in the kernel supports different port flavors"},{"line_number":384,"context_line":"(quoted descriptions originate from linux/uapi/linux/devlink.h [13]_):"},{"line_number":385,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"56abe2f8_986f2ef7","line":382,"range":{"start_line":381,"start_character":0,"end_line":382,"end_character":50},"in_reply_to":"c585e709_9ee67d8a","updated":"2021-05-11 13:17:45.000000000","message":"Maybe we could - just wanted to reuse the existing vnic_type and multiplex via a capability stored in \"binding:profile\" instead.\n\nI suggested \"transportnode\" initially but it could be \"nic-switch-port\" or something like that instead to distinguish with \"switchdev\".\n\nNeed to explore what using a different type would entail.\n\n\u003d\u003d\u003d\u003d\u003d\n(Copied from the comment above)\n\n$ sudo devlink dev eswitch show pci/0000:82:00.0\ndevlink answers: Operation not supported\n\nsudo cat /sys/bus/pci/devices/0000\\:82\\:00.0/net/enp3s0f0/phys_switch_id \ncat: \u0027/sys/bus/pci/devices/0000:82:00.0/net/enp3s0f0/phys_switch_id\u0027: Operation not supported\n\n\nsudo lspci -v | grep -i mella\n82:00.0 Ethernet controller: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller (rev 01)\n\n\tSubsystem: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller\n82:00.1 Ethernet controller: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller (rev 01)\n\tSubsystem: Mellanox Technologies MT42822 BlueField-2 integrated ConnectX-6 Dx network controller\n82:00.2 DMA controller: Mellanox Technologies MT42822 BlueField-2 SoC Management Interface (rev 01) (prog-if 00 [8237])\n\tSubsystem: Mellanox Technologies MT42822 BlueField-2 SoC Management Interface\n82:00.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)\n\tSubsystem: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function\n82:00.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)\n\tSubsystem: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function\n82:00.5 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)\n\tSubsystem: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function\n82:00.6 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function (rev 01)\n\tSubsystem: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function\nsudo cat /sys/bus/pci/devices/0000\\:82\\:00.0/net/enp3s0f0/phys_switch_id \ncat: \u0027/sys/bus/pci/devices/0000:82:00.0/net/enp3s0f0/phys_switch_id\u0027: Operation not supported\n\nsudo cat /sys/bus/pci/devices/0000\\:82\\:00.3/net/enp130s0f0np0v0/phys_switch_id \ncat: \u0027/sys/bus/pci/devices/0000:82:00.3/net/enp130s0f0np0v0/phys_switch_id\u0027: Operation not supported\n\n\nPF numbers are scoped to controllers at the SmartNIC DPU host side so PF numbers themselves are not enough to identify a PF.\n\nhttps://github.com/torvalds/linux/blob/v5.12/net/core/devlink.c#L8649-L8650","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":403,"context_line":"both uplink ports at the hypervisor side, the set of interfaces as shown by"},{"line_number":404,"context_line":"``devlink port`` will be as follows::"},{"line_number":405,"context_line":""},{"line_number":406,"context_line":"  pci/0000:05:00.0/1: type eth netdev enp5s0f0 flavour physical port 0"},{"line_number":407,"context_line":"  pci/0000:05:00.1/1: type eth netdev enp5s0f1 flavour physical port 1"},{"line_number":408,"context_line":"  pci/0000:05:02.3/1: type eth netdev enp5s0f1np0v0 flavour virtual port 0"},{"line_number":409,"context_line":"  pci/0000:05:02.4/1: type eth netdev enp5s0f1np0v1 flavour virtual port 0"}],"source_content_type":"text/x-rst","patch_set":4,"id":"27ce1185_8519d1b1","line":406,"range":{"start_line":406,"start_character":2,"end_line":406,"end_character":70},"updated":"2021-05-07 12:08:25.000000000","message":"we can use this info to get teh physical port number instead of usign the mac correct?","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":403,"context_line":"both uplink ports at the hypervisor side, the set of interfaces as shown by"},{"line_number":404,"context_line":"``devlink port`` will be as follows::"},{"line_number":405,"context_line":""},{"line_number":406,"context_line":"  pci/0000:05:00.0/1: type eth netdev enp5s0f0 flavour physical port 0"},{"line_number":407,"context_line":"  pci/0000:05:00.1/1: type eth netdev enp5s0f1 flavour physical port 1"},{"line_number":408,"context_line":"  pci/0000:05:02.3/1: type eth netdev enp5s0f1np0v0 flavour virtual port 0"},{"line_number":409,"context_line":"  pci/0000:05:02.4/1: type eth netdev enp5s0f1np0v1 flavour virtual port 0"}],"source_content_type":"text/x-rst","patch_set":4,"id":"9bf3f536_b5a05ebc","line":406,"range":{"start_line":406,"start_character":2,"end_line":406,"end_character":70},"in_reply_to":"27ce1185_8519d1b1","updated":"2021-05-11 13:17:45.000000000","message":"PF numbers are relative to controller numbers and we do not see those from the hypervisor host side since it does not see the eSwitch.\n\nIf there was more than one controller, PF numbers alone would be rendered meaningless without a controller number.\n\nhttps://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id\u003d66b17082d10a3b806eec3da8fdebe8a9cd2c6612\nhttps://github.com/torvalds/linux/blob/v5.12/net/core/devlink.c#L8648-L8668 (controller numbers may be used as a port of a phys_port_name port attribute in the kernel)\n\nI wish we could simply use PF numbers but there might be hardware with multiple controllers at some point.\n\nI would rather factor this into the design early and use the data that uniquely identifies a PF from both sides.\n\nSince the default PF MAC address can be driven from the SmartNIC host it seems like a useful primitive for this.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1933e7abc61d9926e1fb5cba9bb19aac9ef7ff52","unresolved":false,"context_lines":[{"line_number":403,"context_line":"both uplink ports at the hypervisor side, the set of interfaces as shown by"},{"line_number":404,"context_line":"``devlink port`` will be as follows::"},{"line_number":405,"context_line":""},{"line_number":406,"context_line":"  pci/0000:05:00.0/1: type eth netdev enp5s0f0 flavour physical port 0"},{"line_number":407,"context_line":"  pci/0000:05:00.1/1: type eth netdev enp5s0f1 flavour physical port 1"},{"line_number":408,"context_line":"  pci/0000:05:02.3/1: type eth netdev enp5s0f1np0v0 flavour virtual port 0"},{"line_number":409,"context_line":"  pci/0000:05:02.4/1: type eth netdev enp5s0f1np0v1 flavour virtual port 0"}],"source_content_type":"text/x-rst","patch_set":4,"id":"3adabb5a_1d80faa2","line":406,"range":{"start_line":406,"start_character":2,"end_line":406,"end_character":70},"in_reply_to":"9bf3f536_b5a05ebc","updated":"2021-05-18 12:45:55.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":416,"context_line":""},{"line_number":417,"context_line":"Notice the virtual port indexes are all set to 0 - in this example the device"},{"line_number":418,"context_line":"driver does not provide any indexing information via devlink attributes for"},{"line_number":419,"context_line":"\"virtual\" ports."},{"line_number":420,"context_line":""},{"line_number":421,"context_line":"Transport node host ``devlink port`` output::"},{"line_number":422,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"060b41df_d1c34803","line":419,"updated":"2021-05-07 12:08:25.000000000","message":"can we retive the VF logical number form sysfs or netlink as we do today.\n\nhttps://github.com/openstack/os-vif/blob/d8af3568b8b92748f61029a96c46fd513b6795c2/vif_plug_ovs/linux_net.py#L360-L378","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":416,"context_line":""},{"line_number":417,"context_line":"Notice the virtual port indexes are all set to 0 - in this example the device"},{"line_number":418,"context_line":"driver does not provide any indexing information via devlink attributes for"},{"line_number":419,"context_line":"\"virtual\" ports."},{"line_number":420,"context_line":""},{"line_number":421,"context_line":"Transport node host ``devlink port`` output::"},{"line_number":422,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"2b7b8422_b160a4a3","line":419,"in_reply_to":"060b41df_d1c34803","updated":"2021-05-11 13:17:45.000000000","message":"Yes, that\u0027s how I planned to do it - this provides us with numbers relative to a PF.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1933e7abc61d9926e1fb5cba9bb19aac9ef7ff52","unresolved":false,"context_lines":[{"line_number":416,"context_line":""},{"line_number":417,"context_line":"Notice the virtual port indexes are all set to 0 - in this example the device"},{"line_number":418,"context_line":"driver does not provide any indexing information via devlink attributes for"},{"line_number":419,"context_line":"\"virtual\" ports."},{"line_number":420,"context_line":""},{"line_number":421,"context_line":"Transport node host ``devlink port`` output::"},{"line_number":422,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"bcff06c4_45a9a261","line":419,"in_reply_to":"2b7b8422_b160a4a3","updated":"2021-05-18 12:45:55.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":449,"context_line":""},{"line_number":450,"context_line":"Finding the right entry via a physfn symlink can be done by resolving virtfn"},{"line_number":451,"context_line":"symlinks one by one and comparing the result with the ``vf_pci_addr`` that"},{"line_number":452,"context_line":"is of interest."},{"line_number":453,"context_line":""},{"line_number":454,"context_line":"As for finding the right PF representor by a MAC address of hypervisor host PF,"},{"line_number":455,"context_line":"it depends on the availability of information about a mapping of a hypervisor"}],"source_content_type":"text/x-rst","patch_set":4,"id":"a5255b6c_f1e64f21","line":452,"updated":"2021-05-07 12:08:25.000000000","message":"yes we already do this in os-vif.\n\nwe might be able to move this out of the ovs plugin in os-vif and into a public part of the lib or we can copy the code we already have into nova \n\nhttps://github.com/openstack/os-vif/blob/d8af3568b8b92748f61029a96c46fd513b6795c2/vif_plug_ovs/linux_net.py#L360-L378","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":449,"context_line":""},{"line_number":450,"context_line":"Finding the right entry via a physfn symlink can be done by resolving virtfn"},{"line_number":451,"context_line":"symlinks one by one and comparing the result with the ``vf_pci_addr`` that"},{"line_number":452,"context_line":"is of interest."},{"line_number":453,"context_line":""},{"line_number":454,"context_line":"As for finding the right PF representor by a MAC address of hypervisor host PF,"},{"line_number":455,"context_line":"it depends on the availability of information about a mapping of a hypervisor"}],"source_content_type":"text/x-rst","patch_set":4,"id":"c35f2ba7_21353f55","line":452,"in_reply_to":"a5255b6c_f1e64f21","updated":"2021-05-11 13:17:45.000000000","message":"Ack, looks exactly like what we need.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":454,"context_line":"As for finding the right PF representor by a MAC address of hypervisor host PF,"},{"line_number":455,"context_line":"it depends on the availability of information about a mapping of a hypervisor"},{"line_number":456,"context_line":"PF MAC to a PF representor MAC."},{"line_number":457,"context_line":""},{"line_number":458,"context_line":"VF logical number and PF MAC information can be extracted at runtime right"},{"line_number":459,"context_line":"before a port update since those are done by the Nova Compute manager during"},{"line_number":460,"context_line":"instance creation."}],"source_content_type":"text/x-rst","patch_set":4,"id":"1d3719bf_9d9c5a8d","line":457,"updated":"2021-05-07 12:08:25.000000000","message":"again while we can use the mac can we not jus tuse the eswitch id\nlocated at\nphys_port_name_path \u003d \"/sys/class/net/%s/phys_switch_id\" % ifname\n\nhttps://github.com/openstack/os-vif/blob/d8af3568b8b92748f61029a96c46fd513b6795c2/vif_plug_ovs/linux_net.py#L411-L423","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1933e7abc61d9926e1fb5cba9bb19aac9ef7ff52","unresolved":true,"context_lines":[{"line_number":454,"context_line":"As for finding the right PF representor by a MAC address of hypervisor host PF,"},{"line_number":455,"context_line":"it depends on the availability of information about a mapping of a hypervisor"},{"line_number":456,"context_line":"PF MAC to a PF representor MAC."},{"line_number":457,"context_line":""},{"line_number":458,"context_line":"VF logical number and PF MAC information can be extracted at runtime right"},{"line_number":459,"context_line":"before a port update since those are done by the Nova Compute manager during"},{"line_number":460,"context_line":"instance creation."}],"source_content_type":"text/x-rst","patch_set":4,"id":"cc8db0f4_c5972f4e","line":457,"in_reply_to":"1d2d46d0_3a7dbb92","updated":"2021-05-18 12:45:55.000000000","message":"what are the chanes we could ask melonox and other to implement that api?\nwe have done that in the past but we obviously do not have to block this on that support.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":454,"context_line":"As for finding the right PF representor by a MAC address of hypervisor host PF,"},{"line_number":455,"context_line":"it depends on the availability of information about a mapping of a hypervisor"},{"line_number":456,"context_line":"PF MAC to a PF representor MAC."},{"line_number":457,"context_line":""},{"line_number":458,"context_line":"VF logical number and PF MAC information can be extracted at runtime right"},{"line_number":459,"context_line":"before a port update since those are done by the Nova Compute manager during"},{"line_number":460,"context_line":"instance creation."}],"source_content_type":"text/x-rst","patch_set":4,"id":"1d2d46d0_3a7dbb92","line":457,"in_reply_to":"1d3719bf_9d9c5a8d","updated":"2021-05-11 13:17:45.000000000","message":"Ah, that\u0027s the core of the problem: the eSwitch is not visible at the hypervisor host.\n\nI wish we could just use the switch ID from both sides but the hypervisor host simply sees interfaces of devlink flavors \"physical\" and \"virtual\" whereas the SmartNIC host sees \"PCI PF\" and \"PCI VF\" ones for representors. So there is no switch ID visible from the hypervisor side.\n\nThe Neutron spec references the following kernel commits that allow PF mac lookup by a PF representor (as their MACs differ):\n\n\"[11] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id\u003d2a916ecc405686c1d86f632281bc06aa75ebae4e\n[12] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id\u003df099fde16db3d2594a54ba8c94ce9fa3557aa3e1\"\n\nSo there is a generic API to obtain this information provided the kernel driver supports it.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":true,"context_lines":[{"line_number":454,"context_line":"As for finding the right PF representor by a MAC address of hypervisor host PF,"},{"line_number":455,"context_line":"it depends on the availability of information about a mapping of a hypervisor"},{"line_number":456,"context_line":"PF MAC to a PF representor MAC."},{"line_number":457,"context_line":""},{"line_number":458,"context_line":"VF logical number and PF MAC information can be extracted at runtime right"},{"line_number":459,"context_line":"before a port update since those are done by the Nova Compute manager during"},{"line_number":460,"context_line":"instance creation."}],"source_content_type":"text/x-rst","patch_set":4,"id":"3eefc514_6366249b","line":457,"in_reply_to":"cc8db0f4_c5972f4e","updated":"2021-05-21 19:01:48.000000000","message":"The problem with the switch ID is that the kernel only guarantees that it will be unique across switches within the same system. If there are multiple SmartNIC DPUs in one server, technically, it\u0027s two different systems:\n\nhttps://www.kernel.org/doc/html/latest/networking/switchdev.html#switch-id\n\"The switchdev driver must implement the net_device operation ndo_get_port_parent_id for each port netdev, returning the same physical ID for each port of a switch. The ID must be unique between switches on the same system. The ID does not need to be unique between switches on different systems.\"\n\nSo, besides this concern alone, I doubt they would want to break isolation in terms of eSwitch awareness at the host level but there is no harm in asking.\n\n\nI have a potential alternative approach though with using a concept of a Hierarchy ID present in PCIe 4.0:\n\n\"7.9.18 Hierarchy ID Extended Capability\nThe Hierarchy ID Extended Capability provides an optional mechanism for passing a unique identifier to Functions within a Hierarchy. At most one instance of this capability is permitted in a Function. This capability is not applicable to Bridges, Root Complex Event Collectors, and RCRBs.\"\n\nhttps://members.pcisig.com/wg/PCI-SIG/document/previewpdf/11116 (should be accessible via corp e-mail for member organizations).\n\nIf the NIC firmware exposed hierarchy IDs for PFs and VFs at the hypervisor host side and allowed querying that via PF and VF representors at the SmartNIC DPU host side, I think we would have a more reliable approach to this problem.\n\nIf this feels sane, maybe it\u0027s worth bringing up with Mellanox/NVIDIA or other vendors as a preferential feature. I can try to do that, however, I think we need to have a base case supported with what we have at this point.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":455,"context_line":"it depends on the availability of information about a mapping of a hypervisor"},{"line_number":456,"context_line":"PF MAC to a PF representor MAC."},{"line_number":457,"context_line":""},{"line_number":458,"context_line":"VF logical number and PF MAC information can be extracted at runtime right"},{"line_number":459,"context_line":"before a port update since those are done by the Nova Compute manager during"},{"line_number":460,"context_line":"instance creation."},{"line_number":461,"context_line":""},{"line_number":462,"context_line":""},{"line_number":463,"context_line":"Alternatives"}],"source_content_type":"text/x-rst","patch_set":4,"id":"83f7f36b_49da066d","line":460,"range":{"start_line":458,"start_character":0,"end_line":460,"end_character":18},"updated":"2021-05-07 12:08:25.000000000","message":"they could or we could do it once at host init and store it in the db.\n\nusing the mac is assuming that the PF of the VF will be bound to a network driver not vfio-pci by the way and that it has a netdev assocaiated with it. will this always be the case as we know its not always the case on all plathforms.\n\nsee https://review.opendev.org/c/openstack/nova/+/777679 and https://bugs.launchpad.net/nova/+bug/1915255","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":455,"context_line":"it depends on the availability of information about a mapping of a hypervisor"},{"line_number":456,"context_line":"PF MAC to a PF representor MAC."},{"line_number":457,"context_line":""},{"line_number":458,"context_line":"VF logical number and PF MAC information can be extracted at runtime right"},{"line_number":459,"context_line":"before a port update since those are done by the Nova Compute manager during"},{"line_number":460,"context_line":"instance creation."},{"line_number":461,"context_line":""},{"line_number":462,"context_line":""},{"line_number":463,"context_line":"Alternatives"}],"source_content_type":"text/x-rst","patch_set":4,"id":"b290ae1f_fd71316b","line":460,"range":{"start_line":458,"start_character":0,"end_line":460,"end_character":18},"in_reply_to":"83f7f36b_49da066d","updated":"2021-05-21 19:01:48.000000000","message":"Ack, made a comment above and will add a note to the spec that there will be a requirement that vfio-pci is not used for PFs (only VFs).","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":479,"context_line":"  needs to concentrate on reusability. This is why possible an introduction of"},{"line_number":480,"context_line":"  transport node services specific to OpenStack needs to be avoided (i.e. it is"},{"line_number":481,"context_line":"  better to extend OVN to do that and handle VF plugging at the Nova side)."},{"line_number":482,"context_line":""},{"line_number":483,"context_line":"One alternative approach involves tracking cards using a separate service with"},{"line_number":484,"context_line":"its own API and possibly introducing a different VNIC type: this does not have"},{"line_number":485,"context_line":"a benefit of code reuse and requires another service to be added and integrated"}],"source_content_type":"text/x-rst","patch_set":4,"id":"d866ddda_af4d61d6","line":482,"updated":"2021-05-07 12:08:25.000000000","message":"that more or less makes sense.\n\n\"port representor plugging\" or \"port plugging\" in the context of nova has a specific meaning by the way which is the act of adding the port to the contolplane of the network backend.\n\nso ovs-vsctl add-port ... or its equvalent.\n\nits where we assocate a networking device on the host with a logical neutron port.\nthat its generally perfromed by nova and when the networking backend completes wiring it up on its end we expect neutron to notify nova of its completeion via the networking-vif-plugged external event.\n\nso port representor plugging is done on the compute host by defintion but the eSwitch programming is not nessisarly done on the hypervior host.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":496,"context_line":"PciDevices get additional information associated with them without affecting"},{"line_number":497,"context_line":"the DB model:"},{"line_number":498,"context_line":""},{"line_number":499,"context_line":"* a synthetic capability called \"transportnode\" if they match vendor and device"},{"line_number":500,"context_line":"  ids included into the [pci]transport_node_list;"},{"line_number":501,"context_line":"* a \"vpd\" capability which stores the information available in the PCI(e) VPD"},{"line_number":502,"context_line":"  capability (initially, just the board serial number but it may be extended at"},{"line_number":503,"context_line":"  a later point if needed)."}],"source_content_type":"text/x-rst","patch_set":4,"id":"1c42fbfb_e965e867","line":500,"range":{"start_line":499,"start_character":0,"end_line":500,"end_character":49},"updated":"2021-05-07 12:08:25.000000000","message":"im still not conviced this is required","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":496,"context_line":"PciDevices get additional information associated with them without affecting"},{"line_number":497,"context_line":"the DB model:"},{"line_number":498,"context_line":""},{"line_number":499,"context_line":"* a synthetic capability called \"transportnode\" if they match vendor and device"},{"line_number":500,"context_line":"  ids included into the [pci]transport_node_list;"},{"line_number":501,"context_line":"* a \"vpd\" capability which stores the information available in the PCI(e) VPD"},{"line_number":502,"context_line":"  capability (initially, just the board serial number but it may be extended at"},{"line_number":503,"context_line":"  a later point if needed)."}],"source_content_type":"text/x-rst","patch_set":4,"id":"a78847dd_91140c03","line":500,"range":{"start_line":499,"start_character":0,"end_line":500,"end_character":49},"in_reply_to":"1c42fbfb_e965e867","updated":"2021-05-21 19:01:48.000000000","message":"Done (removed)","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":503,"context_line":"  a later point if needed)."},{"line_number":504,"context_line":""},{"line_number":505,"context_line":""},{"line_number":506,"context_line":"Periodic hypervisor resource updates will add newly discovered PciDevices and"},{"line_number":507,"context_line":"get the associated card serial number information. However, old devices will"},{"line_number":508,"context_line":"not get this information without explicit action. Adding new devices to the"},{"line_number":509,"context_line":"[pci]transport_node_list should trigger a rescan of the VPD information for"},{"line_number":510,"context_line":"them."},{"line_number":511,"context_line":""},{"line_number":512,"context_line":""},{"line_number":513,"context_line":"REST API impact"}],"source_content_type":"text/x-rst","patch_set":4,"id":"9f0edc56_12fe7e27","line":510,"range":{"start_line":506,"start_character":1,"end_line":510,"end_character":5},"updated":"2021-05-07 12:08:25.000000000","message":"we do not deiscover pci device in a perodic task today and we shoudl not add one.\n\nwe do it in init host.\n\nthere is a perodic task the queryies the pci devices but it nver updated the db and i dont think we should change that.\n\nignoring the fact i dont think we should add [pci]transport_node_list i really dont like the idea of updating it trigering a rescan.\n\n\nif we want to update it you shoudl restart the nova-compute agent although we really should not expect to do that often so a perodic is overkill. i would be open potentially to having the exsiting pic tracker intialisation re run on SIG_HUP wehn our mutable config values are reloaed and making the pci_whitelist a mutable config option potentially but changing the pci_whitelist on a host with instance is generally a bad idea. we nolonger delete device for the db if they are still in use by vms but in the past it was very easy to corrupt your db entries if you did this so like cpu_dedicated_set we generall do not recommend updateing it on any host with vms.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":503,"context_line":"  a later point if needed)."},{"line_number":504,"context_line":""},{"line_number":505,"context_line":""},{"line_number":506,"context_line":"Periodic hypervisor resource updates will add newly discovered PciDevices and"},{"line_number":507,"context_line":"get the associated card serial number information. However, old devices will"},{"line_number":508,"context_line":"not get this information without explicit action. Adding new devices to the"},{"line_number":509,"context_line":"[pci]transport_node_list should trigger a rescan of the VPD information for"},{"line_number":510,"context_line":"them."},{"line_number":511,"context_line":""},{"line_number":512,"context_line":""},{"line_number":513,"context_line":"REST API impact"}],"source_content_type":"text/x-rst","patch_set":4,"id":"823d441d_c6f7b315","line":510,"range":{"start_line":506,"start_character":1,"end_line":510,"end_character":5},"in_reply_to":"9f0edc56_12fe7e27","updated":"2021-05-21 19:01:48.000000000","message":"Ack, reworked the related parts of the spec a bit to avoid a config option completely.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":513,"context_line":"REST API impact"},{"line_number":514,"context_line":"---------------"},{"line_number":515,"context_line":""},{"line_number":516,"context_line":"N/A"},{"line_number":517,"context_line":""},{"line_number":518,"context_line":"Security impact"},{"line_number":519,"context_line":"---------------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"50b4c800_1699bd33","line":516,"range":{"start_line":516,"start_character":0,"end_line":516,"end_character":3},"updated":"2021-05-07 12:08:25.000000000","message":"a new exception and 400 error code will be returned stating that live migration is not supported with remote smartnics.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":513,"context_line":"REST API impact"},{"line_number":514,"context_line":"---------------"},{"line_number":515,"context_line":""},{"line_number":516,"context_line":"N/A"},{"line_number":517,"context_line":""},{"line_number":518,"context_line":"Security impact"},{"line_number":519,"context_line":"---------------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"db053711_fdc628f4","line":516,"range":{"start_line":516,"start_character":0,"end_line":516,"end_character":3},"in_reply_to":"50b4c800_1699bd33","updated":"2021-05-21 19:01:48.000000000","message":"Ack, will look into whether live migration can easily be supported using the existing SR-IOV flow. If not, will add this note.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":528,"context_line":"Other end user impact"},{"line_number":529,"context_line":"---------------------"},{"line_number":530,"context_line":""},{"line_number":531,"context_line":"* Live migration of instances is a future work item for various reasons. It"},{"line_number":532,"context_line":"  involves using vDPA in some form (hardware or software) or the net_failover"},{"line_number":533,"context_line":"  driver and additional non-trivial interactions to re-bind a port to a"},{"line_number":534,"context_line":"  different transport node."},{"line_number":535,"context_line":""},{"line_number":536,"context_line":"Performance Impact"},{"line_number":537,"context_line":"------------------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"4d6c1a86_9de3d487","line":534,"range":{"start_line":531,"start_character":73,"end_line":534,"end_character":27},"updated":"2021-05-07 12:08:25.000000000","message":"well we already have livemigration with VFs today.\nif you use vnic_type direct we hotunplug the vf before migrating and attach the new vf on the remote side.\nif you use vnic_type macvtap we can supprot transpartent sriov live migration today.\n\nif you do not want to supprot this its fine but i woudl like you to call this out explcitly in the alternitive sections with why this will not work in this case.\n\nhttps://specs.openstack.org/openstack/nova-specs/specs/stein/approved/libvirt-neutron-sriov-livemigration.html\n\nI do not have time to do it in this cycle is also a valid reason by the way :)\n\nwe do need to call this out in the rest api impact section and explity block it since its a functional regresion compoare to hardware offloaded ovs or sriov as we have them today.\nin general i think we should keep parity and support it the same way that we do for sriov if we can.\n\nvdpa will use the vnic_type vdpa not direct and form a nova point of view if the contol plane is run on the hypervior host or on the smartnics should be transparent in terms of live migration.\nonce qemu gains support for vdpa live migration it should work transparenetly the same in both cases.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":true,"context_lines":[{"line_number":528,"context_line":"Other end user impact"},{"line_number":529,"context_line":"---------------------"},{"line_number":530,"context_line":""},{"line_number":531,"context_line":"* Live migration of instances is a future work item for various reasons. It"},{"line_number":532,"context_line":"  involves using vDPA in some form (hardware or software) or the net_failover"},{"line_number":533,"context_line":"  driver and additional non-trivial interactions to re-bind a port to a"},{"line_number":534,"context_line":"  different transport node."},{"line_number":535,"context_line":""},{"line_number":536,"context_line":"Performance Impact"},{"line_number":537,"context_line":"------------------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"d2193511_7218a0a8","line":534,"range":{"start_line":531,"start_character":73,"end_line":534,"end_character":27},"in_reply_to":"4d6c1a86_9de3d487","updated":"2021-05-21 19:01:48.000000000","message":"I will look more into it and see if we can fit this in. My hope is that the SR-IOV flow you referenced can be reused easily and there will not actually be a lot to do here. Hopefully the Neutron part will be simple enough as well.\n\nBy placing live migration out of scope of the initial implementation I tried to reduce the scope of the spec to a reasonable size - it already touches on many things. But maybe I overestimate the amount of work.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":537,"context_line":"------------------"},{"line_number":538,"context_line":""},{"line_number":539,"context_line":"* Additional steps need to be performed to extract serial number information of"},{"line_number":540,"context_line":"  PCI(e) add-in cards from the PFs and VFs exposed by them."},{"line_number":541,"context_line":""},{"line_number":542,"context_line":"Other deployer impact"},{"line_number":543,"context_line":"---------------------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"031e0a75_5e5d1046","line":540,"updated":"2021-05-07 12:08:25.000000000","message":"yep technically true although this hsoudl only be required at agent start so in general i think this will be minor.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":537,"context_line":"------------------"},{"line_number":538,"context_line":""},{"line_number":539,"context_line":"* Additional steps need to be performed to extract serial number information of"},{"line_number":540,"context_line":"  PCI(e) add-in cards from the PFs and VFs exposed by them."},{"line_number":541,"context_line":""},{"line_number":542,"context_line":"Other deployer impact"},{"line_number":543,"context_line":"---------------------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"9a6d3852_f89029ec","line":540,"in_reply_to":"031e0a75_5e5d1046","updated":"2021-05-21 19:01:48.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":542,"context_line":"Other deployer impact"},{"line_number":543,"context_line":"---------------------"},{"line_number":544,"context_line":""},{"line_number":545,"context_line":"* [pci]transport_node_list config option is added which allows one to specify"},{"line_number":546,"context_line":"  device and vendor IDs of PCI(e) devices that will be considered as the ones"},{"line_number":547,"context_line":"  associated with a transport node;"},{"line_number":548,"context_line":"* Having the ``devlink`` binary is not required in a Linux distribution of"}],"source_content_type":"text/x-rst","patch_set":4,"id":"b3796ffe_1bd0ee8a","line":545,"range":{"start_line":545,"start_character":2,"end_line":545,"end_character":26},"updated":"2021-05-07 12:08:25.000000000","message":"again i dont think this is needed. if we need to mark which pci device shoudl have the transport node capablity/trait then i think we shoudl extend the pci whitelist to allow generic placement traits or use provider.yaml we could also jsut add a simple remote_managed\u003dTrue|False tag to the pci whitelist similar to the trusted_vf tag.\n\nthat is simpler to configure then a parallel configuration option.\nhttps://github.com/openstack/nova/commit/88e21d8e5e9abf2c0aba42aabbac3d21e750140c","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":542,"context_line":"Other deployer impact"},{"line_number":543,"context_line":"---------------------"},{"line_number":544,"context_line":""},{"line_number":545,"context_line":"* [pci]transport_node_list config option is added which allows one to specify"},{"line_number":546,"context_line":"  device and vendor IDs of PCI(e) devices that will be considered as the ones"},{"line_number":547,"context_line":"  associated with a transport node;"},{"line_number":548,"context_line":"* Having the ``devlink`` binary is not required in a Linux distribution of"}],"source_content_type":"text/x-rst","patch_set":4,"id":"1682d239_b965d70f","line":545,"range":{"start_line":545,"start_character":2,"end_line":545,"end_character":26},"in_reply_to":"b3796ffe_1bd0ee8a","updated":"2021-05-21 19:01:48.000000000","message":"Agreed, this looks simpler. Thanks!","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":545,"context_line":"* [pci]transport_node_list config option is added which allows one to specify"},{"line_number":546,"context_line":"  device and vendor IDs of PCI(e) devices that will be considered as the ones"},{"line_number":547,"context_line":"  associated with a transport node;"},{"line_number":548,"context_line":"* Having the ``devlink`` binary is not required in a Linux distribution of"},{"line_number":549,"context_line":"  choice - if it is not present or does not expose devlink-info this method"},{"line_number":550,"context_line":"  of obtaining a serial number is ignored;"},{"line_number":551,"context_line":"* Reading PCI(e) device VPD is supported since kernel 2.6.26"}],"source_content_type":"text/x-rst","patch_set":4,"id":"3037f9a9_506ad8c4","line":548,"range":{"start_line":548,"start_character":15,"end_line":548,"end_character":22},"updated":"2021-05-07 12:08:25.000000000","message":"no we shoudl not use the devlink cli.\nwe should use the pyroute2 devlink integration if we need to interact with devlink.\n\nhttps://github.com/svinota/pyroute2/blob/32bf56f5f7c8866d9839e1ae2eab841c827eeffb/pyroute2/netlink/devlink/__init__.py\n\n\nideally however we would get this info from libvirt.\nin general unlless there is no other choice we avoid using clis form nova if possible.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":545,"context_line":"* [pci]transport_node_list config option is added which allows one to specify"},{"line_number":546,"context_line":"  device and vendor IDs of PCI(e) devices that will be considered as the ones"},{"line_number":547,"context_line":"  associated with a transport node;"},{"line_number":548,"context_line":"* Having the ``devlink`` binary is not required in a Linux distribution of"},{"line_number":549,"context_line":"  choice - if it is not present or does not expose devlink-info this method"},{"line_number":550,"context_line":"  of obtaining a serial number is ignored;"},{"line_number":551,"context_line":"* Reading PCI(e) device VPD is supported since kernel 2.6.26"}],"source_content_type":"text/x-rst","patch_set":4,"id":"cf3db4c7_79ab48ef","line":548,"range":{"start_line":548,"start_character":15,"end_line":548,"end_character":22},"in_reply_to":"2b76cdfc_b9271935","updated":"2021-05-21 19:01:48.000000000","message":"Ack","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":545,"context_line":"* [pci]transport_node_list config option is added which allows one to specify"},{"line_number":546,"context_line":"  device and vendor IDs of PCI(e) devices that will be considered as the ones"},{"line_number":547,"context_line":"  associated with a transport node;"},{"line_number":548,"context_line":"* Having the ``devlink`` binary is not required in a Linux distribution of"},{"line_number":549,"context_line":"  choice - if it is not present or does not expose devlink-info this method"},{"line_number":550,"context_line":"  of obtaining a serial number is ignored;"},{"line_number":551,"context_line":"* Reading PCI(e) device VPD is supported since kernel 2.6.26"}],"source_content_type":"text/x-rst","patch_set":4,"id":"2b76cdfc_b9271935","line":548,"range":{"start_line":548,"start_character":15,"end_line":548,"end_character":22},"in_reply_to":"3037f9a9_506ad8c4","updated":"2021-05-11 13:17:45.000000000","message":"Agreed, if the necessary info is available via pyroute2 we should use it instead.\n\nI will explore what\u0027s available and update this line accordingly.\n\nOn Libvirt: +1.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":548,"context_line":"* Having the ``devlink`` binary is not required in a Linux distribution of"},{"line_number":549,"context_line":"  choice - if it is not present or does not expose devlink-info this method"},{"line_number":550,"context_line":"  of obtaining a serial number is ignored;"},{"line_number":551,"context_line":"* Reading PCI(e) device VPD is supported since kernel 2.6.26"},{"line_number":552,"context_line":"  (see kernel commit 94e6108803469a37ee1e3c92dafdd1d59298602f) and devices"},{"line_number":553,"context_line":"  that support PCI local bus 2.1+ (and any PCIe revision) use the same binary"},{"line_number":554,"context_line":"  format for it. The VPD capability is optional per the PCI(e) specs, however,"},{"line_number":555,"context_line":"  production SmartNICs/DPUs observed so far do contain it (engineering samples"},{"line_number":556,"context_line":"  may not have VPD so only use generally available hardware for this)."},{"line_number":557,"context_line":""},{"line_number":558,"context_line":"During the deployment planning it is also important to take control traffic"},{"line_number":559,"context_line":"paths into account. Nova compute is expected to trigger representor plugging"}],"source_content_type":"text/x-rst","patch_set":4,"id":"128d4c6a_179fff9c","line":556,"range":{"start_line":551,"start_character":2,"end_line":556,"end_character":70},"updated":"2021-05-07 12:08:25.000000000","message":"looking at my connectx6-dx card locally this info does not appear to be in libvirt doay is there any work to add it\n\n[centos@vdpa-ironic-1 ~]$ virsh nodedev-dumpxml net_eth2_b8_ce_f6_3e_3d_fa\n\u003cdevice\u003e\n  \u003cname\u003enet_eth2_b8_ce_f6_3e_3d_fa\u003c/name\u003e\n  \u003cpath\u003e/sys/devices/pci0000:00/0000:00:07.0/0000:04:00.0/net/eth2\u003c/path\u003e\n  \u003cparent\u003epci_0000_04_00_0\u003c/parent\u003e\n  \u003ccapability type\u003d\u0027net\u0027\u003e\n    \u003cinterface\u003eeth2\u003c/interface\u003e\n    \u003caddress\u003eb8:ce:f6:3e:3d:fa\u003c/address\u003e\n    \u003clink state\u003d\u0027down\u0027/\u003e\n    \u003cfeature name\u003d\u0027rx\u0027/\u003e\n    \u003cfeature name\u003d\u0027tx\u0027/\u003e\n    \u003cfeature name\u003d\u0027sg\u0027/\u003e\n    \u003cfeature name\u003d\u0027tso\u0027/\u003e\n    \u003cfeature name\u003d\u0027gso\u0027/\u003e\n    \u003cfeature name\u003d\u0027gro\u0027/\u003e\n    \u003cfeature name\u003d\u0027rxvlan\u0027/\u003e\n    \u003cfeature name\u003d\u0027txvlan\u0027/\u003e\n    \u003cfeature name\u003d\u0027rxhash\u0027/\u003e\n    \u003cfeature name\u003d\u0027rdma\u0027/\u003e\n    \u003cfeature name\u003d\u0027txudptnl\u0027/\u003e\n    \u003ccapability type\u003d\u002780203\u0027/\u003e\n  \u003c/capability\u003e\n\u003c/device\u003e\n\n\n[centos@vdpa-ironic-1 ~]$ virsh nodedev-dumpxml pci_0000_04_00_0\n\u003cdevice\u003e\n  \u003cname\u003epci_0000_04_00_0\u003c/name\u003e\n  \u003cpath\u003e/sys/devices/pci0000:00/0000:00:07.0/0000:04:00.0\u003c/path\u003e\n  \u003cparent\u003epci_0000_00_07_0\u003c/parent\u003e\n  \u003cdriver\u003e\n    \u003cname\u003emlx5_core\u003c/name\u003e\n  \u003c/driver\u003e\n  \u003ccapability type\u003d\u0027pci\u0027\u003e\n    \u003cclass\u003e0x020000\u003c/class\u003e\n    \u003cdomain\u003e0\u003c/domain\u003e\n    \u003cbus\u003e4\u003c/bus\u003e\n    \u003cslot\u003e0\u003c/slot\u003e\n    \u003cfunction\u003e0\u003c/function\u003e\n    \u003cproduct id\u003d\u00270x101d\u0027\u003eMT2892 Family [ConnectX-6 Dx]\u003c/product\u003e\n    \u003cvendor id\u003d\u00270x15b3\u0027\u003eMellanox Technologies\u003c/vendor\u003e\n    \u003ccapability type\u003d\u0027virt_functions\u0027 maxCount\u003d\u00278\u0027/\u003e\n  \u003c/capability\u003e\n\u003c/device\u003e\n\nalthough i can confirm its avaiable in lscpi so its present in the pci config space but not currently enable in libvirt. normally i would strongly suggest we enabel libvirt to return this info first\nrather then looking it up directly.\n\n\nCapabilities: [48] Vital Product Data\n\t\tProduct Name: ConnectX-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe 4.0 x8, No Crypto                                                                                                         \n\t\tRead-only fields:\n\t\t\t[PN] Part number: MCX621102AN-ADAT         \n\t\t\t[EC] Engineering changes: A8\n\t\t\t[V2] Vendor specific: MCX621102AN-ADAT         \n\t\t\t[SN] Serial number: MT2106X16246   \n\t\t\t[V3] Vendor specific: 1ac1ab892d70eb118000b8cef63e3dfa\n\t\t\t[VA] Vendor specific: MLX:MN\u003dMLNX:CSKU\u003dV2:UUID\u003dV3:PCI\u003dV0:MODL\u003dCX621102A      \n\t\t\t[V0] Vendor specific: PCIeGen4 x8  \n\t\t\t[RV] Reserved: checksum good, 1 byte(s) reserved\n\n\nbut if i look in sysfs it looks like the info is also avaiable although with some non text data\n\nsudo python3 -c \"print(open(\u0027/sys/devices/pci0000:00/0000:00:07.0/0000:04:00.0/vpd\u0027,\u0027rb\u0027).read())\" \nb\u0027\\x82\\xb6\\x00ConnectX-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe 4.0 x8, No Crypto                                                                                                         \\x90\\xc1\\x00PN\\x19MCX621102AN-ADAT         EC\\x02A8V2\\x19MCX621102AN-ADAT         SN\\x0fMT2106X16246   V3 1ac1ab892d70eb118000b8cef63e3dfaVA7MLX:MN\u003dMLNX:CSKU\u003dV2:UUID\u003dV3:PCI\u003dV0:MODL\u003dCX621102A      V0\\rPCIeGen4 x8  RV\\x022\\x00x\u0027\n\nthe serial number is present SN\\x0fMT2106X16246 is present although with an SN\\x0f prefix\nso we proably coudl parse it directly if requried without adding a new tool depency \n\nnornally we woudl require libvirt to be updated to provide this info before proceeding with the feature but im not sure how long that would take.\n\n\nfull lspci output below.\n04:00.0 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx]\n\tSubsystem: Mellanox Technologies Device 0012\n\tControl: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+\n\tStatus: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL\u003dfast \u003eTAbort- \u003cTAbort- \u003cMAbort- \u003eSERR- \u003cPERR- INTx-\n\tLatency: 0, Cache Line Size: 256 bytes\n\tInterrupt: pin A routed to IRQ 16\n\tRegion 0: Memory at f6000000 (64-bit, prefetchable) [size\u003d32M]\n\tExpansion ROM at fbc00000 [disabled] [size\u003d1M]\n\tCapabilities: [60] Express (v2) Endpoint, MSI 00\n\t\tDevCap:\tMaxPayload 512 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited\n\t\t\tExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W\n\t\tDevCtl:\tCorrErr- NonFatalErr+ FatalErr+ UnsupReq-\n\t\t\tRlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-\n\t\t\tMaxPayload 256 bytes, MaxReadReq 4096 bytes\n\t\tDevSta:\tCorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-\n\t\tLnkCap:\tPort #0, Speed 16GT/s, Width x8, ASPM not supported\n\t\t\tClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+\n\t\tLnkCtl:\tASPM Disabled; RCB 64 bytes, Disabled- CommClk-\n\t\t\tExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-\n\t\tLnkSta:\tSpeed 5GT/s (downgraded), Width x8 (ok)\n\t\t\tTrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-\n\t\tDevCap2: Completion Timeout: Range ABC, TimeoutDis+ NROPrPrP- LTR-\n\t\t\t 10BitTagComp+ 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-\n\t\t\t EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-\n\t\t\t FRS- TPHComp- ExtTPHComp-\n\t\t\t AtomicOpsCap: 32bit- 64bit- 128bitCAS-\n\t\tDevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,\n\t\t\t AtomicOpsCtl: ReqEn-\n\t\tLnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS-\n\t\tLnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-\n\t\t\t Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-\n\t\t\t Compliance De-emphasis: -6dB\n\t\tLnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-\n\t\t\t EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-\n\t\t\t Retimer- 2Retimers- CrosslinkRes: unsupported\n\tCapabilities: [48] Vital Product Data\n\t\tProduct Name: ConnectX-6 Dx EN adapter card, 25GbE, Dual-port SFP28, PCIe 4.0 x8, No Crypto                                                                                                         \n\t\tRead-only fields:\n\t\t\t[PN] Part number: MCX621102AN-ADAT         \n\t\t\t[EC] Engineering changes: A8\n\t\t\t[V2] Vendor specific: MCX621102AN-ADAT         \n\t\t\t[SN] Serial number: MT2106X16246   \n\t\t\t[V3] Vendor specific: 1ac1ab892d70eb118000b8cef63e3dfa\n\t\t\t[VA] Vendor specific: MLX:MN\u003dMLNX:CSKU\u003dV2:UUID\u003dV3:PCI\u003dV0:MODL\u003dCX621102A      \n\t\t\t[V0] Vendor specific: PCIeGen4 x8  \n\t\t\t[RV] Reserved: checksum good, 1 byte(s) reserved\n\t\tEnd\n\tCapabilities: [9c] MSI-X: Enable+ Count\u003d64 Masked-\n\t\tVector table: BAR\u003d0 offset\u003d00002000\n\t\tPBA: BAR\u003d0 offset\u003d00003000\n\tCapabilities: [c0] Vendor Specific Information: Len\u003d18 \u003c?\u003e\n\tCapabilities: [40] Power Management version 3\n\t\tFlags: PMEClk- DSI- D1- D2- AuxCurrent\u003d375mA PME(D0-,D1-,D2-,D3hot-,D3cold+)\n\t\tStatus: D0 NoSoftRst+ PME-Enable- DSel\u003d0 DScale\u003d0 PME-\n\tCapabilities: [100 v1] Advanced Error Reporting\n\t\tUESta:\tDLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-\n\t\tUEMsk:\tDLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-\n\t\tUESvrt:\tDLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt+ UnxCmplt+ RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-\n\t\tCESta:\tRxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+\n\t\tCEMsk:\tRxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+\n\t\tAERCap:\tFirst Error Pointer: 04, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-\n\t\t\tMultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-\n\t\tHeaderLog: 00000000 00000000 00000000 00000000\n\tCapabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)\n\t\tARICap:\tMFVC- ACS-, Next Function: 1\n\t\tARICtl:\tMFVC- ACS-, Function Group: 0\n\tCapabilities: [180 v1] Single Root I/O Virtualization (SR-IOV)\n\t\tIOVCap:\tMigration-, Interrupt Message Number: 000\n\t\tIOVCtl:\tEnable- Migration- Interrupt- MSE- ARIHierarchy+\n\t\tIOVSta:\tMigration-\n\t\tInitial VFs: 8, Total VFs: 8, Number of VFs: 0, Function Dependency Link: 00\n\t\tVF offset: 2, stride: 1, Device ID: 101e\n\t\tSupported Page Size: 000007ff, System Page Size: 00000001\n\t\tRegion 0: Memory at 0000000000000000 (64-bit, prefetchable)\n\t\tVF Migration: offset: 00000000, BIR: 0\n\tCapabilities: [1c0 v1] Secondary PCI Express\n\t\tLnkCtl3: LnkEquIntrruptEn- PerformEqu-\n\t\tLaneErrStat: 0\n\tCapabilities: [230 v1] Access Control Services\n\t\tACSCap:\tSrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-\n\t\tACSCtl:\tSrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-\n\tCapabilities: [320 v1] Lane Margining at the Receiver \u003c?\u003e\n\tCapabilities: [370 v1] Physical Layer 16.0 GT/s \u003c?\u003e\n\tCapabilities: [420 v1] Data Link Feature \u003c?\u003e\n\tKernel driver in use: mlx5_core\n\tKernel modules: mlx5_core","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":548,"context_line":"* Having the ``devlink`` binary is not required in a Linux distribution of"},{"line_number":549,"context_line":"  choice - if it is not present or does not expose devlink-info this method"},{"line_number":550,"context_line":"  of obtaining a serial number is ignored;"},{"line_number":551,"context_line":"* Reading PCI(e) device VPD is supported since kernel 2.6.26"},{"line_number":552,"context_line":"  (see kernel commit 94e6108803469a37ee1e3c92dafdd1d59298602f) and devices"},{"line_number":553,"context_line":"  that support PCI local bus 2.1+ (and any PCIe revision) use the same binary"},{"line_number":554,"context_line":"  format for it. The VPD capability is optional per the PCI(e) specs, however,"},{"line_number":555,"context_line":"  production SmartNICs/DPUs observed so far do contain it (engineering samples"},{"line_number":556,"context_line":"  may not have VPD so only use generally available hardware for this)."},{"line_number":557,"context_line":""},{"line_number":558,"context_line":"During the deployment planning it is also important to take control traffic"},{"line_number":559,"context_line":"paths into account. Nova compute is expected to trigger representor plugging"}],"source_content_type":"text/x-rst","patch_set":4,"id":"4caffafb_25abf1e6","line":556,"range":{"start_line":551,"start_character":2,"end_line":556,"end_character":70},"in_reply_to":"128d4c6a_179fff9c","updated":"2021-05-11 13:17:45.000000000","message":"There isn\u0027t any work done in libvirt yet.\n\nThe plan was to start with a Nova spec and have an in-Nova implementation of this to get to a working setup faster while libvirt work is in progress.\n\nBut libvirt seems to be the right place for this functionality long term.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1933e7abc61d9926e1fb5cba9bb19aac9ef7ff52","unresolved":true,"context_lines":[{"line_number":548,"context_line":"* Having the ``devlink`` binary is not required in a Linux distribution of"},{"line_number":549,"context_line":"  choice - if it is not present or does not expose devlink-info this method"},{"line_number":550,"context_line":"  of obtaining a serial number is ignored;"},{"line_number":551,"context_line":"* Reading PCI(e) device VPD is supported since kernel 2.6.26"},{"line_number":552,"context_line":"  (see kernel commit 94e6108803469a37ee1e3c92dafdd1d59298602f) and devices"},{"line_number":553,"context_line":"  that support PCI local bus 2.1+ (and any PCIe revision) use the same binary"},{"line_number":554,"context_line":"  format for it. The VPD capability is optional per the PCI(e) specs, however,"},{"line_number":555,"context_line":"  production SmartNICs/DPUs observed so far do contain it (engineering samples"},{"line_number":556,"context_line":"  may not have VPD so only use generally available hardware for this)."},{"line_number":557,"context_line":""},{"line_number":558,"context_line":"During the deployment planning it is also important to take control traffic"},{"line_number":559,"context_line":"paths into account. Nova compute is expected to trigger representor plugging"}],"source_content_type":"text/x-rst","patch_set":4,"id":"e742a18a_34e35c7b","line":556,"range":{"start_line":551,"start_character":2,"end_line":556,"end_character":70},"in_reply_to":"4caffafb_25abf1e6","updated":"2021-05-18 12:45:55.000000000","message":"in the past we have blocked the nova impelamtion for similar feature until support was added to libvirt\njust as an fyi. if there is work you can point ot to show that libvirt work is in progress and that it will eventlly be something we can delegate then i would not be agaisnt nova retirivng this info in the short term as long as we dont have to do it in the long term. if no work has been started in the libvirt comunity to enable it i think we shoudl atelast start a mail tread to deterim if they would consider adding support in the future.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":true,"context_lines":[{"line_number":548,"context_line":"* Having the ``devlink`` binary is not required in a Linux distribution of"},{"line_number":549,"context_line":"  choice - if it is not present or does not expose devlink-info this method"},{"line_number":550,"context_line":"  of obtaining a serial number is ignored;"},{"line_number":551,"context_line":"* Reading PCI(e) device VPD is supported since kernel 2.6.26"},{"line_number":552,"context_line":"  (see kernel commit 94e6108803469a37ee1e3c92dafdd1d59298602f) and devices"},{"line_number":553,"context_line":"  that support PCI local bus 2.1+ (and any PCIe revision) use the same binary"},{"line_number":554,"context_line":"  format for it. The VPD capability is optional per the PCI(e) specs, however,"},{"line_number":555,"context_line":"  production SmartNICs/DPUs observed so far do contain it (engineering samples"},{"line_number":556,"context_line":"  may not have VPD so only use generally available hardware for this)."},{"line_number":557,"context_line":""},{"line_number":558,"context_line":"During the deployment planning it is also important to take control traffic"},{"line_number":559,"context_line":"paths into account. Nova compute is expected to trigger representor plugging"}],"source_content_type":"text/x-rst","patch_set":4,"id":"389c00cd_91009cdc","line":556,"range":{"start_line":551,"start_character":2,"end_line":556,"end_character":70},"in_reply_to":"e742a18a_34e35c7b","updated":"2021-05-21 19:01:48.000000000","message":"OK, I\u0027ll prepare a conceptual proposal for Libvirt with some references and send it to libvir-list@redhat.com.\n\n\n\u003e but if i look in sysfs it looks like the info is also avaiable although with some non text data\n\nYes, the binary format is the one described in PCI (2.1+ IIRC) and PCIe (4.0+) specs. It\u0027s quite old and stable. There is one implementation for parsing it available in lspci.\n\nI also have a gist in Python which I used during prototyping (this, of course, needs some unit testing - just showing what I am talking about):\nhttps://gist.github.com/dshcherb/40e982989599a757e5b1e25999501019","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":556,"context_line":"  may not have VPD so only use generally available hardware for this)."},{"line_number":557,"context_line":""},{"line_number":558,"context_line":"During the deployment planning it is also important to take control traffic"},{"line_number":559,"context_line":"paths into account. Nova compute is expected to trigger representor plugging"},{"line_number":560,"context_line":"indirectly via Neutron as a client via the control network: Neutron is then"},{"line_number":561,"context_line":"responsible for propagating this information back to the responsible agents on"},{"line_number":562,"context_line":"a relevant transport node (via the OVN mechanism driver in this case). Also,"},{"line_number":563,"context_line":"Placement service updates from hypervisor nodes happen over the control"},{"line_number":564,"context_line":"network. This may happen via dedicated ports programmed on the eSwitch which"},{"line_number":565,"context_line":"needs to be done via some form of a deployment automation. Alternatively, LoMs"},{"line_number":566,"context_line":"on many motherboards may be used for that communication but the overall goal"}],"source_content_type":"text/x-rst","patch_set":4,"id":"b2d72276_8c05b381","line":563,"range":{"start_line":559,"start_character":19,"end_line":563,"end_character":69},"updated":"2021-05-07 12:08:25.000000000","message":"so form a design level i think that is not a direction we should go in.\nwe shoudl not add any api calls form nova to neutron during port plugging.\n\nnova should pass all info required to neutorn during port binding.\nport pluging is when nova addes the VF in this case to the ovs brdige\nand should not invovle nueton api calls.\n\nfor what its worth this is approaching the level of a -2 issue since its a fundamental architectural change that i do not believe should be done.\n\nits not quite to that level but im not ok with adding cause form os-vif to neutron and i dont think nova should handel pluggin internally any more so exting the libvirt driver vif.py to call neutron is also not an option in my view. we can have the port pluggin be a no op in the libvirt driver in this case and pass all info during port binding as a path forward perhaps.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":true,"context_lines":[{"line_number":556,"context_line":"  may not have VPD so only use generally available hardware for this)."},{"line_number":557,"context_line":""},{"line_number":558,"context_line":"During the deployment planning it is also important to take control traffic"},{"line_number":559,"context_line":"paths into account. Nova compute is expected to trigger representor plugging"},{"line_number":560,"context_line":"indirectly via Neutron as a client via the control network: Neutron is then"},{"line_number":561,"context_line":"responsible for propagating this information back to the responsible agents on"},{"line_number":562,"context_line":"a relevant transport node (via the OVN mechanism driver in this case). Also,"},{"line_number":563,"context_line":"Placement service updates from hypervisor nodes happen over the control"},{"line_number":564,"context_line":"network. This may happen via dedicated ports programmed on the eSwitch which"},{"line_number":565,"context_line":"needs to be done via some form of a deployment automation. Alternatively, LoMs"},{"line_number":566,"context_line":"on many motherboards may be used for that communication but the overall goal"}],"source_content_type":"text/x-rst","patch_set":4,"id":"603fec67_a72f039a","line":563,"range":{"start_line":559,"start_character":19,"end_line":563,"end_character":69},"in_reply_to":"022826f2_185c5e68","updated":"2021-05-21 19:01:48.000000000","message":"Regarding using a new port type (VNIC_TYPE_DIRECT_DPU in your example), there is already a type VNIC_SMARTNIC which was added to support SmartNIC DPUs in Ironic. Maybe we can reuse it here as well.\n\nhttps://opendev.org/openstack/neutron-lib/commit/f3f220ed62410f1408cdadbcb45cbd8657836cea\nVNIC_SMARTNIC \u003d \u0027smart-nic\u0027","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":13686,"name":"Frode Nordahl","email":"fnordahl@ubuntu.com","username":"fnordahl"},"change_message_id":"e2fd0ddfa458c535426587e8fcfd737b62658bcc","unresolved":true,"context_lines":[{"line_number":556,"context_line":"  may not have VPD so only use generally available hardware for this)."},{"line_number":557,"context_line":""},{"line_number":558,"context_line":"During the deployment planning it is also important to take control traffic"},{"line_number":559,"context_line":"paths into account. Nova compute is expected to trigger representor plugging"},{"line_number":560,"context_line":"indirectly via Neutron as a client via the control network: Neutron is then"},{"line_number":561,"context_line":"responsible for propagating this information back to the responsible agents on"},{"line_number":562,"context_line":"a relevant transport node (via the OVN mechanism driver in this case). Also,"},{"line_number":563,"context_line":"Placement service updates from hypervisor nodes happen over the control"},{"line_number":564,"context_line":"network. This may happen via dedicated ports programmed on the eSwitch which"},{"line_number":565,"context_line":"needs to be done via some form of a deployment automation. Alternatively, LoMs"},{"line_number":566,"context_line":"on many motherboards may be used for that communication but the overall goal"}],"source_content_type":"text/x-rst","patch_set":4,"id":"4673a3c0_eff7d56e","line":563,"range":{"start_line":559,"start_character":19,"end_line":563,"end_character":69},"in_reply_to":"422e98f8_b0b338f7","updated":"2021-05-11 13:28:05.000000000","message":"Sean, it appears that we might be talking past each other on some of the details in this spec.\n\nTo demonstrate what we are proposing to do I dusted off the PoC code we did prior to writing this spec:\n\nnova-compute PoC: https://pastebin.ubuntu.com/p/kbZ3zz7rPJ/\nneutron-server PoC: https://pastebin.ubuntu.com/p/gKKvTtYTFH/\nfixture ran on third host for purpose of PoC: https://pastebin.ubuntu.com/p/JQV6hywv7V/\n\nAs you can see we are not adding any API calls or suchlike, we are just extending the already existing conversation between Nova and Neutron.\n\nAfter Nova plugs or not plugs a VIF it will already today wait for a notification from Neutron about success of port binding. Neutron does not emit this notification until it has feedback from its agents or OVN that the port is successfully bound, so even if Nova does not do any plugging it will not resume the instance until networking is fully plumbed.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1933e7abc61d9926e1fb5cba9bb19aac9ef7ff52","unresolved":true,"context_lines":[{"line_number":556,"context_line":"  may not have VPD so only use generally available hardware for this)."},{"line_number":557,"context_line":""},{"line_number":558,"context_line":"During the deployment planning it is also important to take control traffic"},{"line_number":559,"context_line":"paths into account. Nova compute is expected to trigger representor plugging"},{"line_number":560,"context_line":"indirectly via Neutron as a client via the control network: Neutron is then"},{"line_number":561,"context_line":"responsible for propagating this information back to the responsible agents on"},{"line_number":562,"context_line":"a relevant transport node (via the OVN mechanism driver in this case). Also,"},{"line_number":563,"context_line":"Placement service updates from hypervisor nodes happen over the control"},{"line_number":564,"context_line":"network. This may happen via dedicated ports programmed on the eSwitch which"},{"line_number":565,"context_line":"needs to be done via some form of a deployment automation. Alternatively, LoMs"},{"line_number":566,"context_line":"on many motherboards may be used for that communication but the overall goal"}],"source_content_type":"text/x-rst","patch_set":4,"id":"022826f2_185c5e68","line":563,"range":{"start_line":559,"start_character":19,"end_line":563,"end_character":69},"in_reply_to":"4673a3c0_eff7d56e","updated":"2021-05-18 12:45:55.000000000","message":"no after nove pluggs the vif we wait for a network-vif-plugged event for neutron on the sucess fo the port plugging.\n\nport bidning is what happens eariler wehn we set the binding:host-id on the neutron port.\nport binding is complete when the api call returns form tha tport update.\n\nport plugging happens after that.\n\n\nso looking at\n\nhttps://pastebin.ubuntu.com/p/kbZ3zz7rPJ/\n\nthe first half o the patch i think we agree will be needed \n\ndiff --git a/nova/network/neutron.py b/nova/network/neutron.py\nindex db4cb44d47..2f0aadf170 100644\n--- a/nova/network/neutron.py\n+++ b/nova/network/neutron.py\n@@ -1517,6 +1517,11 @@ class API(base.Base):\n             pci_dev \u003d pci_devices.pop()\n             profile \u003d copy.deepcopy(get_binding_profile(port_req_body[\u0027port\u0027]))\n             profile.update(self._get_pci_device_profile(pci_dev))\n+            # XXX hack for the purpose of PoC\n+            profile.update(\n+                {\u0027board_serial_number\u0027: \u0027UNIQUEBOARDSERIAL\u0027,\n+                 \u0027pf_mac\u0027: \u0027de:ad:be:ef:ca:fe\u0027,\n+                 \u0027vf_num\u0027: 42})\n             port_req_body[\u0027port\u0027][constants.BINDING_PROFILE] \u003d profile\n \n     @staticmethod\n\nor at least something like that to pass the require info in _populate_neutron_binding_profile\nhttps://github.com/openstack/nova/blob/82141b12c356fe374822038bdf0a6f0aaa32047b/nova/network/neutron.py#L1512-L1538\n\n\nthat is part of port binding not part of port plugging.\n\nthe second half of that patch is incorrect.\n\n\ninstead of skipping port plugging\n\ndiff --git a/nova/virt/libvirt/driver.py b/nova/virt/libvirt/driver.py\nindex fbd033690a..39c15fc074 100644\n--- a/nova/virt/libvirt/driver.py\n+++ b/nova/virt/libvirt/driver.py\n@@ -1163,14 +1163,33 @@ class LibvirtDriver(driver.ComputeDriver):\n \n         return uuids\n \n+    @staticmethod\n+    def _should_plug(vif):\n+        # XXX hack for the purpose of PoC\n+        vif_info \u003d jsonutils.loads(str(vif))\n+        if (\u0027profile\u0027 in vif_info\n+                and \u0027capabilities\u0027 in vif_info[\u0027profile\u0027]\n+                # here we could check for a \u0027transport_node\u0027 capability\n+                and \u0027switchdev\u0027 in vif_info[\u0027profile\u0027][\u0027capabilities\u0027]):\n+            return False\n+        return True\n+\n     def plug_vifs(self, instance, network_info):\n         \"\"\"Plug VIFs into networks.\"\"\"\n         for vif in network_info:\n+            # XXX hack for the purpose of PoC\n+            if not self._should_plug(vif):\n+                LOG.info(\u0027Not plugging VIF: \"{}\"\u0027.format(vif))\n+                continue\n             self.vif_driver.plug(instance, vif)\n \n     def _unplug_vifs(self, instance, network_info, ignore_errors):\n         \"\"\"Unplug VIFs from networks.\"\"\"\n         for vif in network_info:\n+            # XXX hack for the purpose of PoC\n+            if not self._should_plug(vif):\n+                LOG.info(\u0027Not unplugging VIF: \"{}\"\u0027.format(vif))\n+                continue\n             try:\n                 self.vif_driver.unplug(instance, vif)\n             except exception.NovaException:\n\n\nyou should be modifing _nova_to_osvif_vif_ovs to matach on some aspect of the data returnidn form the port binding call to use the noop plugin.\n\nhttps://github.com/openstack/nova/blob/82141b12c356fe374822038bdf0a6f0aaa32047b/nova/network/os_vif_util.py#L328-L359\n\ne.g.\n\ndef _nova_to_osvif_vif_ovs(vif):\n    vif_name \u003d _get_vif_name(vif)\n    vnic_type \u003d vif.get(\u0027vnic_type\u0027, model.VNIC_TYPE_NORMAL)\n    profile \u003d objects.vif.VIFPortProfileOpenVSwitch(\n        interface_id\u003dvif.get(\u0027ovs_interfaceid\u0027) or vif[\u0027id\u0027],\n        datapath_type\u003dvif[\u0027details\u0027].get(\n            model.VIF_DETAILS_OVS_DATAPATH_TYPE))\n    # you could also look at someing in the binidng deatails or\n    # binding profile instead of using a dedicated vnic_type\n    if vnic_type in (model.VNIC_TYPE_DIRECT_DPU):\n       # we shoudl extend https://github.com/openstack/os-vif/blob/master/vif_plug_noop/noop.py\n       # to supprot something other then objects.vif.VIFVHostUser like \n       #  objects.vif.VIFGeneric or objects.vif.VIFHostDevice, but its not actully used in\n       # this plugin.\n       obj \u003d _get_vif_instance(vif, objects.vif.VIFVHostUser,\n                                plugin\u003d\"noop\",\n                                vif_name\u003d_get_vif_name(vif))\n    elif vnic_type in (model.VNIC_TYPE_DIRECT, model.VNIC_TYPE_VDPA):\n        obj \u003d _get_vnic_direct_vif_instance(\n            vif,\n            port_profile\u003d_get_ovs_representor_port_profile(vif),\n            plugin\u003d\"ovs\")\n        _set_representor_datapath_offload_settings(vif, obj)\n    elif vif.is_hybrid_plug_enabled():\n        obj \u003d _get_vif_instance(\n            vif,\n            objects.vif.VIFBridge,\n            port_profile\u003dprofile,\n            plugin\u003d\"ovs\",\n            vif_name\u003dvif_name,\n            bridge_name\u003d_get_hybrid_bridge_name(vif))\n    else:\n        profile.create_port \u003d vif.get(\u0027delegate_create\u0027, False)\n        obj \u003d _get_vif_instance(\n            vif,\n            objects.vif.VIFOpenVSwitch,\n            port_profile\u003dprofile,\n            plugin\u003d\"ovs\",\n            vif_name\u003dvif_name)\n        if vif[\"network\"][\"bridge\"] is not None:\n            obj.bridge_name \u003d vif[\"network\"][\"bridge\"]\n    return obj","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"eba8738e4b779a4ebd7133f7e3296d31e02e0ff1","unresolved":true,"context_lines":[{"line_number":556,"context_line":"  may not have VPD so only use generally available hardware for this)."},{"line_number":557,"context_line":""},{"line_number":558,"context_line":"During the deployment planning it is also important to take control traffic"},{"line_number":559,"context_line":"paths into account. Nova compute is expected to trigger representor plugging"},{"line_number":560,"context_line":"indirectly via Neutron as a client via the control network: Neutron is then"},{"line_number":561,"context_line":"responsible for propagating this information back to the responsible agents on"},{"line_number":562,"context_line":"a relevant transport node (via the OVN mechanism driver in this case). Also,"},{"line_number":563,"context_line":"Placement service updates from hypervisor nodes happen over the control"},{"line_number":564,"context_line":"network. This may happen via dedicated ports programmed on the eSwitch which"},{"line_number":565,"context_line":"needs to be done via some form of a deployment automation. Alternatively, LoMs"},{"line_number":566,"context_line":"on many motherboards may be used for that communication but the overall goal"}],"source_content_type":"text/x-rst","patch_set":4,"id":"6354042a_93b262dd","line":563,"range":{"start_line":559,"start_character":19,"end_line":563,"end_character":69},"in_reply_to":"603fec67_a72f039a","updated":"2021-05-25 13:40:47.000000000","message":"maybe although that was previously exclusitivly used for ironic smart nic support.\nwe avoided reusing it for cyborg integration as it was unclear if it could break something.\ni did not think it would mind you so im not against using it here but that is one consideration we need to keep in mind.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":556,"context_line":"  may not have VPD so only use generally available hardware for this)."},{"line_number":557,"context_line":""},{"line_number":558,"context_line":"During the deployment planning it is also important to take control traffic"},{"line_number":559,"context_line":"paths into account. Nova compute is expected to trigger representor plugging"},{"line_number":560,"context_line":"indirectly via Neutron as a client via the control network: Neutron is then"},{"line_number":561,"context_line":"responsible for propagating this information back to the responsible agents on"},{"line_number":562,"context_line":"a relevant transport node (via the OVN mechanism driver in this case). Also,"},{"line_number":563,"context_line":"Placement service updates from hypervisor nodes happen over the control"},{"line_number":564,"context_line":"network. This may happen via dedicated ports programmed on the eSwitch which"},{"line_number":565,"context_line":"needs to be done via some form of a deployment automation. Alternatively, LoMs"},{"line_number":566,"context_line":"on many motherboards may be used for that communication but the overall goal"}],"source_content_type":"text/x-rst","patch_set":4,"id":"422e98f8_b0b338f7","line":563,"range":{"start_line":559,"start_character":19,"end_line":563,"end_character":69},"in_reply_to":"b2d72276_8c05b381","updated":"2021-05-11 13:17:45.000000000","message":"Let me try to explain this better.\n\nThe suggestion isn\u0027t to add any new API calls to the process from the Nova side.\n\nNova would merely pass additional information to identity a representor to be plugged:\n\nhttps://github.com/openstack/nova/blob/23.0.0/nova/network/neutron.py#L3360-L3419 (_update_port_binding_for_instance)\n\nNova has no visibility into representor names used at the SmartNIC DPU host side so it would need to learn those somehow in order to use OVS directly.\n\nHow can we manage this without giving the hypervisor host extra visibility into the SmartNIC OS? It\u0027s almost like giving a hypervisor host access to programming a ToR.\n\nAs far as I can tell, a fundamental point of moving the control plane from the hypervisor host is introducing isolation to the design. So Nova should not even know that there is OVS or something else running on the SmartNIC (much like it does not know what\u0027s on the ToR) - it merely needs to grab a VF and wait for a completion from Neutron.\n\nFor now, the proposal was about OVN only so Neutron wouldn\u0027t handle plugging at all - only pass some information downstream to OVN.\n\nFrode has some code just to demonstrate the overall concept if it helps.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"5b55c3ddf7705f6c47c7d334eefb4c0c1b02306f","unresolved":true,"context_lines":[{"line_number":738,"context_line":".. [5] https://lwn.net/Articles/692942/"},{"line_number":739,"context_line":".. [6] https://www.kernel.org/doc/html/latest/networking/devlink/index.html"},{"line_number":740,"context_line":".. [7] https://docs.openstack.org/neutron/latest/admin/config-ovs-offload.html;"},{"line_number":741,"context_line":".. [8] https://patents.google.com/patent/US8521941B2/"},{"line_number":742,"context_line":".. [9] https://www.snia.org/sites/default/orig/DSI2015/presentations/Rev/Jeff%20DodsonSNIA_Tutorial_PCIe_Shared_IO_2015_revision.pdf"},{"line_number":743,"context_line":".. [10] https://www.kernel.org/doc/html/latest/networking/devlink/devlink-port.html#pci-controllers"},{"line_number":744,"context_line":".. [11] https://www.kernel.org/doc/html/latest/networking/devlink/devlink-port.html#devlink-port"}],"source_content_type":"text/x-rst","patch_set":4,"id":"4e4562d9_175b88c7","line":741,"range":{"start_line":741,"start_character":7,"end_line":741,"end_character":53},"updated":"2021-05-07 12:08:25.000000000","message":"by the way whil this is not appliable here, reading patents and proposing to implement its soltion without workign for the company that owns the patent proably would block you spec form a legal stand point alone since you cannot comply with the contributors license agreement.\n\nspecifically since you cant comply with part 3 and 4\nhttps://review.opendev.org/static/cla.html\n\n\n3. Grant of Patent License. Subject to the terms and conditions of this Agreement, You hereby grant to the Project Manager and to recipients of software distributed by the Project Manager a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by You that are necessarily infringed by Your Contribution(s) alone or by combination of Your Contribution(s) with the Work to which such Contribution(s) was submitted. If any entity institutes patent litigation against You or any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that Your Contribution, or the Work to which You have contributed, constitutes direct or contributory patent infringement, then any patent licenses granted to that entity under this Agreement for that Contribution or Work shall terminate as of the date such litigation is filed. \n\n\n4. You represent that you are legally entitled to grant the above license. If your employer(s) has rights to intellectual property that you create that includes your Contributions, You represent that you have received permission to make Contributions on behalf of that employer, that your employer has waived such rights for your Contributions to the Project Manager, or that your employer has executed a separate Corporate Contributor License Agreement with the Project Manager. \n\n\ni have not actually read the patent and im assuming you are not infringing it but in general looking for the existence of a pathent in the area that you are workign in can put you at larger liability if there are issues in the future since it demonstrate prior knolage of any alleged infringement so as a matter of course especially with dealing with opensouce i generally recommend avoiding patents.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"ba66ffe8c0791df34addd55366e5d2b557dd8def","unresolved":false,"context_lines":[{"line_number":738,"context_line":".. [5] https://lwn.net/Articles/692942/"},{"line_number":739,"context_line":".. [6] https://www.kernel.org/doc/html/latest/networking/devlink/index.html"},{"line_number":740,"context_line":".. [7] https://docs.openstack.org/neutron/latest/admin/config-ovs-offload.html;"},{"line_number":741,"context_line":".. [8] https://patents.google.com/patent/US8521941B2/"},{"line_number":742,"context_line":".. [9] https://www.snia.org/sites/default/orig/DSI2015/presentations/Rev/Jeff%20DodsonSNIA_Tutorial_PCIe_Shared_IO_2015_revision.pdf"},{"line_number":743,"context_line":".. [10] https://www.kernel.org/doc/html/latest/networking/devlink/devlink-port.html#pci-controllers"},{"line_number":744,"context_line":".. [11] https://www.kernel.org/doc/html/latest/networking/devlink/devlink-port.html#devlink-port"}],"source_content_type":"text/x-rst","patch_set":4,"id":"40f0c573_81dbc0f1","line":741,"range":{"start_line":741,"start_character":7,"end_line":741,"end_character":53},"in_reply_to":"37d5bd12_a211f63b","updated":"2021-05-21 19:01:48.000000000","message":"Done","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77ae216de5436c5b8d8e26e2a3ff4cab5ee63b48","unresolved":true,"context_lines":[{"line_number":738,"context_line":".. [5] https://lwn.net/Articles/692942/"},{"line_number":739,"context_line":".. [6] https://www.kernel.org/doc/html/latest/networking/devlink/index.html"},{"line_number":740,"context_line":".. [7] https://docs.openstack.org/neutron/latest/admin/config-ovs-offload.html;"},{"line_number":741,"context_line":".. [8] https://patents.google.com/patent/US8521941B2/"},{"line_number":742,"context_line":".. [9] https://www.snia.org/sites/default/orig/DSI2015/presentations/Rev/Jeff%20DodsonSNIA_Tutorial_PCIe_Shared_IO_2015_revision.pdf"},{"line_number":743,"context_line":".. [10] https://www.kernel.org/doc/html/latest/networking/devlink/devlink-port.html#pci-controllers"},{"line_number":744,"context_line":".. [11] https://www.kernel.org/doc/html/latest/networking/devlink/devlink-port.html#devlink-port"}],"source_content_type":"text/x-rst","patch_set":4,"id":"e671a2b7_cec0eb68","line":741,"range":{"start_line":741,"start_character":7,"end_line":741,"end_character":53},"in_reply_to":"4e4562d9_175b88c7","updated":"2021-05-11 13:17:45.000000000","message":"OK, it doesn\u0027t have to be here. I added it just to demonstrate that IOV sharing capabilities can be implemented in a certain way in hardware. I assumed that using hardware that was built using certain patented work does not infringe the patent - i.e. we are not talking about building NICs here. As such, OpenStack is able to work with different types of hardware that may have patented functionality internally.\n\nWill remove this anyway to avoid any potential issues","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"1933e7abc61d9926e1fb5cba9bb19aac9ef7ff52","unresolved":true,"context_lines":[{"line_number":738,"context_line":".. [5] https://lwn.net/Articles/692942/"},{"line_number":739,"context_line":".. [6] https://www.kernel.org/doc/html/latest/networking/devlink/index.html"},{"line_number":740,"context_line":".. [7] https://docs.openstack.org/neutron/latest/admin/config-ovs-offload.html;"},{"line_number":741,"context_line":".. [8] https://patents.google.com/patent/US8521941B2/"},{"line_number":742,"context_line":".. [9] https://www.snia.org/sites/default/orig/DSI2015/presentations/Rev/Jeff%20DodsonSNIA_Tutorial_PCIe_Shared_IO_2015_revision.pdf"},{"line_number":743,"context_line":".. [10] https://www.kernel.org/doc/html/latest/networking/devlink/devlink-port.html#pci-controllers"},{"line_number":744,"context_line":".. [11] https://www.kernel.org/doc/html/latest/networking/devlink/devlink-port.html#devlink-port"}],"source_content_type":"text/x-rst","patch_set":4,"id":"37d5bd12_a211f63b","line":741,"range":{"start_line":741,"start_character":7,"end_line":741,"end_character":53},"in_reply_to":"e671a2b7_cec0eb68","updated":"2021-05-18 12:45:55.000000000","message":"yes just useing a capablity of a hardware is fine but direclty refernice text or a design form a patent ectra puts us in a gray area so its best to just avoid that.","commit_id":"b87dc80f25138766b5210e596c81dd4fe259040c"}],"specs/yoga/approved/integration-with-off-path-network-backends.rst":[{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"346b4af75bb926b9a5729bdc20e2499d8071d0fa","unresolved":true,"context_lines":[{"line_number":152,"context_line":"OVS case, the external_ids[\"hostname\"] field in the Open_vSwitch table differs"},{"line_number":153,"context_line":"from the hypervisor hostname). Likewise, representor plugging and flow"},{"line_number":154,"context_line":"programming happens on the SmartNIC host, not on the hypervisor host. As a"},{"line_number":155,"context_line":"result, Nova (with the help of os-vif) can no longer be responsible for VIF"},{"line_number":156,"context_line":"plugging in the same way. For instance, compared to the OVS hardware offload"},{"line_number":157,"context_line":"scenario, OVS bridges and port representors are no longer exposed to the"},{"line_number":158,"context_line":"hypervisor host OS. In summary, no networking agents are present on the"},{"line_number":159,"context_line":"hypervisor host in this architecture."},{"line_number":160,"context_line":""},{"line_number":161,"context_line":"Since Nova and networking agents run on different hosts, there needs to be a"},{"line_number":162,"context_line":"different set of interactions in order to:"}],"source_content_type":"text/x-rst","patch_set":11,"id":"ecc33f1e_e3b3d01a","line":159,"range":{"start_line":155,"start_character":8,"end_line":159,"end_character":37},"updated":"2021-11-12 15:28:31.000000000","message":"yes so in this case the noop os-vif plugin can likely be used instead if we want to avoid special casing this directly in nova.\n\nthats an implemation detail im happy to discuss in the code review and defer form the sepc by the way.","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"6e535cf9630c7e0c8ec61f4c21241cddd23e5612","unresolved":false,"context_lines":[{"line_number":152,"context_line":"OVS case, the external_ids[\"hostname\"] field in the Open_vSwitch table differs"},{"line_number":153,"context_line":"from the hypervisor hostname). Likewise, representor plugging and flow"},{"line_number":154,"context_line":"programming happens on the SmartNIC host, not on the hypervisor host. As a"},{"line_number":155,"context_line":"result, Nova (with the help of os-vif) can no longer be responsible for VIF"},{"line_number":156,"context_line":"plugging in the same way. For instance, compared to the OVS hardware offload"},{"line_number":157,"context_line":"scenario, OVS bridges and port representors are no longer exposed to the"},{"line_number":158,"context_line":"hypervisor host OS. In summary, no networking agents are present on the"},{"line_number":159,"context_line":"hypervisor host in this architecture."},{"line_number":160,"context_line":""},{"line_number":161,"context_line":"Since Nova and networking agents run on different hosts, there needs to be a"},{"line_number":162,"context_line":"different set of interactions in order to:"}],"source_content_type":"text/x-rst","patch_set":11,"id":"6f3efb3b_ec43bd2f","line":159,"range":{"start_line":155,"start_character":8,"end_line":159,"end_character":37},"in_reply_to":"dc9fb676_16635dc4","updated":"2021-11-15 12:09:55.000000000","message":"Done","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"82a4915d2026e9d9392daea09a8ca225dd9b84b2","unresolved":true,"context_lines":[{"line_number":152,"context_line":"OVS case, the external_ids[\"hostname\"] field in the Open_vSwitch table differs"},{"line_number":153,"context_line":"from the hypervisor hostname). Likewise, representor plugging and flow"},{"line_number":154,"context_line":"programming happens on the SmartNIC host, not on the hypervisor host. As a"},{"line_number":155,"context_line":"result, Nova (with the help of os-vif) can no longer be responsible for VIF"},{"line_number":156,"context_line":"plugging in the same way. For instance, compared to the OVS hardware offload"},{"line_number":157,"context_line":"scenario, OVS bridges and port representors are no longer exposed to the"},{"line_number":158,"context_line":"hypervisor host OS. In summary, no networking agents are present on the"},{"line_number":159,"context_line":"hypervisor host in this architecture."},{"line_number":160,"context_line":""},{"line_number":161,"context_line":"Since Nova and networking agents run on different hosts, there needs to be a"},{"line_number":162,"context_line":"different set of interactions in order to:"}],"source_content_type":"text/x-rst","patch_set":11,"id":"dc9fb676_16635dc4","line":159,"range":{"start_line":155,"start_character":8,"end_line":159,"end_character":37},"in_reply_to":"ecc33f1e_e3b3d01a","updated":"2021-11-12 16:01:46.000000000","message":"Ack, that\u0027s what I have in the suggested code.\n\nhttps://review.opendev.org/c/openstack/nova/+/812111/3/nova/network/os_vif_util.py#346\n\nI\u0027ll also add some notes about avoid VF VLAN programming at the host side (in short, Nova doesn\u0027t pass a VLAN number to Libvirt in our case https://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py#L479-L485).","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"346b4af75bb926b9a5729bdc20e2499d8071d0fa","unresolved":true,"context_lines":[{"line_number":165,"context_line":"  present;"},{"line_number":166,"context_line":"* Select a suitable VF at the hypervisor host side and create a PCI device"},{"line_number":167,"context_line":"  claim for it;"},{"line_number":168,"context_line":"* Run the necessary logic as described in the Neutron specification [17]_."},{"line_number":169,"context_line":""},{"line_number":170,"context_line":"The SmartNIC DPU selection in particular becomes an issue to address due to"},{"line_number":171,"context_line":"the following:"}],"source_content_type":"text/x-rst","patch_set":11,"id":"b6e02fa8_18ee6d72","line":168,"range":{"start_line":168,"start_character":0,"end_line":168,"end_character":1},"updated":"2021-11-12 15:28:31.000000000","message":"this is refering to neutro using the information we have populated in the binding profile to look up the chasis and update the ovn nortdb. also 17 is not the neutron spec. that is 1.\n17 is https://github.com/fnordahl/ovn-vif\n\nby the way i assume https://github.com/fnordahl/ovn-vif is being upstreamed to ovn?","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"6e535cf9630c7e0c8ec61f4c21241cddd23e5612","unresolved":false,"context_lines":[{"line_number":165,"context_line":"  present;"},{"line_number":166,"context_line":"* Select a suitable VF at the hypervisor host side and create a PCI device"},{"line_number":167,"context_line":"  claim for it;"},{"line_number":168,"context_line":"* Run the necessary logic as described in the Neutron specification [17]_."},{"line_number":169,"context_line":""},{"line_number":170,"context_line":"The SmartNIC DPU selection in particular becomes an issue to address due to"},{"line_number":171,"context_line":"the following:"}],"source_content_type":"text/x-rst","patch_set":11,"id":"58909fc1_970b85ed","line":168,"range":{"start_line":168,"start_character":0,"end_line":168,"end_character":1},"in_reply_to":"8ae95664_4fd51174","updated":"2021-11-15 12:09:55.000000000","message":"Done","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"82a4915d2026e9d9392daea09a8ca225dd9b84b2","unresolved":true,"context_lines":[{"line_number":165,"context_line":"  present;"},{"line_number":166,"context_line":"* Select a suitable VF at the hypervisor host side and create a PCI device"},{"line_number":167,"context_line":"  claim for it;"},{"line_number":168,"context_line":"* Run the necessary logic as described in the Neutron specification [17]_."},{"line_number":169,"context_line":""},{"line_number":170,"context_line":"The SmartNIC DPU selection in particular becomes an issue to address due to"},{"line_number":171,"context_line":"the following:"}],"source_content_type":"text/x-rst","patch_set":11,"id":"8ae95664_4fd51174","line":168,"range":{"start_line":168,"start_character":0,"end_line":168,"end_character":1},"in_reply_to":"b6e02fa8_18ee6d72","updated":"2021-11-12 16:01:46.000000000","message":"Yes, hosting ovn-vif under ovn-org was discussed at the ML (https://mail.openvswitch.org/pipermail/ovs-dev/2021-November/389200.html) and there was quorum on doing that in the IRC (#openvswitch) yesterday. So it is going to appear under ovn-org/ovn-vif soon.","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"346b4af75bb926b9a5729bdc20e2499d8071d0fa","unresolved":true,"context_lines":[{"line_number":222,"context_line":""},{"line_number":223,"context_line":"* The main use-case is to support allocation of VFs associated with off-path"},{"line_number":224,"context_line":"  SmartNIC DPUs and their necessary configuration at the SmartNIC DPU side;"},{"line_number":225,"context_line":"* From the operator perspective, being able to use multiple SmartNIC DPUs per"},{"line_number":226,"context_line":"  host is desirable."},{"line_number":227,"context_line":""},{"line_number":228,"context_line":"Desired Outcome Overview"},{"line_number":229,"context_line":"------------------------"}],"source_content_type":"text/x-rst","patch_set":11,"id":"0262e514_774595b5","line":226,"range":{"start_line":225,"start_character":0,"end_line":226,"end_character":20},"updated":"2021-11-12 15:28:31.000000000","message":"+1\n\nnot technically requied for minimal implemation but sicne all this entails on the nova side is passing the device serial number to neutron we should just do it right from the start.","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"6e535cf9630c7e0c8ec61f4c21241cddd23e5612","unresolved":false,"context_lines":[{"line_number":222,"context_line":""},{"line_number":223,"context_line":"* The main use-case is to support allocation of VFs associated with off-path"},{"line_number":224,"context_line":"  SmartNIC DPUs and their necessary configuration at the SmartNIC DPU side;"},{"line_number":225,"context_line":"* From the operator perspective, being able to use multiple SmartNIC DPUs per"},{"line_number":226,"context_line":"  host is desirable."},{"line_number":227,"context_line":""},{"line_number":228,"context_line":"Desired Outcome Overview"},{"line_number":229,"context_line":"------------------------"}],"source_content_type":"text/x-rst","patch_set":11,"id":"54b163cf_a2b6fe8a","line":226,"range":{"start_line":225,"start_character":0,"end_line":226,"end_character":20},"in_reply_to":"0262e514_774595b5","updated":"2021-11-15 12:09:55.000000000","message":"Ack","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"346b4af75bb926b9a5729bdc20e2499d8071d0fa","unresolved":true,"context_lines":[{"line_number":283,"context_line":"* Retrieval of the PCI card serial numbers stored in PCI VPD as presented in"},{"line_number":284,"context_line":"  node device XML format exposed by Libvirt for PFs and VFs."},{"line_number":285,"context_line":""},{"line_number":286,"context_line":"  * Whether or not PCI VPD is exposed for VFs as well as PFs is specific to"},{"line_number":287,"context_line":"    the device firmware (sometimes there is an NVRAM option to enable to expose"},{"line_number":288,"context_line":"    this data on VFs in addition to PFs) - it might be useful to populate"},{"line_number":289,"context_line":"    VF-specific information based on the PF information in case PCI VPD is not"},{"line_number":290,"context_line":"    exposed for VFs;"},{"line_number":291,"context_line":"* Store the card serial number information (if present) in the PciDevice"},{"line_number":292,"context_line":"  extra_info column under the \"vpd\" capability;"},{"line_number":293,"context_line":"* Extend the ``pci_passthrough_whitelist`` handling implementation to take"}],"source_content_type":"text/x-rst","patch_set":11,"id":"772c9fc1_d2a6d44c","line":290,"range":{"start_line":286,"start_character":1,"end_line":290,"end_character":20},"updated":"2021-11-12 15:28:31.000000000","message":"yes if the vf has this we can use it but if not we can use the info from the parent PF","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"6e535cf9630c7e0c8ec61f4c21241cddd23e5612","unresolved":false,"context_lines":[{"line_number":283,"context_line":"* Retrieval of the PCI card serial numbers stored in PCI VPD as presented in"},{"line_number":284,"context_line":"  node device XML format exposed by Libvirt for PFs and VFs."},{"line_number":285,"context_line":""},{"line_number":286,"context_line":"  * Whether or not PCI VPD is exposed for VFs as well as PFs is specific to"},{"line_number":287,"context_line":"    the device firmware (sometimes there is an NVRAM option to enable to expose"},{"line_number":288,"context_line":"    this data on VFs in addition to PFs) - it might be useful to populate"},{"line_number":289,"context_line":"    VF-specific information based on the PF information in case PCI VPD is not"},{"line_number":290,"context_line":"    exposed for VFs;"},{"line_number":291,"context_line":"* Store the card serial number information (if present) in the PciDevice"},{"line_number":292,"context_line":"  extra_info column under the \"vpd\" capability;"},{"line_number":293,"context_line":"* Extend the ``pci_passthrough_whitelist`` handling implementation to take"}],"source_content_type":"text/x-rst","patch_set":11,"id":"03672c6f_540843bf","line":290,"range":{"start_line":286,"start_character":1,"end_line":290,"end_character":20},"in_reply_to":"772c9fc1_d2a6d44c","updated":"2021-11-15 12:09:55.000000000","message":"Ack","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"346b4af75bb926b9a5729bdc20e2499d8071d0fa","unresolved":true,"context_lines":[{"line_number":290,"context_line":"    exposed for VFs;"},{"line_number":291,"context_line":"* Store the card serial number information (if present) in the PciDevice"},{"line_number":292,"context_line":"  extra_info column under the \"vpd\" capability;"},{"line_number":293,"context_line":"* Extend the ``pci_passthrough_whitelist`` handling implementation to take"},{"line_number":294,"context_line":"  ``remote_managed\u003dTrue|False`` tag into account;"},{"line_number":295,"context_line":"* For each function added to an instance, collect a PF MAC and VF logical"},{"line_number":296,"context_line":"  number as seen by the hypervisor host and pass them to Neutron along with"},{"line_number":297,"context_line":"  the card serial number during port update requests that happen during"}],"source_content_type":"text/x-rst","patch_set":11,"id":"1b043489_f33460f4","line":294,"range":{"start_line":293,"start_character":0,"end_line":294,"end_character":49},"updated":"2021-11-12 15:28:31.000000000","message":"ack, an then later we will include remote_managed\u003dtrue as a required tag when we create the pci request form the neutron port based on the vnic type correct.\n\nthis looks resonable to me both in terms of name and usage.","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"6e535cf9630c7e0c8ec61f4c21241cddd23e5612","unresolved":false,"context_lines":[{"line_number":290,"context_line":"    exposed for VFs;"},{"line_number":291,"context_line":"* Store the card serial number information (if present) in the PciDevice"},{"line_number":292,"context_line":"  extra_info column under the \"vpd\" capability;"},{"line_number":293,"context_line":"* Extend the ``pci_passthrough_whitelist`` handling implementation to take"},{"line_number":294,"context_line":"  ``remote_managed\u003dTrue|False`` tag into account;"},{"line_number":295,"context_line":"* For each function added to an instance, collect a PF MAC and VF logical"},{"line_number":296,"context_line":"  number as seen by the hypervisor host and pass them to Neutron along with"},{"line_number":297,"context_line":"  the card serial number during port update requests that happen during"}],"source_content_type":"text/x-rst","patch_set":11,"id":"8af33c7e_eea0c45f","line":294,"range":{"start_line":293,"start_character":0,"end_line":294,"end_character":49},"in_reply_to":"1b043489_f33460f4","updated":"2021-11-15 12:09:55.000000000","message":"Ack","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"346b4af75bb926b9a5729bdc20e2499d8071d0fa","unresolved":true,"context_lines":[{"line_number":297,"context_line":"  the card serial number during port update requests that happen during"},{"line_number":298,"context_line":"  instance creation (see the relevant section below for more details);"},{"line_number":299,"context_line":""},{"line_number":300,"context_line":"  * Note that if VFIO is used, this specification assumes that the ``vfio-pci``"},{"line_number":301,"context_line":"    driver will only be bound to VFs, not PFs and that PFs will be utilized for"},{"line_number":302,"context_line":"    hypervisor host purposes (e.g. connecting to the rest of the control"},{"line_number":303,"context_line":"    plane);"},{"line_number":304,"context_line":"  * Storing of VF logical number and PF MAC could be in ``extra_info`` could"},{"line_number":305,"context_line":"    be done to avoid extra database lookups;"},{"line_number":306,"context_line":"* Add logic to handle ports of type ``VNIC_SMARTNIC`` (\"smart-nic\");"}],"source_content_type":"text/x-rst","patch_set":11,"id":"96d5ea49_ae3244dd","line":303,"range":{"start_line":300,"start_character":2,"end_line":303,"end_character":11},"updated":"2021-11-12 15:28:31.000000000","message":"this would be good to capture in documentation as a note for operators but i think that is perfectly reasonable. its really just calling out the tribal knowledge we already have with regards to vfio-pci usage. Even for stand sriov we only supprot VF if the pf is not bound to vfio-pci so this is not a new requriement form that perspective.\n\nits why you cannot run dpdk on the PF and have libvirt allocat VF to vms\nlibvirt need a kernel netdev driver to be in use by the pf to interact with the vfs properly.","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"6e535cf9630c7e0c8ec61f4c21241cddd23e5612","unresolved":false,"context_lines":[{"line_number":297,"context_line":"  the card serial number during port update requests that happen during"},{"line_number":298,"context_line":"  instance creation (see the relevant section below for more details);"},{"line_number":299,"context_line":""},{"line_number":300,"context_line":"  * Note that if VFIO is used, this specification assumes that the ``vfio-pci``"},{"line_number":301,"context_line":"    driver will only be bound to VFs, not PFs and that PFs will be utilized for"},{"line_number":302,"context_line":"    hypervisor host purposes (e.g. connecting to the rest of the control"},{"line_number":303,"context_line":"    plane);"},{"line_number":304,"context_line":"  * Storing of VF logical number and PF MAC could be in ``extra_info`` could"},{"line_number":305,"context_line":"    be done to avoid extra database lookups;"},{"line_number":306,"context_line":"* Add logic to handle ports of type ``VNIC_SMARTNIC`` (\"smart-nic\");"}],"source_content_type":"text/x-rst","patch_set":11,"id":"d593067d_7180b239","line":303,"range":{"start_line":300,"start_character":2,"end_line":303,"end_character":11},"in_reply_to":"96d5ea49_ae3244dd","updated":"2021-11-15 12:09:55.000000000","message":"Ack","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"346b4af75bb926b9a5729bdc20e2499d8071d0fa","unresolved":true,"context_lines":[{"line_number":313,"context_line":"    requests containing port_ids that have ``VNIC_TYPE_SMARTNIC`` port type."},{"line_number":314,"context_line":"    Nova will need to learn to query the port type by its ID to perform that"},{"line_number":315,"context_line":"    check;"},{"line_number":316,"context_line":"* Add a new compute driver capability called ``supports_remote_managed_ports``"},{"line_number":317,"context_line":"  and a respective ``COMPUTE_REMOTE_MANAGED_PORTS`` trait to ``os-traits``;"},{"line_number":318,"context_line":""},{"line_number":319,"context_line":"  * Only the Libvirt driver will be set to have this trait since this is the"},{"line_number":320,"context_line":"    first driver to support ``remote_managed`` ports;"},{"line_number":321,"context_line":"* Implement a prefilter that will check for the presence of port ids that have"},{"line_number":322,"context_line":"  ``VNIC_TYPE_SMARTNIC`` port type and add the ``COMPUTE_REMOTE_MANAGED_PORTS``"},{"line_number":323,"context_line":"  to the request spec in this case. This will make sure that instances are"},{"line_number":324,"context_line":"  scheduled on compute nodes that have the necessary virt driver supporting"},{"line_number":325,"context_line":"  remote managed ports enabled;"},{"line_number":326,"context_line":"* Add stubs in the Nova API to prevent the following lifecycle operations for"},{"line_number":327,"context_line":"  instances with VNIC_TYPE_SMARTNIC ports:"},{"line_number":328,"context_line":""}],"source_content_type":"text/x-rst","patch_set":11,"id":"21a5f948_b415eeea","line":325,"range":{"start_line":316,"start_character":0,"end_line":325,"end_character":31},"updated":"2021-11-12 15:28:31.000000000","message":"+1\nthis will definetly help with schduling and is in line with similar exisitng compute capablity traits","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"6e535cf9630c7e0c8ec61f4c21241cddd23e5612","unresolved":false,"context_lines":[{"line_number":313,"context_line":"    requests containing port_ids that have ``VNIC_TYPE_SMARTNIC`` port type."},{"line_number":314,"context_line":"    Nova will need to learn to query the port type by its ID to perform that"},{"line_number":315,"context_line":"    check;"},{"line_number":316,"context_line":"* Add a new compute driver capability called ``supports_remote_managed_ports``"},{"line_number":317,"context_line":"  and a respective ``COMPUTE_REMOTE_MANAGED_PORTS`` trait to ``os-traits``;"},{"line_number":318,"context_line":""},{"line_number":319,"context_line":"  * Only the Libvirt driver will be set to have this trait since this is the"},{"line_number":320,"context_line":"    first driver to support ``remote_managed`` ports;"},{"line_number":321,"context_line":"* Implement a prefilter that will check for the presence of port ids that have"},{"line_number":322,"context_line":"  ``VNIC_TYPE_SMARTNIC`` port type and add the ``COMPUTE_REMOTE_MANAGED_PORTS``"},{"line_number":323,"context_line":"  to the request spec in this case. This will make sure that instances are"},{"line_number":324,"context_line":"  scheduled on compute nodes that have the necessary virt driver supporting"},{"line_number":325,"context_line":"  remote managed ports enabled;"},{"line_number":326,"context_line":"* Add stubs in the Nova API to prevent the following lifecycle operations for"},{"line_number":327,"context_line":"  instances with VNIC_TYPE_SMARTNIC ports:"},{"line_number":328,"context_line":""}],"source_content_type":"text/x-rst","patch_set":11,"id":"ff65652a_2be4909c","line":325,"range":{"start_line":316,"start_character":0,"end_line":325,"end_character":31},"in_reply_to":"21a5f948_b415eeea","updated":"2021-11-15 12:09:55.000000000","message":"Ack","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"d3064e65417b086c74c9b9355801da027ae3ec17","unresolved":false,"context_lines":[{"line_number":313,"context_line":"    requests containing port_ids that have ``VNIC_TYPE_SMARTNIC`` port type."},{"line_number":314,"context_line":"    Nova will need to learn to query the port type by its ID to perform that"},{"line_number":315,"context_line":"    check;"},{"line_number":316,"context_line":"* Add a new compute driver capability called ``supports_remote_managed_ports``"},{"line_number":317,"context_line":"  and a respective ``COMPUTE_REMOTE_MANAGED_PORTS`` trait to ``os-traits``;"},{"line_number":318,"context_line":""},{"line_number":319,"context_line":"  * Only the Libvirt driver will be set to have this trait since this is the"},{"line_number":320,"context_line":"    first driver to support ``remote_managed`` ports;"},{"line_number":321,"context_line":"* Implement a prefilter that will check for the presence of port ids that have"},{"line_number":322,"context_line":"  ``VNIC_TYPE_SMARTNIC`` port type and add the ``COMPUTE_REMOTE_MANAGED_PORTS``"},{"line_number":323,"context_line":"  to the request spec in this case. This will make sure that instances are"},{"line_number":324,"context_line":"  scheduled on compute nodes that have the necessary virt driver supporting"},{"line_number":325,"context_line":"  remote managed ports enabled;"},{"line_number":326,"context_line":"* Add stubs in the Nova API to prevent the following lifecycle operations for"},{"line_number":327,"context_line":"  instances with VNIC_TYPE_SMARTNIC ports:"},{"line_number":328,"context_line":""}],"source_content_type":"text/x-rst","patch_set":11,"id":"d450aacd_850a4b09","line":325,"range":{"start_line":316,"start_character":0,"end_line":325,"end_character":31},"in_reply_to":"bfb66db1_9da0e7e2","updated":"2021-11-16 15:42:58.000000000","message":"if you have a lot of host with pci device but only a small subset have  this off path support i think its still useful for limiting but ya the prefilet is not strictly requried, meaning you probably should implement that near the end of the serises since its a nice to have as the pci filter will indeed only pass host that have device with  remove_managed\u003dTrue present.\n\n\nthe important thing is that for the prefileter to be useful the COMPUTE_REMOTE_MANAGED_PORTS trait need to be reported only when there are device of that type avaiable on the host and the dirver supprots them rather then uncoditonally because you have a version of the livbrit drvier that support the feature.","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"77c62a384c0b3ac3b1b8d23c88d607abee083a3f","unresolved":false,"context_lines":[{"line_number":313,"context_line":"    requests containing port_ids that have ``VNIC_TYPE_SMARTNIC`` port type."},{"line_number":314,"context_line":"    Nova will need to learn to query the port type by its ID to perform that"},{"line_number":315,"context_line":"    check;"},{"line_number":316,"context_line":"* Add a new compute driver capability called ``supports_remote_managed_ports``"},{"line_number":317,"context_line":"  and a respective ``COMPUTE_REMOTE_MANAGED_PORTS`` trait to ``os-traits``;"},{"line_number":318,"context_line":""},{"line_number":319,"context_line":"  * Only the Libvirt driver will be set to have this trait since this is the"},{"line_number":320,"context_line":"    first driver to support ``remote_managed`` ports;"},{"line_number":321,"context_line":"* Implement a prefilter that will check for the presence of port ids that have"},{"line_number":322,"context_line":"  ``VNIC_TYPE_SMARTNIC`` port type and add the ``COMPUTE_REMOTE_MANAGED_PORTS``"},{"line_number":323,"context_line":"  to the request spec in this case. This will make sure that instances are"},{"line_number":324,"context_line":"  scheduled on compute nodes that have the necessary virt driver supporting"},{"line_number":325,"context_line":"  remote managed ports enabled;"},{"line_number":326,"context_line":"* Add stubs in the Nova API to prevent the following lifecycle operations for"},{"line_number":327,"context_line":"  instances with VNIC_TYPE_SMARTNIC ports:"},{"line_number":328,"context_line":""}],"source_content_type":"text/x-rst","patch_set":11,"id":"a4f04209_dbe9f999","line":325,"range":{"start_line":316,"start_character":0,"end_line":325,"end_character":31},"in_reply_to":"d450aacd_850a4b09","updated":"2021-11-24 20:11:00.000000000","message":"Took some digging to figure out how to do that. Patch set 8 contains a suggested implementation for a runtime check for the presence of the right pools in addition to the compute driver capability check:\n\nhttps://review.opendev.org/c/openstack/nova/+/812111/8/nova/compute/resource_tracker.py#1147\nhttps://review.opendev.org/c/openstack/nova/+/812111/8/nova/compute/resource_tracker.py#1102\nhttps://review.opendev.org/c/openstack/nova/+/812111/8/nova/pci/stats.py#691\n\nhttps://review.opendev.org/c/openstack/nova/+/812111/8/nova/scheduler/request_filter.py#369\n\nThis should cover the performance concern Sean raised while not introducing race conditions w.r.t. free device availability or having to do polling and regular updates of the relevant trait.\n\nThe compute capability will remain static but trait addition has a dynamic check for PCI device pool presence with the right spec.","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":9708,"name":"Balazs Gibizer","display_name":"gibi","email":"gibizer@gmail.com","username":"gibi"},"change_message_id":"d32565d794682981b668bbc3b604586d9b3749be","unresolved":false,"context_lines":[{"line_number":313,"context_line":"    requests containing port_ids that have ``VNIC_TYPE_SMARTNIC`` port type."},{"line_number":314,"context_line":"    Nova will need to learn to query the port type by its ID to perform that"},{"line_number":315,"context_line":"    check;"},{"line_number":316,"context_line":"* Add a new compute driver capability called ``supports_remote_managed_ports``"},{"line_number":317,"context_line":"  and a respective ``COMPUTE_REMOTE_MANAGED_PORTS`` trait to ``os-traits``;"},{"line_number":318,"context_line":""},{"line_number":319,"context_line":"  * Only the Libvirt driver will be set to have this trait since this is the"},{"line_number":320,"context_line":"    first driver to support ``remote_managed`` ports;"},{"line_number":321,"context_line":"* Implement a prefilter that will check for the presence of port ids that have"},{"line_number":322,"context_line":"  ``VNIC_TYPE_SMARTNIC`` port type and add the ``COMPUTE_REMOTE_MANAGED_PORTS``"},{"line_number":323,"context_line":"  to the request spec in this case. This will make sure that instances are"},{"line_number":324,"context_line":"  scheduled on compute nodes that have the necessary virt driver supporting"},{"line_number":325,"context_line":"  remote managed ports enabled;"},{"line_number":326,"context_line":"* Add stubs in the Nova API to prevent the following lifecycle operations for"},{"line_number":327,"context_line":"  instances with VNIC_TYPE_SMARTNIC ports:"},{"line_number":328,"context_line":""}],"source_content_type":"text/x-rst","patch_set":11,"id":"bfb66db1_9da0e7e2","line":325,"range":{"start_line":316,"start_character":0,"end_line":325,"end_character":31},"in_reply_to":"ff65652a_2be4909c","updated":"2021-11-16 08:32:12.000000000","message":"Wondering if we need this prefilter or not. If the instance being booted has smartnic ports then there will be InstancePciRequests with remote_managed\u003dTrue flags. So the PciFilter in the scheduling can only pass a host with PCI devices with remove_managed\u003dTrue present. So I think the prefilter is not mandatory at all. Still adding the prefilter would help limiting the number of host returned by placement. So I\u0027m OK to have it.","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"346b4af75bb926b9a5729bdc20e2499d8071d0fa","unresolved":true,"context_lines":[{"line_number":333,"context_line":"  * Suspend;"},{"line_number":334,"context_line":"  * Attach/detach a VNIC_TYPE_SMARTNIC port;"},{"line_number":335,"context_line":"  * Rebuild;"},{"line_number":336,"context_line":"* Do not wait on VIF plug events for SmartNIC ports. VIF plugging events may"},{"line_number":337,"context_line":"  arrive early from Neutron after a port update before Nova begins to wait"},{"line_number":338,"context_line":"  for those in the virt driver code. Per the Yoga Nova PTG discussion, it was"},{"line_number":339,"context_line":"  concluded that a better mechanism needs to be implemented and until that time"},{"line_number":340,"context_line":"  it is better to assume that Nova does not need to wait for plug events for"},{"line_number":341,"context_line":"  ``VNIC_TYPE_SMARTNIC`` port type."},{"line_number":342,"context_line":""},{"line_number":343,"context_line":"Identifying Port Representors"},{"line_number":344,"context_line":"-----------------------------"}],"source_content_type":"text/x-rst","patch_set":11,"id":"4337b974_a214c6f9","line":341,"range":{"start_line":336,"start_character":0,"end_line":341,"end_character":35},"updated":"2021-11-12 15:28:31.000000000","message":"well no. we should wait but we shoudl waith for bind time events only.\n\n\nso on spwaning an new vm on the host we should wait when we do bind the port for the network-vif-plugged event as we do with normal ovn today. howewver we shoudl not wait later in the driver when we create the domain xml and start the vm.\n\nwe should just extend has_bind_time_event to return true if the vnic type is smart-nic.\nhttps://github.com/openstack/nova/blob/master/nova/network/model.py#L488-L497\n\nthis can be adressed in a followup patch to this spec if you do not respin it otherwise please update this when you have time.\n\nwhile this wont guarentee that then network backend is fully set up it will make it less likely that we will race with it.\n\nbut yes you are correct that we need a better mechanisum in general to deal with this.","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"6e535cf9630c7e0c8ec61f4c21241cddd23e5612","unresolved":true,"context_lines":[{"line_number":333,"context_line":"  * Suspend;"},{"line_number":334,"context_line":"  * Attach/detach a VNIC_TYPE_SMARTNIC port;"},{"line_number":335,"context_line":"  * Rebuild;"},{"line_number":336,"context_line":"* Do not wait on VIF plug events for SmartNIC ports. VIF plugging events may"},{"line_number":337,"context_line":"  arrive early from Neutron after a port update before Nova begins to wait"},{"line_number":338,"context_line":"  for those in the virt driver code. Per the Yoga Nova PTG discussion, it was"},{"line_number":339,"context_line":"  concluded that a better mechanism needs to be implemented and until that time"},{"line_number":340,"context_line":"  it is better to assume that Nova does not need to wait for plug events for"},{"line_number":341,"context_line":"  ``VNIC_TYPE_SMARTNIC`` port type."},{"line_number":342,"context_line":""},{"line_number":343,"context_line":"Identifying Port Representors"},{"line_number":344,"context_line":"-----------------------------"}],"source_content_type":"text/x-rst","patch_set":11,"id":"f1cd2517_2cfde67c","line":341,"range":{"start_line":336,"start_character":0,"end_line":341,"end_character":35},"in_reply_to":"4337b974_a214c6f9","updated":"2021-11-15 12:09:55.000000000","message":"I will rephrase this to avoid too many implementation-specific details and later raise a follow-up.\n\nIn testing we hit an issue in the virt driver code since \"network-vif-plugged\" arrives early before `_create_guest_with_network` sets up a wait for `network-vif-plugged` events via `virtapi.wait_for_instance_event`. So the event is discarded and then the virt driver code blocks waiting for an already discarded event:\n                         \n1. Neutron notifies Nova when the port status becomes \"UP\" after Nova does a port update:\nhttps://github.com/openstack/neutron/blob/a922e853150003014d03da519d17e6f21d2a333e/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py#L1062-L1063\n\n2. Nova catches up and sets up a wait for the \"network-vif-plugged\" event:\n * https://github.com/openstack/nova/blob/e28afc564700a1a35e3bf0269687d5734251b88a/nova/virt/libvirt/driver.py#L7221-L7240\n * https://github.com/openstack/nova/blob/e28afc564700a1a35e3bf0269687d5734251b88a/nova/virt/libvirt/driver.py#L7191-L7198\n \n\nIt looked like this:\n\n```\n\n2021-10-18 09:32:39.301 821806 DEBUG nova.compute.manager [req-df431bd7-edf6-44fd-ae12-37bc8b9b7207 f2ec7cf280b8484e971f496a4b4e1082 482dd29b860841d9969abf5a5818078c - 263fddd8a9e4401da7569b3b0cb88364 263fddd8a9\ne4401da7569b3b0cb88364] [instance: 2d9fc853-074e-418b-b784-9ec4705dff25] Received event network-vif-plugged-7f6dae4d-8ed5-422b-9b7a-c723bf5c7e66 external_instance_event /usr/lib/python3/dist-packages/nova/compute/manager.py:10494\n2021-10-18 09:32:39.302 821806 DEBUG nova.compute.manager [req-df431bd7-edf6-44fd-ae12-37bc8b9b7207 f2ec7cf280b8484e971f496a4b4e1082 482dd29b860841d9969abf5a5818078c - 263fddd8a9e4401da7569b3b0cb88364 263fddd8a9e4401da7569b3b0cb88364] [instance: 2d9fc853-074e-418b-b784-9ec4705dff25] No waiting events found dispatching network-vif-plugged-7f6dae4d-8ed5-422b-9b7a-c723bf5c7e66 pop_instance_event /usr/lib/python3/dist-packages/nova/compute/manager.py:317\n2021-10-18 09:32:39.303 821806 WARNING nova.compute.manager [req-df431bd7-edf6-44fd-ae12-37bc8b9b7207 f2ec7cf280b8484e971f496a4b4e1082 482dd29b860841d9969abf5a5818078c - 263fddd8a9e4401da7569b3b0cb88364 263fddd8a9e4401da7569b3b0cb88364] [instance: 2d9fc853-074e-418b-b784-9ec4705dff25] Received unexpected event network-vif-plugged-7f6dae4d-8ed5-422b-9b7a-c723bf5c7e66 for instance with vm_state building and task_state spawning.\n\n2021-10-18 09:32:39.458 821806 DEBUG nova.network.neutron [req-cb3b35e4-fb65-4fe8-9352-b611ca3566ad e49f2e4d47bf447c801793b364babea4 62f990de2ca24ea48228c5ceb9292f0f - c23d0caef00e4f4fb20202765c4650fb c23d0caef00e4f4fb20202765c4650fb] [instance: 2d9fc853-074e-418b-b784-9ec4705dff25] Updating instance_info_cache with network_info: [{\"id\": \"7f6dae4d-8ed5-422b-9b7a-c723bf5c7e66\", \"address\": \"fa:16:3e:d0:fe:7c\", \"network\": {\"id\": \"ed2cceda-b3e5-42de-b243-3d34cdc9208d\", \"bridge\": \"br-int\", \"label\": \"network\", \"subnets\": [{\"cidr\": \"10.42.0.0/22\", \"dns\": [], \"gateway\": {\"address\": \"10.42.0.1\", \"type\": \"gateway\", \"version\": 4, \"meta\": {}}, \"ips\": [{\"address\": \"10.42.0.166\", \"type\": \"fixed\", \"version\": 4, \"meta\": {}, \"floating_ips\": []}], \"routes\": [], \"version\": 4, \"meta\": {\"dhcp_server\": \"10.42.0.1\"}}], \"meta\": {\"injected\": false, \"tenant_id\": \"62f990de2ca24ea48228c5ceb9292f0f\", \"mtu\": 8942, \"physical_network\": null, \"tunneled\": true}}, \"type\": \"ovs\", \"details\": {\"port_filter\": true}, \"devname\": \"tap7f6dae4d-8e\", \"ovs_interfaceid\": \"7f6dae4d-8ed5-422b-9b7a-c723bf5c7e66\", \"qbh_params\": null, \"qbg_params\": null, \"active\": false, \"vnic_type\": \"smart-nic\", \"profile\": {\"pci_vendor_info\": \"15b3:101e\", \"pci_slot\": \"0000:82:01.2\", \"physical_network\": null, \"card_serial_number\": \"MT2113X00000\", \"pf_mac_address\": \"b8:ce:f6:be:ef:ca\", \"vf_num\": \"7\"}, \"preserve_on_delete\": true, \"delegate_create\": true, \"meta\": {}}] update_instance_cache_with_nw_info /usr/lib/python3/dist-packages/nova/network/neutron.py:117\n\n2021-10-18 09:32:39.502 821806 DEBUG nova.compute.manager [req-cb3b35e4-fb65-4fe8-9352-b611ca3566ad e49f2e4d47bf447c801793b364babea4 62f990de2ca24ea48228c5ceb9292f0f - c23d0caef00e4f4fb20202765c4650fb c23d0caef00e4f4fb20202765c4650fb] [instance: 2d9fc853-074e-418b-b784-9ec4705dff25] Preparing to wait for external event network-vif-plugged-7f6dae4d-8ed5-422b-9b7a-c723bf5c7e66 prepare_for_instance_event /usr/lib/python3/dist-packages/nova/compute/manager.py:280\n\n\n2021-10-18 09:37:42.850 821806 WARNING nova.virt.libvirt.driver [req-cb3b35e4-fb65-4fe8-9352-b611ca3566ad e49f2e4d47bf447c801793b364babea4 62f990de2ca24ea48228c5ceb9292f0f - c23d0caef00e4f4fb20202765c4650fb c23d0caef00e4f4fb20202765c4650fb] [instance: 2d9fc853-074e-418b-b784-9ec4705dff25] Timeout waiting for [(\u0027network-vif-plugged\u0027, \u00277f6dae4d-8ed5-422b-9b7a-c723bf5c7e66\u0027)] for instance with vm_state building and task_state spawning: eventlet.timeout.Timeout: 300 seconds\n\n\n2021-10-18 09:37:43.714 821806 ERROR nova.compute.manager [req-cb3b35e4-fb65-4fe8-9352-b611ca3566ad e49f2e4d47bf447c801793b364babea4 62f990de2ca24ea48228c5ceb9292f0f - c23d0caef00e4f4fb20202765c4650fb c23d0caef00e4f4fb20202765c4650fb] [instance: 2d9fc853-074e-418b-b784-9ec4705dff25] Instance failed to spawn: nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed\n\n```","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"38b20f0c19bc2851118f7a7b1cbf06c9fb0b5c7f","unresolved":false,"context_lines":[{"line_number":333,"context_line":"  * Suspend;"},{"line_number":334,"context_line":"  * Attach/detach a VNIC_TYPE_SMARTNIC port;"},{"line_number":335,"context_line":"  * Rebuild;"},{"line_number":336,"context_line":"* Do not wait on VIF plug events for SmartNIC ports. VIF plugging events may"},{"line_number":337,"context_line":"  arrive early from Neutron after a port update before Nova begins to wait"},{"line_number":338,"context_line":"  for those in the virt driver code. Per the Yoga Nova PTG discussion, it was"},{"line_number":339,"context_line":"  concluded that a better mechanism needs to be implemented and until that time"},{"line_number":340,"context_line":"  it is better to assume that Nova does not need to wait for plug events for"},{"line_number":341,"context_line":"  ``VNIC_TYPE_SMARTNIC`` port type."},{"line_number":342,"context_line":""},{"line_number":343,"context_line":"Identifying Port Representors"},{"line_number":344,"context_line":"-----------------------------"}],"source_content_type":"text/x-rst","patch_set":11,"id":"860a0dad_2d3b9d25","line":341,"range":{"start_line":336,"start_character":0,"end_line":341,"end_character":35},"in_reply_to":"f1cd2517_2cfde67c","updated":"2021-11-15 13:30:09.000000000","message":"Discussed on IRC:\n\nhttps://meetings.opendev.org/irclogs/%23openstack-nova/%23openstack-nova.2021-11-15.log.html#t2021-11-15T13:10:49\n\n\n* extend `VIF.has_bind_time_event` to handle `VNIC_TYPE_SMARTNIC`;\n* fix `_get_neutron_events` to call `vif.has_bind_time_event` for event filtering.\n\nhttps://github.com/openstack/nova/blob/e28afc564700a1a35e3bf0269687d5734251b88a/nova/virt/libvirt/driver.py#L7191-L7198","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"346b4af75bb926b9a5729bdc20e2499d8071d0fa","unresolved":true,"context_lines":[{"line_number":480,"context_line":""},{"line_number":481,"context_line":"Supporting one SmartNIC DPU per host initially and extending it at a later"},{"line_number":482,"context_line":"point has been discarded due to difficulties in the data model extension."},{"line_number":483,"context_line":""},{"line_number":484,"context_line":"Data model impact"},{"line_number":485,"context_line":"-----------------"},{"line_number":486,"context_line":""}],"source_content_type":"text/x-rst","patch_set":11,"id":"563ed077_fae6eddf","line":483,"updated":"2021-11-12 15:28:31.000000000","message":"yep that and as described in this spec supporting multiple form the start does not add much complexity so there is littel value in limiting the scope to just one device per host.","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"6e535cf9630c7e0c8ec61f4c21241cddd23e5612","unresolved":false,"context_lines":[{"line_number":480,"context_line":""},{"line_number":481,"context_line":"Supporting one SmartNIC DPU per host initially and extending it at a later"},{"line_number":482,"context_line":"point has been discarded due to difficulties in the data model extension."},{"line_number":483,"context_line":""},{"line_number":484,"context_line":"Data model impact"},{"line_number":485,"context_line":"-----------------"},{"line_number":486,"context_line":""}],"source_content_type":"text/x-rst","patch_set":11,"id":"3512b432_b70df164","line":483,"in_reply_to":"563ed077_fae6eddf","updated":"2021-11-15 12:09:55.000000000","message":"Ack","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"94e2b0abc5328916307f680ac9b3d287ad262a7a","unresolved":true,"context_lines":[{"line_number":699,"context_line":"In order to make this useful overall there are additional cross-project"},{"line_number":700,"context_line":"changes required. Specifically, to make this work with OVN:"},{"line_number":701,"context_line":""},{"line_number":702,"context_line":"* ovn-controller needs to learn how to plug representors into correct bridges"},{"line_number":703,"context_line":"  at the SmartNIC DPU node side since the os-vif-like functionality to hook VFs"},{"line_number":704,"context_line":"  up is still needed;"},{"line_number":705,"context_line":""}],"source_content_type":"text/x-rst","patch_set":11,"id":"242fae8c_1a1510ee","line":702,"updated":"2021-11-11 13:26:49.000000000","message":"Note that the relevant changes to OVN got accepted in the end after several iterations.\n\nhttps://patchwork.ozlabs.org/project/ovn/list/?series\u003d267834\u0026state\u003d3\u0026archive\u003dboth\nhttps://patchwork.ozlabs.org/project/ovn/list/?state\u003d*\u0026series\u003d270569\nhttps://github.com/openvswitch/ovs/commit/bfee9f6c011518c7690d3ce3b290a2b7189a377d\n\nIf any more changes are requested to the spec, I\u0027ll update the references here.","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"346b4af75bb926b9a5729bdc20e2499d8071d0fa","unresolved":true,"context_lines":[{"line_number":699,"context_line":"In order to make this useful overall there are additional cross-project"},{"line_number":700,"context_line":"changes required. Specifically, to make this work with OVN:"},{"line_number":701,"context_line":""},{"line_number":702,"context_line":"* ovn-controller needs to learn how to plug representors into correct bridges"},{"line_number":703,"context_line":"  at the SmartNIC DPU node side since the os-vif-like functionality to hook VFs"},{"line_number":704,"context_line":"  up is still needed;"},{"line_number":705,"context_line":""}],"source_content_type":"text/x-rst","patch_set":11,"id":"7dfa6e52_fdb50b9f","line":702,"in_reply_to":"242fae8c_1a1510ee","updated":"2021-11-12 15:28:31.000000000","message":"ack that great to hear.","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"6e535cf9630c7e0c8ec61f4c21241cddd23e5612","unresolved":false,"context_lines":[{"line_number":699,"context_line":"In order to make this useful overall there are additional cross-project"},{"line_number":700,"context_line":"changes required. Specifically, to make this work with OVN:"},{"line_number":701,"context_line":""},{"line_number":702,"context_line":"* ovn-controller needs to learn how to plug representors into correct bridges"},{"line_number":703,"context_line":"  at the SmartNIC DPU node side since the os-vif-like functionality to hook VFs"},{"line_number":704,"context_line":"  up is still needed;"},{"line_number":705,"context_line":""}],"source_content_type":"text/x-rst","patch_set":11,"id":"875642f9_2bc7c2ea","line":702,"in_reply_to":"7dfa6e52_fdb50b9f","updated":"2021-11-15 12:09:55.000000000","message":"Done","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"94e2b0abc5328916307f680ac9b3d287ad262a7a","unresolved":true,"context_lines":[{"line_number":763,"context_line":".. [13] https://opendev.org/openstack/neutron/src/tag/19.0.0/neutron/plugins/ml2/drivers/mech_agent.py#L109-L116"},{"line_number":764,"context_line":".. [14] https://opendev.org/openstack/ironic-specs/commit/f358fbdde9a1cadc838327b8bf34ee54a7e7f43a"},{"line_number":765,"context_line":".. [15] https://docs.openstack.org/api-ref/network/v2/index.html?expanded\u003dcreate-port-detail#id72"},{"line_number":766,"context_line":".. [16] https://patchwork.ozlabs.org/project/ovn/list/?series\u003d267834\u0026state\u003d%2A\u0026archive\u003dboth"},{"line_number":767,"context_line":".. [17] https://github.com/fnordahl/ovn-vif"},{"line_number":768,"context_line":".. [18] https://bugs.launchpad.net/neutron/+bug/1932154"},{"line_number":769,"context_line":".. [19] https://review.opendev.org/c/openstack/neutron-specs/+/788821"}],"source_content_type":"text/x-rst","patch_set":11,"id":"b4da3bb4_a4db53d6","line":766,"range":{"start_line":766,"start_character":0,"end_line":766,"end_character":91},"updated":"2021-11-11 13:26:49.000000000","message":"Note that the relevant changes to OVN got accepted in the end after several iterations.\n\nhttps://patchwork.ozlabs.org/project/ovn/list/?series\u003d267834\u0026state\u003d3\u0026archive\u003dboth\nhttps://patchwork.ozlabs.org/project/ovn/list/?state\u003d*\u0026series\u003d270569\nhttps://github.com/openvswitch/ovs/commit/bfee9f6c011518c7690d3ce3b290a2b7189a377d\n\nIf any more changes are requested to the spec, I\u0027ll update the references here.","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"6e535cf9630c7e0c8ec61f4c21241cddd23e5612","unresolved":false,"context_lines":[{"line_number":763,"context_line":".. [13] https://opendev.org/openstack/neutron/src/tag/19.0.0/neutron/plugins/ml2/drivers/mech_agent.py#L109-L116"},{"line_number":764,"context_line":".. [14] https://opendev.org/openstack/ironic-specs/commit/f358fbdde9a1cadc838327b8bf34ee54a7e7f43a"},{"line_number":765,"context_line":".. [15] https://docs.openstack.org/api-ref/network/v2/index.html?expanded\u003dcreate-port-detail#id72"},{"line_number":766,"context_line":".. [16] https://patchwork.ozlabs.org/project/ovn/list/?series\u003d267834\u0026state\u003d%2A\u0026archive\u003dboth"},{"line_number":767,"context_line":".. [17] https://github.com/fnordahl/ovn-vif"},{"line_number":768,"context_line":".. [18] https://bugs.launchpad.net/neutron/+bug/1932154"},{"line_number":769,"context_line":".. [19] https://review.opendev.org/c/openstack/neutron-specs/+/788821"}],"source_content_type":"text/x-rst","patch_set":11,"id":"b575e1cd_d728b059","line":766,"range":{"start_line":766,"start_character":0,"end_line":766,"end_character":91},"in_reply_to":"b4da3bb4_a4db53d6","updated":"2021-11-15 12:09:55.000000000","message":"Done","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"94e2b0abc5328916307f680ac9b3d287ad262a7a","unresolved":true,"context_lines":[{"line_number":769,"context_line":".. [19] https://review.opendev.org/c/openstack/neutron-specs/+/788821"},{"line_number":770,"context_line":".. [20] https://review.opendev.org/c/openstack/neutron/+/808961"},{"line_number":771,"context_line":".. [21] https://gitlab.com/libvirt/libvirt/-/commits/master?search\u003dPCI.*VPD"},{"line_number":772,"context_line":".. [22] https://listman.redhat.com/archives/libvir-list/2021-October/msg00874.html"},{"line_number":773,"context_line":".. [23] https://review.opendev.org/c/openstack/nova-specs/+/791047"}],"source_content_type":"text/x-rst","patch_set":11,"id":"1cdff427_698f2112","line":772,"range":{"start_line":772,"start_character":8,"end_line":772,"end_character":82},"updated":"2021-11-11 13:26:49.000000000","message":"https://listman.redhat.com/archives/libvir-list/2021-November/msg00303.html","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"6e535cf9630c7e0c8ec61f4c21241cddd23e5612","unresolved":false,"context_lines":[{"line_number":769,"context_line":".. [19] https://review.opendev.org/c/openstack/neutron-specs/+/788821"},{"line_number":770,"context_line":".. [20] https://review.opendev.org/c/openstack/neutron/+/808961"},{"line_number":771,"context_line":".. [21] https://gitlab.com/libvirt/libvirt/-/commits/master?search\u003dPCI.*VPD"},{"line_number":772,"context_line":".. [22] https://listman.redhat.com/archives/libvir-list/2021-October/msg00874.html"},{"line_number":773,"context_line":".. [23] https://review.opendev.org/c/openstack/nova-specs/+/791047"}],"source_content_type":"text/x-rst","patch_set":11,"id":"c545a0d6_caaf6d09","line":772,"range":{"start_line":772,"start_character":8,"end_line":772,"end_character":82},"in_reply_to":"1cdff427_698f2112","updated":"2021-11-15 12:09:55.000000000","message":"Done","commit_id":"d29a5db0ecbcd7b534135c61934af4484db8cf45"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"d3064e65417b086c74c9b9355801da027ae3ec17","unresolved":true,"context_lines":[{"line_number":295,"context_line":"    exposed for VFs;"},{"line_number":296,"context_line":"* Store the card serial number information (if present) in the PciDevice"},{"line_number":297,"context_line":"  extra_info column under the \"vpd\" capability;"},{"line_number":298,"context_line":"* Extend the ``pci_passthrough_whitelist`` handling implementation to take"},{"line_number":299,"context_line":"  ``remote_managed\u003dTrue|False`` tag into account;"},{"line_number":300,"context_line":"* For each function added to an instance, collect a PF MAC and VF logical"},{"line_number":301,"context_line":"  number as seen by the hypervisor host and pass them to Neutron along with"},{"line_number":302,"context_line":"  the card serial number during port update requests that happen during"}],"source_content_type":"text/x-rst","patch_set":14,"id":"151b552d_a6334c69","line":299,"range":{"start_line":298,"start_character":1,"end_line":299,"end_character":49},"updated":"2021-11-16 15:42:58.000000000","message":"nit we can evaulate this more durign the implementaiton review but im not sure we will really need\nremote_managed\u003dFalse\n\nthe current WIP will add that to any device that does not have remote_managed\u003dTrue\n\nwe could get a similar effect without the db migration by using a filter to remvoe remote mandaged device when they are not requested as we do with PFs and VDPA\n\nhttps://github.com/openstack/nova/blob/master/nova/pci/stats.py#L520-L552\n\nthat is the normal approach we take but both will work.\n\nthe other apporch would be to excluede all pci device with a physical_network tag form being select via a flavor alias.\n\ni think that is arguable teh correct thing to do in general but its out of scope of this spec and its likely a latent bug.","commit_id":"ba6772ae23d3a713fc297431366dd5b26ed02809"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"69384ed69f8bea4ab48dbf32436cd9cf23a0ef5f","unresolved":true,"context_lines":[{"line_number":295,"context_line":"    exposed for VFs;"},{"line_number":296,"context_line":"* Store the card serial number information (if present) in the PciDevice"},{"line_number":297,"context_line":"  extra_info column under the \"vpd\" capability;"},{"line_number":298,"context_line":"* Extend the ``pci_passthrough_whitelist`` handling implementation to take"},{"line_number":299,"context_line":"  ``remote_managed\u003dTrue|False`` tag into account;"},{"line_number":300,"context_line":"* For each function added to an instance, collect a PF MAC and VF logical"},{"line_number":301,"context_line":"  number as seen by the hypervisor host and pass them to Neutron along with"},{"line_number":302,"context_line":"  the card serial number during port update requests that happen during"}],"source_content_type":"text/x-rst","patch_set":14,"id":"65234958_2cde6c47","line":299,"range":{"start_line":298,"start_character":1,"end_line":299,"end_character":49},"in_reply_to":"151b552d_a6334c69","updated":"2021-11-25 12:30:31.000000000","message":"Added filtering in patch set 9:\n\nhttps://review.opendev.org/c/openstack/nova/+/812111/9/nova/pci/stats.py#482\nhttps://review.opendev.org/c/openstack/nova/+/812111/9/nova/tests/unit/pci/test_stats.py#887","commit_id":"ba6772ae23d3a713fc297431366dd5b26ed02809"},{"author":{"_account_id":24824,"name":"Dmitrii Shcherbakov","username":"dmitriis"},"change_message_id":"4fcee4a655401b02c1f976032beec62761d04585","unresolved":true,"context_lines":[{"line_number":735,"context_line":"  up is still needed;"},{"line_number":736,"context_line":""},{"line_number":737,"context_line":"  * Representor plugging and related OVN changes: [16]_ [17]_ [24]_ (note that"},{"line_number":738,"context_line":"    [24]_ will be hosted at [25]_ shortly per [26]_ and a follow-up discussion"},{"line_number":739,"context_line":"    that happened in the #openvswitch IRC channel);"},{"line_number":740,"context_line":"* The OVN driver code in Neutron needs to learn about SmartNIC DPU node"},{"line_number":741,"context_line":"  hostnames and respective PCIe add-in-card serial numbers gathered via VPD:"}],"source_content_type":"text/x-rst","patch_set":14,"id":"b4c991a0_f6ec76f8","line":738,"range":{"start_line":738,"start_character":14,"end_line":738,"end_character":33},"updated":"2021-11-16 10:40:48.000000000","message":"Note that this actually happened today (see https://github.com/ovn-org/ovn-vif) so this is no longer a forward-looking statement.","commit_id":"ba6772ae23d3a713fc297431366dd5b26ed02809"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"d3064e65417b086c74c9b9355801da027ae3ec17","unresolved":false,"context_lines":[{"line_number":735,"context_line":"  up is still needed;"},{"line_number":736,"context_line":""},{"line_number":737,"context_line":"  * Representor plugging and related OVN changes: [16]_ [17]_ [24]_ (note that"},{"line_number":738,"context_line":"    [24]_ will be hosted at [25]_ shortly per [26]_ and a follow-up discussion"},{"line_number":739,"context_line":"    that happened in the #openvswitch IRC channel);"},{"line_number":740,"context_line":"* The OVN driver code in Neutron needs to learn about SmartNIC DPU node"},{"line_number":741,"context_line":"  hostnames and respective PCIe add-in-card serial numbers gathered via VPD:"}],"source_content_type":"text/x-rst","patch_set":14,"id":"f17f8adf_6a54da01","line":738,"range":{"start_line":738,"start_character":14,"end_line":738,"end_character":33},"in_reply_to":"b4c991a0_f6ec76f8","updated":"2021-11-16 15:42:58.000000000","message":"Ack","commit_id":"ba6772ae23d3a713fc297431366dd5b26ed02809"}]}