)]}'
{"specs/mitaka/approved/sriov-physical-function-passthrough.rst":[{"author":{"_account_id":6772,"name":"Stephen Gordon","email":"sgordon@redhat.com","username":"sgordon"},"change_message_id":"5790ccc78f91c0a4bb35d42dd4979e9a47891229","unresolved":false,"context_lines":[{"line_number":16,"context_line":""},{"line_number":17,"context_line":"In the case of SR-IOV enabled cards, it is possible to treat the card either as"},{"line_number":18,"context_line":"a number of virtual devices (called VFs - virtual functions) or as a full"},{"line_number":19,"context_line":"device (PF - physical function)."},{"line_number":20,"context_line":""},{"line_number":21,"context_line":"Nova\u0027s current handling exposes only virtual functions as resources that can"},{"line_number":22,"context_line":"be requested by instances - and this is the most common use case by far."}],"source_content_type":"text/x-rst","patch_set":1,"id":"1a4dcd0f_bc41ef64","line":19,"updated":"2015-08-13 12:01:02.000000000","message":"May not always be the full card, PF may also relate to one port on a multi-port card.","commit_id":"ead05233a91c38684d0f0d1183c71b5372df4b65"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"adcc6fff48600ab935aaa4caa31f05917bcf7c18","unresolved":false,"context_lines":[{"line_number":16,"context_line":""},{"line_number":17,"context_line":"In the case of SR-IOV enabled cards, it is possible to treat the card either as"},{"line_number":18,"context_line":"a number of virtual devices (called VFs - virtual functions) or as a full"},{"line_number":19,"context_line":"device (PF - physical function)."},{"line_number":20,"context_line":""},{"line_number":21,"context_line":"Nova\u0027s current handling exposes only virtual functions as resources that can"},{"line_number":22,"context_line":"be requested by instances - and this is the most common use case by far."}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a1a9d01_72419aa8","line":19,"in_reply_to":"1a4dcd0f_bc41ef64","updated":"2015-09-29 15:37:37.000000000","message":"Done - will replace card with port in the rest of the document.","commit_id":"ead05233a91c38684d0f0d1183c71b5372df4b65"},{"author":{"_account_id":6772,"name":"Stephen Gordon","email":"sgordon@redhat.com","username":"sgordon"},"change_message_id":"5790ccc78f91c0a4bb35d42dd4979e9a47891229","unresolved":false,"context_lines":[{"line_number":56,"context_line":""},{"line_number":57,"context_line":"Certain NFV workloads may need to have the full control of the physical device,"},{"line_number":58,"context_line":"in order to avoid any of the VF overhead and functionality, and to be able to"},{"line_number":59,"context_line":"exclusively use the full bandwidth of the card. However, due to the dynamic"},{"line_number":60,"context_line":"nature of the elastic cloud, and the promise of Nova to we do not wish to have"},{"line_number":61,"context_line":"to pre-provision certain SR-IOV cards to be used as PFs as this defeats the"},{"line_number":62,"context_line":"promise of the infrastructure management tool that allows for quick"}],"source_content_type":"text/x-rst","patch_set":1,"id":"1a4dcd0f_9f48a52e","line":59,"updated":"2015-08-13 12:01:02.000000000","message":"My recollection is there are certain L2 operations you can do with a PF but not a VF? We aren\u0027t really giving them full control of the raw device here but rather a specific port.","commit_id":"ead05233a91c38684d0f0d1183c71b5372df4b65"},{"author":{"_account_id":8802,"name":"Vladik Romanovsky","email":"vromanso@redhat.com","username":"vladikr"},"change_message_id":"a306a2153ffaf4bbbf8465272e4d7b3280021e22","unresolved":false,"context_lines":[{"line_number":56,"context_line":""},{"line_number":57,"context_line":"Certain NFV workloads may need to have the full control of the physical device,"},{"line_number":58,"context_line":"in order to avoid any of the VF overhead and functionality, and to be able to"},{"line_number":59,"context_line":"exclusively use the full bandwidth of the card. However, due to the dynamic"},{"line_number":60,"context_line":"nature of the elastic cloud, and the promise of Nova to we do not wish to have"},{"line_number":61,"context_line":"to pre-provision certain SR-IOV cards to be used as PFs as this defeats the"},{"line_number":62,"context_line":"promise of the infrastructure management tool that allows for quick"}],"source_content_type":"text/x-rst","patch_set":1,"id":"fa1b9901_4cb40d81","line":59,"in_reply_to":"1a4dcd0f_9f48a52e","updated":"2015-08-19 22:09:11.000000000","message":"I think in the end it mostly came down to not being able to set up Promiscuous unicast on VFs, only on PF.\n\nThe explanation looked like:\nPromiscuous unicast,\nNeeded by L2 services that needs a VNF to attract all the inbound traffic in one interface, a VLAN, QinQ, regardless of the destination MAC (the MAC does not belong to the VNF, but to another system at the other end of the service)","commit_id":"ead05233a91c38684d0f0d1183c71b5372df4b65"},{"author":{"_account_id":12171,"name":"Moshe Levi","email":"moshele@nvidia.com","username":"moshele"},"change_message_id":"0435a85f98b4264f9e621d21ed587a9088be8d4d","unresolved":false,"context_lines":[{"line_number":190,"context_line":"  consumed or claimed."},{"line_number":191,"context_line":""},{"line_number":192,"context_line":"* Add code to allow for filtering out the PFs (or VFs) with the whitelist"},{"line_number":193,"context_line":"  config option, and remove the hard-coded check in the PCIDevTracker class."},{"line_number":194,"context_line":""},{"line_number":195,"context_line":"* Add support for specifying the PF requirement through the pci_alias"},{"line_number":196,"context_line":"  configuration options, so that it can be requested through flavor"}],"source_content_type":"text/x-rst","patch_set":1,"id":"ba15a1d1_7f1f1c08","line":193,"updated":"2015-09-17 08:26:07.000000000","message":"so I had an approved bp for enhance PCI passthrough whitelist to support regex in Liberty but I missed the deadline for the code.\nIt seems to be related to what you need.\nblueprint: https://review.openstack.org/#/c/213428/\ncode: https://review.openstack.org/#/c/199488/","commit_id":"ead05233a91c38684d0f0d1183c71b5372df4b65"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"adcc6fff48600ab935aaa4caa31f05917bcf7c18","unresolved":false,"context_lines":[{"line_number":190,"context_line":"  consumed or claimed."},{"line_number":191,"context_line":""},{"line_number":192,"context_line":"* Add code to allow for filtering out the PFs (or VFs) with the whitelist"},{"line_number":193,"context_line":"  config option, and remove the hard-coded check in the PCIDevTracker class."},{"line_number":194,"context_line":""},{"line_number":195,"context_line":"* Add support for specifying the PF requirement through the pci_alias"},{"line_number":196,"context_line":"  configuration options, so that it can be requested through flavor"}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a1a9d01_2cc75097","line":193,"in_reply_to":"ba15a1d1_7f1f1c08","updated":"2015-09-29 15:37:37.000000000","message":"Your proposal seems very useful. Looking at it some more I don\u0027t think any work here is necessary actually (though will need to try it out) other than just removing the hard-coded check in the libvirt driver. Will update the spec to reflect this.","commit_id":"ead05233a91c38684d0f0d1183c71b5372df4b65"},{"author":{"_account_id":12171,"name":"Moshe Levi","email":"moshele@nvidia.com","username":"moshele"},"change_message_id":"0435a85f98b4264f9e621d21ed587a9088be8d4d","unresolved":false,"context_lines":[{"line_number":191,"context_line":""},{"line_number":192,"context_line":"* Add code to allow for filtering out the PFs (or VFs) with the whitelist"},{"line_number":193,"context_line":"  config option, and remove the hard-coded check in the PCIDevTracker class."},{"line_number":194,"context_line":""},{"line_number":195,"context_line":"* Add support for specifying the PF requirement through the pci_alias"},{"line_number":196,"context_line":"  configuration options, so that it can be requested through flavor"},{"line_number":197,"context_line":"  extra-specs."}],"source_content_type":"text/x-rst","patch_set":1,"id":"ba15a1d1_5598f7bf","line":194,"updated":"2015-09-17 08:26:07.000000000","message":"shouldn\u0027t there be also change in  PciPassthroughFilter?","commit_id":"ead05233a91c38684d0f0d1183c71b5372df4b65"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"adcc6fff48600ab935aaa4caa31f05917bcf7c18","unresolved":false,"context_lines":[{"line_number":191,"context_line":""},{"line_number":192,"context_line":"* Add code to allow for filtering out the PFs (or VFs) with the whitelist"},{"line_number":193,"context_line":"  config option, and remove the hard-coded check in the PCIDevTracker class."},{"line_number":194,"context_line":""},{"line_number":195,"context_line":"* Add support for specifying the PF requirement through the pci_alias"},{"line_number":196,"context_line":"  configuration options, so that it can be requested through flavor"},{"line_number":197,"context_line":"  extra-specs."}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a1a9d01_d237464b","line":194,"in_reply_to":"ba15a1d1_5598f7bf","updated":"2015-09-29 15:37:37.000000000","message":"Technically yes, but since the filter only calls into the pci_stats code that is really covered by the second work item, I didn\u0027t feel the need to mention it. It might be clearer if I do point it out though.","commit_id":"ead05233a91c38684d0f0d1183c71b5372df4b65"},{"author":{"_account_id":12171,"name":"Moshe Levi","email":"moshele@nvidia.com","username":"moshele"},"change_message_id":"0435a85f98b4264f9e621d21ed587a9088be8d4d","unresolved":false,"context_lines":[{"line_number":192,"context_line":"* Add code to allow for filtering out the PFs (or VFs) with the whitelist"},{"line_number":193,"context_line":"  config option, and remove the hard-coded check in the PCIDevTracker class."},{"line_number":194,"context_line":""},{"line_number":195,"context_line":"* Add support for specifying the PF requirement through the pci_alias"},{"line_number":196,"context_line":"  configuration options, so that it can be requested through flavor"},{"line_number":197,"context_line":"  extra-specs."},{"line_number":198,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"ba15a1d1_50d0051f","line":195,"updated":"2015-09-17 08:26:07.000000000","message":"what do you mean by PF requirement? is it if it has SR-IOV capabilities?","commit_id":"ead05233a91c38684d0f0d1183c71b5372df4b65"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"adcc6fff48600ab935aaa4caa31f05917bcf7c18","unresolved":false,"context_lines":[{"line_number":192,"context_line":"* Add code to allow for filtering out the PFs (or VFs) with the whitelist"},{"line_number":193,"context_line":"  config option, and remove the hard-coded check in the PCIDevTracker class."},{"line_number":194,"context_line":""},{"line_number":195,"context_line":"* Add support for specifying the PF requirement through the pci_alias"},{"line_number":196,"context_line":"  configuration options, so that it can be requested through flavor"},{"line_number":197,"context_line":"  extra-specs."},{"line_number":198,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a1a9d01_4c02c43e","line":195,"in_reply_to":"ba15a1d1_50d0051f","updated":"2015-09-29 15:37:37.000000000","message":"Poor wording here - so currently it is possible to specify a pci_alias list and then tag the flavor with it. This creates a \"PciRequest\" for an instance that works as a filter for available pci devices on a host. Currently it is only able to filter on product and vendor string which is not enough to tell us if it\u0027s a PF or not, so we would need to expand the capabilities of the pci_alias schema.","commit_id":"ead05233a91c38684d0f0d1183c71b5372df4b65"},{"author":{"_account_id":8788,"name":"Miguel Angel Ajo","email":"mangelajo@redhat.com","username":"mangelajo"},"change_message_id":"d6727299c0096d53751140978ec87455a3a3f74b","unresolved":false,"context_lines":[{"line_number":194,"context_line":""},{"line_number":195,"context_line":"* Add support for specifying the PF requirement through the pci_alias"},{"line_number":196,"context_line":"  configuration options, so that it can be requested through flavor"},{"line_number":197,"context_line":"  extra-specs."},{"line_number":198,"context_line":""},{"line_number":199,"context_line":""},{"line_number":200,"context_line":"Dependencies"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_7daef82e","line":197,"updated":"2015-09-01 12:37:21.000000000","message":"* Integration with neutron, as the PF is passed down to the VM, we don\u0027t have control over changing the mac address anymore, that may need to be communicated back to neutron somehow, so the dhcp-agent or other agents are capable of updating their data and configuration.\n\n  In the other hand, using a whole PF requires the switch to do the tagging/untagging for tenant isolation, or \"L2 connection\" to any specific external network. That will require a long discussion in the neutron community to see how to handle that. Ideally we may need to leverage the capabilities of a full blown SDN controller, or, provide an small open flow controller capable of controlling and managing the ToR (OF capable switches).\n\n[1] goes a bit in that direction, by integrating with the ryu libraries.\n\n[1] https://review.openstack.org/#/c/153946/","commit_id":"ead05233a91c38684d0f0d1183c71b5372df4b65"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"adcc6fff48600ab935aaa4caa31f05917bcf7c18","unresolved":false,"context_lines":[{"line_number":194,"context_line":""},{"line_number":195,"context_line":"* Add support for specifying the PF requirement through the pci_alias"},{"line_number":196,"context_line":"  configuration options, so that it can be requested through flavor"},{"line_number":197,"context_line":"  extra-specs."},{"line_number":198,"context_line":""},{"line_number":199,"context_line":""},{"line_number":200,"context_line":"Dependencies"}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a1a9d01_2723eb57","line":197,"in_reply_to":"da20952f_587a22be","updated":"2015-09-29 15:37:37.000000000","message":"Agreed - this spec is only meant to cover intrnal work that needs to be done on the Nova side.","commit_id":"ead05233a91c38684d0f0d1183c71b5372df4b65"},{"author":{"_account_id":8802,"name":"Vladik Romanovsky","email":"vromanso@redhat.com","username":"vladikr"},"change_message_id":"31c9121b74796552685c6e0f56a88a91e663b23a","unresolved":false,"context_lines":[{"line_number":194,"context_line":""},{"line_number":195,"context_line":"* Add support for specifying the PF requirement through the pci_alias"},{"line_number":196,"context_line":"  configuration options, so that it can be requested through flavor"},{"line_number":197,"context_line":"  extra-specs."},{"line_number":198,"context_line":""},{"line_number":199,"context_line":""},{"line_number":200,"context_line":"Dependencies"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_587a22be","line":197,"in_reply_to":"da20952f_7daef82e","updated":"2015-09-01 12:55:32.000000000","message":"Thank you for the comment Miguel!\n\nI think, this spec only describes how PFs can be assigned to instances, as regular pci devices.\n\nThere will be another spec that will describe how PFs will be used as Neutron ports, and your comment will belong to it.\n\nI\u0027m currently writing the additional spec.","commit_id":"ead05233a91c38684d0f0d1183c71b5372df4b65"},{"author":{"_account_id":8788,"name":"Miguel Angel Ajo","email":"mangelajo@redhat.com","username":"mangelajo"},"change_message_id":"f317e9a4ccca837a102c506c8b7ddf6d80fba5dc","unresolved":false,"context_lines":[{"line_number":195,"context_line":"* Add support for specifying the PF requirement through the pci_alias"},{"line_number":196,"context_line":"  configuration options, so that it can be requested through flavor"},{"line_number":197,"context_line":"  extra-specs."},{"line_number":198,"context_line":""},{"line_number":199,"context_line":""},{"line_number":200,"context_line":"Dependencies"},{"line_number":201,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_2fe7a613","line":198,"updated":"2015-09-02 12:56:37.000000000","message":"* We need to define a new vnic-type (now we have direct, macvtap or normal from neutron) look for port-create here [1] which has to be understood by nova.\n\n\n[1] http://docs.openstack.org/cli-reference/content/neutronclient_commands.html","commit_id":"ead05233a91c38684d0f0d1183c71b5372df4b65"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"adcc6fff48600ab935aaa4caa31f05917bcf7c18","unresolved":false,"context_lines":[{"line_number":195,"context_line":"* Add support for specifying the PF requirement through the pci_alias"},{"line_number":196,"context_line":"  configuration options, so that it can be requested through flavor"},{"line_number":197,"context_line":"  extra-specs."},{"line_number":198,"context_line":""},{"line_number":199,"context_line":""},{"line_number":200,"context_line":"Dependencies"},{"line_number":201,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a1a9d01_b2adf285","line":198,"in_reply_to":"ba15a1d1_7a0baa2a","updated":"2015-09-29 15:37:37.000000000","message":"So it does seem that there will be a need for a new VNIC type so that we can make sure we are creating the apropriate PciRequest record in Nova that will then result in the instance being scheduled and claiming the PF, however this work really does seem to belong in a different BP which will deal with proper integration between Nova and Neutron when it comes to PFs.\n\nWe\u0027d like to keep the scope of this particular spec limited to the work that needs to be done in Nova only, so I\u0027d keep the new vnic_type out of it.","commit_id":"ead05233a91c38684d0f0d1183c71b5372df4b65"},{"author":{"_account_id":12171,"name":"Moshe Levi","email":"moshele@nvidia.com","username":"moshele"},"change_message_id":"0435a85f98b4264f9e621d21ed587a9088be8d4d","unresolved":false,"context_lines":[{"line_number":195,"context_line":"* Add support for specifying the PF requirement through the pci_alias"},{"line_number":196,"context_line":"  configuration options, so that it can be requested through flavor"},{"line_number":197,"context_line":"  extra-specs."},{"line_number":198,"context_line":""},{"line_number":199,"context_line":""},{"line_number":200,"context_line":"Dependencies"},{"line_number":201,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":1,"id":"ba15a1d1_7a0baa2a","line":198,"in_reply_to":"da20952f_2fe7a613","updated":"2015-09-17 08:26:07.000000000","message":"I agree there should be a new vinc_type and accroding to the new type or direct type nova should now if it need to provide the pci of the port or the pci of the VF\n\nalso I think when you add new  vnic_type you will need to do detach/attached SR_IOV port on suspend /resume/snapshot same like the direct port","commit_id":"ead05233a91c38684d0f0d1183c71b5372df4b65"},{"author":{"_account_id":3217,"name":"Ian Wells","username":"ijw-ubuntu"},"change_message_id":"fc847b283e84879da75856e7215cdc6eb0765215","unresolved":false,"context_lines":[{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/nova/+spec/sriov-physical-function-passthrough"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"Nova has supported passthrough of PCI devices with it\u0027s libvirt driver for a"},{"line_number":14,"context_line":"few releases already, during which time the code has seen some stabilization"},{"line_number":15,"context_line":"and a few minor feature additions."},{"line_number":16,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a2fa921_9693a1a3","line":13,"updated":"2015-10-05 18:02:29.000000000","message":"its not it\u0027s","commit_id":"65d9273eebce0e3953ea614905623d83b19dd005"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"e601b6487658666f53d9558cc5c6ede0bb5678a3","unresolved":false,"context_lines":[{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/nova/+spec/sriov-physical-function-passthrough"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"Nova has supported passthrough of PCI devices with it\u0027s libvirt driver for a"},{"line_number":14,"context_line":"few releases already, during which time the code has seen some stabilization"},{"line_number":15,"context_line":"and a few minor feature additions."},{"line_number":16,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a2fa921_0913a4b9","line":13,"in_reply_to":"7a2fa921_9693a1a3","updated":"2015-10-07 11:30:07.000000000","message":"Done","commit_id":"65d9273eebce0e3953ea614905623d83b19dd005"},{"author":{"_account_id":3217,"name":"Ian Wells","username":"ijw-ubuntu"},"change_message_id":"086769dcc96c47ec5a3f33c6c2e6de4edd367f95","unresolved":false,"context_lines":[{"line_number":16,"context_line":""},{"line_number":17,"context_line":"In the case of SR-IOV enabled cards, it is possible to treat any port on the"},{"line_number":18,"context_line":"card either as a number of virtual devices (called VFs - virtual functions) or"},{"line_number":19,"context_line":"as a full device (PF - physical function)."},{"line_number":20,"context_line":""},{"line_number":21,"context_line":"Nova\u0027s current handling exposes only virtual functions as resources that can"},{"line_number":22,"context_line":"be requested by instances - and this is the most common use case by far."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9a1a9d01_23c2ace9","line":19,"updated":"2015-10-01 04:31:06.000000000","message":"I assume you\u0027re actually thinking to pass through the PF *and all its VFs* to the guest, but I\u0027m not seeing that written down.","commit_id":"65d9273eebce0e3953ea614905623d83b19dd005"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"e601b6487658666f53d9558cc5c6ede0bb5678a3","unresolved":false,"context_lines":[{"line_number":16,"context_line":""},{"line_number":17,"context_line":"In the case of SR-IOV enabled cards, it is possible to treat any port on the"},{"line_number":18,"context_line":"card either as a number of virtual devices (called VFs - virtual functions) or"},{"line_number":19,"context_line":"as a full device (PF - physical function)."},{"line_number":20,"context_line":""},{"line_number":21,"context_line":"Nova\u0027s current handling exposes only virtual functions as resources that can"},{"line_number":22,"context_line":"be requested by instances - and this is the most common use case by far."}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a2fa921_58b5e42e","line":19,"in_reply_to":"7a2fa921_02d3c739","updated":"2015-10-07 11:30:07.000000000","message":"So nova will track usage of PFs and know the relationship between any VFs and mark them unavailable so the Nova scheduler/compute claims logic would not even attempt to do this if we get this. It would be true the other way around as well. If even a single VF is assigned to an instance, Nova will treat the PF as \"taken\" too and not even consider it when scheduling.\n\nIt is my understanding (as Vladik points out above) - once a PF is assigned to the VM the ixgbe driver is unloaded and thus VFs can\u0027t really be used anymore at all. I am unsure if that is the case with other vendors, but it is likely to be the case due to VFIO infrastructure.\n\nIf you read on in the Use-cases section - we specifically address the case when deployers are unsure of the behaviour of their cards, and need to pre-provision them.\n\nI am unsure what PF+VFs means exactly but from Nova perspective - this is what will happen - once a PF is assigned, a PF and all of it\u0027s VFs that are whitelisted and thus tracked by Nova will be marked as in use and made unavailable. I will make sure this is stated more clearly.","commit_id":"65d9273eebce0e3953ea614905623d83b19dd005"},{"author":{"_account_id":3217,"name":"Ian Wells","username":"ijw-ubuntu"},"change_message_id":"fc847b283e84879da75856e7215cdc6eb0765215","unresolved":false,"context_lines":[{"line_number":16,"context_line":""},{"line_number":17,"context_line":"In the case of SR-IOV enabled cards, it is possible to treat any port on the"},{"line_number":18,"context_line":"card either as a number of virtual devices (called VFs - virtual functions) or"},{"line_number":19,"context_line":"as a full device (PF - physical function)."},{"line_number":20,"context_line":""},{"line_number":21,"context_line":"Nova\u0027s current handling exposes only virtual functions as resources that can"},{"line_number":22,"context_line":"be requested by instances - and this is the most common use case by far."}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a2fa921_f6cf75b1","line":19,"in_reply_to":"7a2fa921_9c0e6de4","updated":"2015-10-05 18:02:29.000000000","message":"It\u0027s actively dangerous to passthrough a PF and leave its VFs behind.  It\u0027s not what this says, and rather pointless, to pass through VFs and leave the PF behind and uncontrolled.  So PF+VFs is really the only thing that makes sense.","commit_id":"65d9273eebce0e3953ea614905623d83b19dd005"},{"author":{"_account_id":8802,"name":"Vladik Romanovsky","email":"vromanso@redhat.com","username":"vladikr"},"change_message_id":"38d70300059fc1dacc04ea1b4dbd3630080978c9","unresolved":false,"context_lines":[{"line_number":16,"context_line":""},{"line_number":17,"context_line":"In the case of SR-IOV enabled cards, it is possible to treat any port on the"},{"line_number":18,"context_line":"card either as a number of virtual devices (called VFs - virtual functions) or"},{"line_number":19,"context_line":"as a full device (PF - physical function)."},{"line_number":20,"context_line":""},{"line_number":21,"context_line":"Nova\u0027s current handling exposes only virtual functions as resources that can"},{"line_number":22,"context_line":"be requested by instances - and this is the most common use case by far."}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a2fa921_02d3c739","line":19,"in_reply_to":"7a2fa921_f6cf75b1","updated":"2015-10-05 20:13:06.000000000","message":"Unfortunately, I didn\u0027t understand your comment.\n\nThe idea here it to be able to dynamically pass either a Virtual Function to an instance or a Physical function.\nA physical function may have configured virtual functions (VFs), in case non of the configured VF from a given PF are assigned to instances, we should be able to safely assign the FP to an instance; making the VFs unavailable. [1]\n\nIf we will attempt to assign a PF, when one (or more) of it\u0027s corresponding VFs is assigned to an instance, the unbound operation will block until the VF is released. [2]\n\n[1] For example as described in https://lkml.org/lkml/2015/7/10/796\n[2] For example, here: https://lkml.org/lkml/2014/8/20/484","commit_id":"65d9273eebce0e3953ea614905623d83b19dd005"},{"author":{"_account_id":8788,"name":"Miguel Angel Ajo","email":"mangelajo@redhat.com","username":"mangelajo"},"change_message_id":"50b0ed22a3ac01a46e4befc893b0d3f225723ab5","unresolved":false,"context_lines":[{"line_number":16,"context_line":""},{"line_number":17,"context_line":"In the case of SR-IOV enabled cards, it is possible to treat any port on the"},{"line_number":18,"context_line":"card either as a number of virtual devices (called VFs - virtual functions) or"},{"line_number":19,"context_line":"as a full device (PF - physical function)."},{"line_number":20,"context_line":""},{"line_number":21,"context_line":"Nova\u0027s current handling exposes only virtual functions as resources that can"},{"line_number":22,"context_line":"be requested by instances - and this is the most common use case by far."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9a1a9d01_d716fc57","line":19,"in_reply_to":"9a1a9d01_23c2ace9","updated":"2015-10-01 12:09:50.000000000","message":"Good point, and correct. As far as I understood, the idea is to map the whole PF address space (VFs -which are sub-spaces- included) into the guest.","commit_id":"65d9273eebce0e3953ea614905623d83b19dd005"},{"author":{"_account_id":8802,"name":"Vladik Romanovsky","email":"vromanso@redhat.com","username":"vladikr"},"change_message_id":"a1e39eea4662348f86b3b4f9e86d6b9300af0c8d","unresolved":false,"context_lines":[{"line_number":16,"context_line":""},{"line_number":17,"context_line":"In the case of SR-IOV enabled cards, it is possible to treat any port on the"},{"line_number":18,"context_line":"card either as a number of virtual devices (called VFs - virtual functions) or"},{"line_number":19,"context_line":"as a full device (PF - physical function)."},{"line_number":20,"context_line":""},{"line_number":21,"context_line":"Nova\u0027s current handling exposes only virtual functions as resources that can"},{"line_number":22,"context_line":"be requested by instances - and this is the most common use case by far."}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a2fa921_9c0e6de4","line":19,"in_reply_to":"9a1a9d01_c08634db","updated":"2015-10-05 14:41:24.000000000","message":"Yea, I agree with Robert. Maybe we do need a line that says that only those PFs which don\u0027t have it\u0027s VFs in use by any instance, will be available for assignment.\n\nInitially I thought that the above sentence is pretty clear, that we intend only to pass a single function, PF or VF, but now it looks like it\u0027ll be better to mention that when we are assigning a PF to a guest, it\u0027s corresponding VFs will not magically appear in the guest, as it\u0027s driver is being reset during the assignment.","commit_id":"65d9273eebce0e3953ea614905623d83b19dd005"},{"author":{"_account_id":6685,"name":"Baodong (Robert) Li","email":"baoli@cisco.com","username":"baoli"},"change_message_id":"b14ebb0710fe7a8a076e77aa1735c13b87337c15","unresolved":false,"context_lines":[{"line_number":16,"context_line":""},{"line_number":17,"context_line":"In the case of SR-IOV enabled cards, it is possible to treat any port on the"},{"line_number":18,"context_line":"card either as a number of virtual devices (called VFs - virtual functions) or"},{"line_number":19,"context_line":"as a full device (PF - physical function)."},{"line_number":20,"context_line":""},{"line_number":21,"context_line":"Nova\u0027s current handling exposes only virtual functions as resources that can"},{"line_number":22,"context_line":"be requested by instances - and this is the most common use case by far."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9a1a9d01_c08634db","line":19,"in_reply_to":"9a1a9d01_d716fc57","updated":"2015-10-01 13:19:05.000000000","message":"My understanding from reading the spec is that once the PF is passed to the VM, all of the VFs under the PF will not be available any more for other use. On the other hand, if a VF is already passed to a PM, the corresponding PF should not be available for passthrough. So this is something that need to be clarified.","commit_id":"65d9273eebce0e3953ea614905623d83b19dd005"},{"author":{"_account_id":6685,"name":"Baodong (Robert) Li","email":"baoli@cisco.com","username":"baoli"},"change_message_id":"b14ebb0710fe7a8a076e77aa1735c13b87337c15","unresolved":false,"context_lines":[{"line_number":58,"context_line":"is the need to set the port into promiscuous mode, which may be a requirement"},{"line_number":59,"context_line":"for certain types of VNFs) or to exclusively use the full bandwidth of the"},{"line_number":60,"context_line":"port. However, due to the dynamic nature of the elastic cloud, and the promise"},{"line_number":61,"context_line":"of Nova to we do not wish to have to pre-provision certain SR-IOV cards to be"},{"line_number":62,"context_line":"used as PFs as this defeats the promise of the infrastructure management tool"},{"line_number":63,"context_line":"that allows for quick re-purposing of resources that Nova brings."},{"line_number":64,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"9a1a9d01_204db0c4","line":61,"updated":"2015-10-01 13:19:05.000000000","message":"typo here.","commit_id":"65d9273eebce0e3953ea614905623d83b19dd005"},{"author":{"_account_id":6685,"name":"Baodong (Robert) Li","email":"baoli@cisco.com","username":"baoli"},"change_message_id":"b14ebb0710fe7a8a076e77aa1735c13b87337c15","unresolved":false,"context_lines":[{"line_number":60,"context_line":"port. However, due to the dynamic nature of the elastic cloud, and the promise"},{"line_number":61,"context_line":"of Nova to we do not wish to have to pre-provision certain SR-IOV cards to be"},{"line_number":62,"context_line":"used as PFs as this defeats the promise of the infrastructure management tool"},{"line_number":63,"context_line":"that allows for quick re-purposing of resources that Nova brings."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"Modern SR-IOV enabled cards along with their drivers usually allow for such"},{"line_number":66,"context_line":"reconfiguration to be done on the fly, so once the passthrough of the PF is no"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9a1a9d01_00c0acdf","line":63,"updated":"2015-10-01 13:19:05.000000000","message":"Do you mean to say that PFs and VFs have already been configured on the card, that whether or not it\u0027s a whole PF passthrough or individual VF passthrough is determined at runtime?","commit_id":"65d9273eebce0e3953ea614905623d83b19dd005"},{"author":{"_account_id":3217,"name":"Ian Wells","username":"ijw-ubuntu"},"change_message_id":"fc847b283e84879da75856e7215cdc6eb0765215","unresolved":false,"context_lines":[{"line_number":60,"context_line":"port. However, due to the dynamic nature of the elastic cloud, and the promise"},{"line_number":61,"context_line":"of Nova to we do not wish to have to pre-provision certain SR-IOV cards to be"},{"line_number":62,"context_line":"used as PFs as this defeats the promise of the infrastructure management tool"},{"line_number":63,"context_line":"that allows for quick re-purposing of resources that Nova brings."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"Modern SR-IOV enabled cards along with their drivers usually allow for such"},{"line_number":66,"context_line":"reconfiguration to be done on the fly, so once the passthrough of the PF is no"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a2fa921_f63695b3","line":63,"in_reply_to":"7a2fa921_3973ab4f","updated":"2015-10-05 18:02:29.000000000","message":"I wonder if the kernel in the guest will reset and reconfigure the VF count on boot anyway.  I also wonder if the passthrough will hold if it does.  This may be a case of trying it to find out.","commit_id":"65d9273eebce0e3953ea614905623d83b19dd005"},{"author":{"_account_id":3217,"name":"Ian Wells","username":"ijw-ubuntu"},"change_message_id":"7402634d094ac20756df1f38b4b12a4831a7e018","unresolved":false,"context_lines":[{"line_number":60,"context_line":"port. However, due to the dynamic nature of the elastic cloud, and the promise"},{"line_number":61,"context_line":"of Nova to we do not wish to have to pre-provision certain SR-IOV cards to be"},{"line_number":62,"context_line":"used as PFs as this defeats the promise of the infrastructure management tool"},{"line_number":63,"context_line":"that allows for quick re-purposing of resources that Nova brings."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"Modern SR-IOV enabled cards along with their drivers usually allow for such"},{"line_number":66,"context_line":"reconfiguration to be done on the fly, so once the passthrough of the PF is no"}],"source_content_type":"text/x-rst","patch_set":2,"id":"3a29b11f_b075a1cb","line":63,"in_reply_to":"7a2fa921_989ffc4f","updated":"2015-10-23 00:43:02.000000000","message":"(Apologies for doing this on the old version)\n\nMy concern here is: I have a PF with 0 VFs as configured by the host kernel.  I pass it though to a guest kernel, with its (0) VFs.  The guest kernel reconfigures it to have 64 VFs, which are shiny new devices on the PCI bus with zero obvious relationship to the PF and are therefore not automatically passed through by the existing passthrough binding, but appear in the host kernel.  The guest kernel then reprograms them to forward malicious traffic to the host (ipv6 link local addressing is one particularly useful way to ensure it arrives, if we\u0027re being inventive).\n\nThis situation is possible in pretty much any situation where the host kernel even briefly sees these interfaces before Openstack removes them.  The only case that would work is that the devices, on creation, are always and without exception bound to the vfio driver and not the ixgbe (or whatever) driver, which would be hard to achieve; and technically the expectation of the guest should be that the passthrough is immediate on changing the PF, not 5 seconds later when the periodic orchestration sees their arrival...","commit_id":"65d9273eebce0e3953ea614905623d83b19dd005"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"e601b6487658666f53d9558cc5c6ede0bb5678a3","unresolved":false,"context_lines":[{"line_number":60,"context_line":"port. However, due to the dynamic nature of the elastic cloud, and the promise"},{"line_number":61,"context_line":"of Nova to we do not wish to have to pre-provision certain SR-IOV cards to be"},{"line_number":62,"context_line":"used as PFs as this defeats the promise of the infrastructure management tool"},{"line_number":63,"context_line":"that allows for quick re-purposing of resources that Nova brings."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"Modern SR-IOV enabled cards along with their drivers usually allow for such"},{"line_number":66,"context_line":"reconfiguration to be done on the fly, so once the passthrough of the PF is no"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a2fa921_989ffc4f","line":63,"in_reply_to":"7a2fa921_f63695b3","updated":"2015-10-07 11:30:07.000000000","message":"I think that once the PF is passed through, any configuration is up to the guest. Upon guest releasing the PF - it is up to the driver what happens with the VFs. If they are not re-configured, then we can\u0027t really mix PFs and VFs on the same port, and the admin will have to reserve the port for either PF or VF usage, and whitelist in Nova accordingly.\n\nThe code in Nova will provide for both.","commit_id":"65d9273eebce0e3953ea614905623d83b19dd005"},{"author":{"_account_id":8802,"name":"Vladik Romanovsky","email":"vromanso@redhat.com","username":"vladikr"},"change_message_id":"a1e39eea4662348f86b3b4f9e86d6b9300af0c8d","unresolved":false,"context_lines":[{"line_number":60,"context_line":"port. However, due to the dynamic nature of the elastic cloud, and the promise"},{"line_number":61,"context_line":"of Nova to we do not wish to have to pre-provision certain SR-IOV cards to be"},{"line_number":62,"context_line":"used as PFs as this defeats the promise of the infrastructure management tool"},{"line_number":63,"context_line":"that allows for quick re-purposing of resources that Nova brings."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"Modern SR-IOV enabled cards along with their drivers usually allow for such"},{"line_number":66,"context_line":"reconfiguration to be done on the fly, so once the passthrough of the PF is no"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a2fa921_3973ab4f","line":63,"in_reply_to":"9a1a9d01_00c0acdf","updated":"2015-10-05 14:41:24.000000000","message":"I think the main message here is that we are not planning to reconfigure the sriov card driver on the host, before the attempt to assign a PF to a guest.\nIf a PF driver has X configured VFs, which are not currently in use by any instance, we will attempt to assign it, while it\u0027s driver will not be reconfigured to 0  VFs, before the assignment. (max_vfs\u003d0)","commit_id":"65d9273eebce0e3953ea614905623d83b19dd005"},{"author":{"_account_id":6685,"name":"Baodong (Robert) Li","email":"baoli@cisco.com","username":"baoli"},"change_message_id":"b14ebb0710fe7a8a076e77aa1735c13b87337c15","unresolved":false,"context_lines":[{"line_number":82,"context_line":"Two problems we need to solve are:"},{"line_number":83,"context_line":""},{"line_number":84,"context_line":" 1) How to enable requesting a full physical device. This means extending the"},{"line_number":85,"context_line":"    InstancePCIRequest data model to be able to hold this information."},{"line_number":86,"context_line":""},{"line_number":87,"context_line":" 2) Enable scheduling and resource tracking based on the request that can now"},{"line_number":88,"context_line":"    be for the whole device. This means extending the data model for PCIDevices"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9a1a9d01_6bc3898b","line":85,"updated":"2015-10-01 13:19:05.000000000","message":"One question I have is that for PF passthrough, it seems to me that you can only use libvirt hostdev XML, rather than the interface XML. Currently we use a neutron direct port to indicate SR-IOV. And we require the admin to tag VFs with their connected physical network.  I\u0027d like to see for PF passthrough, how it\u0027s done from the user point of view.","commit_id":"65d9273eebce0e3953ea614905623d83b19dd005"},{"author":{"_account_id":3217,"name":"Ian Wells","username":"ijw-ubuntu"},"change_message_id":"fc847b283e84879da75856e7215cdc6eb0765215","unresolved":false,"context_lines":[{"line_number":82,"context_line":"Two problems we need to solve are:"},{"line_number":83,"context_line":""},{"line_number":84,"context_line":" 1) How to enable requesting a full physical device. This means extending the"},{"line_number":85,"context_line":"    InstancePCIRequest data model to be able to hold this information."},{"line_number":86,"context_line":""},{"line_number":87,"context_line":" 2) Enable scheduling and resource tracking based on the request that can now"},{"line_number":88,"context_line":"    be for the whole device. This means extending the data model for PCIDevices"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a2fa921_b6257d5d","line":85,"in_reply_to":"7a2fa921_9982ffc7","updated":"2015-10-05 18:02:29.000000000","message":"This will clearly require a different binding mechanism to that of SRIOV.  SRIOV binding is saying to Nova \u0027if you put this VF on this VLAN it will find the network\u0027; here we\u0027re saying \u0027Assuming you used the PCI interface you told me you did, it will attach to the network with no further work on your part\u0027.","commit_id":"65d9273eebce0e3953ea614905623d83b19dd005"},{"author":{"_account_id":8802,"name":"Vladik Romanovsky","email":"vromanso@redhat.com","username":"vladikr"},"change_message_id":"38d70300059fc1dacc04ea1b4dbd3630080978c9","unresolved":false,"context_lines":[{"line_number":82,"context_line":"Two problems we need to solve are:"},{"line_number":83,"context_line":""},{"line_number":84,"context_line":" 1) How to enable requesting a full physical device. This means extending the"},{"line_number":85,"context_line":"    InstancePCIRequest data model to be able to hold this information."},{"line_number":86,"context_line":""},{"line_number":87,"context_line":" 2) Enable scheduling and resource tracking based on the request that can now"},{"line_number":88,"context_line":"    be for the whole device. This means extending the data model for PCIDevices"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a2fa921_e2b48bd2","line":85,"in_reply_to":"7a2fa921_b6257d5d","updated":"2015-10-05 20:13:06.000000000","message":"At the moment we are not proposing any work which is related to VLANs.\nThe later work that I\u0027ve mentioned in the previous comment will only update neutron port with the selected PF mac address.\n\nI\u0027m sure Miguel will be able to elaborate about the VLANs, however, this is not related to this spec.","commit_id":"65d9273eebce0e3953ea614905623d83b19dd005"},{"author":{"_account_id":8802,"name":"Vladik Romanovsky","email":"vromanso@redhat.com","username":"vladikr"},"change_message_id":"a1e39eea4662348f86b3b4f9e86d6b9300af0c8d","unresolved":false,"context_lines":[{"line_number":82,"context_line":"Two problems we need to solve are:"},{"line_number":83,"context_line":""},{"line_number":84,"context_line":" 1) How to enable requesting a full physical device. This means extending the"},{"line_number":85,"context_line":"    InstancePCIRequest data model to be able to hold this information."},{"line_number":86,"context_line":""},{"line_number":87,"context_line":" 2) Enable scheduling and resource tracking based on the request that can now"},{"line_number":88,"context_line":"    be for the whole device. This means extending the data model for PCIDevices"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a2fa921_9982ffc7","line":85,"in_reply_to":"9a1a9d01_6bc3898b","updated":"2015-10-05 14:41:24.000000000","message":"This spec only describes how PFs will be exposed for passthrough, without the relation to neutron.\nA following blueprint or spec will address the relation to neutron, by introducing a new vif_type with will eventually use the same mechanism, that is described in this spec and will update neutron port with the mac address of the selected PF.","commit_id":"65d9273eebce0e3953ea614905623d83b19dd005"},{"author":{"_account_id":8802,"name":"Vladik Romanovsky","email":"vromanso@redhat.com","username":"vladikr"},"change_message_id":"a1e39eea4662348f86b3b4f9e86d6b9300af0c8d","unresolved":false,"context_lines":[{"line_number":92,"context_line":"    way we expose the aggregate data about PCI devices to the resource tracker"},{"line_number":93,"context_line":"    (a.k.a. the PCIDeviceStats class) to be able to present PFs and their"},{"line_number":94,"context_line":"    counts, and to make sure to track the corresponding VFs that become"},{"line_number":95,"context_line":"    unavailable once the PF is claimed/used."},{"line_number":96,"context_line":""},{"line_number":97,"context_line":"In addition to the above, it is important to note that the whitelist syntax"},{"line_number":98,"context_line":"of the pci_passthrough_whitelist configuration option support for whitelisting"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a2fa921_b93fbbd5","line":95,"updated":"2015-10-05 14:41:24.000000000","message":"Maybe we should mention separately, that only those PF which don\u0027t have it\u0027s corresponding VFs in use by any instance, will be available for assignment?","commit_id":"65d9273eebce0e3953ea614905623d83b19dd005"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"e601b6487658666f53d9558cc5c6ede0bb5678a3","unresolved":false,"context_lines":[{"line_number":92,"context_line":"    way we expose the aggregate data about PCI devices to the resource tracker"},{"line_number":93,"context_line":"    (a.k.a. the PCIDeviceStats class) to be able to present PFs and their"},{"line_number":94,"context_line":"    counts, and to make sure to track the corresponding VFs that become"},{"line_number":95,"context_line":"    unavailable once the PF is claimed/used."},{"line_number":96,"context_line":""},{"line_number":97,"context_line":"In addition to the above, it is important to note that the whitelist syntax"},{"line_number":98,"context_line":"of the pci_passthrough_whitelist configuration option support for whitelisting"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a2fa921_78d22002","line":95,"in_reply_to":"7a2fa921_828917cd","updated":"2015-10-07 11:30:07.000000000","message":"As already described above, passing throug a PF means all of it\u0027s VFs become unavailable, and also if any single VF is passed - PF is seen as \"used\" by Nova too.\n\nI will make sure I clarify this in the text.","commit_id":"65d9273eebce0e3953ea614905623d83b19dd005"},{"author":{"_account_id":3217,"name":"Ian Wells","username":"ijw-ubuntu"},"change_message_id":"fc847b283e84879da75856e7215cdc6eb0765215","unresolved":false,"context_lines":[{"line_number":92,"context_line":"    way we expose the aggregate data about PCI devices to the resource tracker"},{"line_number":93,"context_line":"    (a.k.a. the PCIDeviceStats class) to be able to present PFs and their"},{"line_number":94,"context_line":"    counts, and to make sure to track the corresponding VFs that become"},{"line_number":95,"context_line":"    unavailable once the PF is claimed/used."},{"line_number":96,"context_line":""},{"line_number":97,"context_line":"In addition to the above, it is important to note that the whitelist syntax"},{"line_number":98,"context_line":"of the pci_passthrough_whitelist configuration option support for whitelisting"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a2fa921_d6e11900","line":95,"in_reply_to":"7a2fa921_b93fbbd5","updated":"2015-10-05 18:02:29.000000000","message":"... or anything else at all.  You get all the VFs or none, you can\u0027t safely pass a PF with only some VFs, so they have to all be whitelisted.  (If you leave one attached to the host kernel you could play really dangerous games as a guest.)","commit_id":"65d9273eebce0e3953ea614905623d83b19dd005"},{"author":{"_account_id":8802,"name":"Vladik Romanovsky","email":"vromanso@redhat.com","username":"vladikr"},"change_message_id":"38d70300059fc1dacc04ea1b4dbd3630080978c9","unresolved":false,"context_lines":[{"line_number":92,"context_line":"    way we expose the aggregate data about PCI devices to the resource tracker"},{"line_number":93,"context_line":"    (a.k.a. the PCIDeviceStats class) to be able to present PFs and their"},{"line_number":94,"context_line":"    counts, and to make sure to track the corresponding VFs that become"},{"line_number":95,"context_line":"    unavailable once the PF is claimed/used."},{"line_number":96,"context_line":""},{"line_number":97,"context_line":"In addition to the above, it is important to note that the whitelist syntax"},{"line_number":98,"context_line":"of the pci_passthrough_whitelist configuration option support for whitelisting"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a2fa921_828917cd","line":95,"in_reply_to":"7a2fa921_d6e11900","updated":"2015-10-05 20:13:06.000000000","message":"The reason this spec has been proposed is because there is a lot of demand to support this functionality (PF passthrough) - there is also a technical way to do so.\nBy mapping the relationship between PFs and it\u0027s VFs we will only pass through PFs which doesn\u0027t have assigned VFs to instances.","commit_id":"65d9273eebce0e3953ea614905623d83b19dd005"},{"author":{"_account_id":6685,"name":"Baodong (Robert) Li","email":"baoli@cisco.com","username":"baoli"},"change_message_id":"019a0181be68c62974931b813647149fe71f8c1a","unresolved":false,"context_lines":[{"line_number":41,"context_line":"to manage and goes against the goal of Nova to provide on demand"},{"line_number":42,"context_line":"resources. We want to be able to give certain instances unlimited access to the"},{"line_number":43,"context_line":"port by assigning the PF to it, but revert back to using VFs when the PF is not"},{"line_number":44,"context_line":"being used, so as to ensure on-demand provisioning of available resources. This"},{"line_number":45,"context_line":"may not be possible with every SR-IOV card and their respective Linux drivers,"},{"line_number":46,"context_line":"in which case certain ports will need to be pre-provisioned as either PFs or"},{"line_number":47,"context_line":"VFs by administratior ahead of time."}],"source_content_type":"text/x-rst","patch_set":3,"id":"7a2fa921_53534239","line":44,"updated":"2015-10-08 15:21:48.000000000","message":"depending on the scheduler\u0027s behavior, would a PF passthrough request never be fulfilled? Also, I know some of the SRIOV cards can support multiple PFs per physical interface. In that case, PF passthrough doesn\u0027t guarantee the whole interface is claimed by the VM.","commit_id":"7dcb97cc79ff4ce57ec5ecd64e907803e4ddabe3"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"0bca01bc22d42ef8e07e8ae44093573ba4a2dc64","unresolved":false,"context_lines":[{"line_number":41,"context_line":"to manage and goes against the goal of Nova to provide on demand"},{"line_number":42,"context_line":"resources. We want to be able to give certain instances unlimited access to the"},{"line_number":43,"context_line":"port by assigning the PF to it, but revert back to using VFs when the PF is not"},{"line_number":44,"context_line":"being used, so as to ensure on-demand provisioning of available resources. This"},{"line_number":45,"context_line":"may not be possible with every SR-IOV card and their respective Linux drivers,"},{"line_number":46,"context_line":"in which case certain ports will need to be pre-provisioned as either PFs or"},{"line_number":47,"context_line":"VFs by administratior ahead of time."}],"source_content_type":"text/x-rst","patch_set":3,"id":"7a2fa921_e4329ba6","line":44,"in_reply_to":"7a2fa921_53534239","updated":"2015-10-09 10:03:36.000000000","message":"Would you care to provide for some more details on the type of cards you are referring to so I can look into it a bit more just in case.\n\nThe current answer would be: If that is the case, the administrator would have to be aware of that and whitelist devices accordingly.","commit_id":"7dcb97cc79ff4ce57ec5ecd64e907803e4ddabe3"},{"author":{"_account_id":6685,"name":"Baodong (Robert) Li","email":"baoli@cisco.com","username":"baoli"},"change_message_id":"20128326bf747a12f49a85e991c322b1dab10cf9","unresolved":false,"context_lines":[{"line_number":41,"context_line":"to manage and goes against the goal of Nova to provide on demand"},{"line_number":42,"context_line":"resources. We want to be able to give certain instances unlimited access to the"},{"line_number":43,"context_line":"port by assigning the PF to it, but revert back to using VFs when the PF is not"},{"line_number":44,"context_line":"being used, so as to ensure on-demand provisioning of available resources. This"},{"line_number":45,"context_line":"may not be possible with every SR-IOV card and their respective Linux drivers,"},{"line_number":46,"context_line":"in which case certain ports will need to be pre-provisioned as either PFs or"},{"line_number":47,"context_line":"VFs by administratior ahead of time."}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a2ca52d_acce8581","line":44,"in_reply_to":"7a2fa921_e4329ba6","updated":"2015-10-13 15:06:52.000000000","message":"Cisco\u0027s VIC adaptors support static and dynamic vNICs, which corresponds to PFs and VFs respectively. On the physical port, a number of static vNICs can be created and multiple dynamic vNICs can be created on each of the static vNICs. Unfortunately, I can\u0027t find an internet link that provides a simplified description on this. But there are plenty of information about cisco\u0027s UCS manager and VIC adaptors on the internet.","commit_id":"7dcb97cc79ff4ce57ec5ecd64e907803e4ddabe3"},{"author":{"_account_id":6685,"name":"Baodong (Robert) Li","email":"baoli@cisco.com","username":"baoli"},"change_message_id":"019a0181be68c62974931b813647149fe71f8c1a","unresolved":false,"context_lines":[{"line_number":91,"context_line":"Two problems we need to solve are:"},{"line_number":92,"context_line":""},{"line_number":93,"context_line":" 1) How to enable requesting a full physical device. This means extending the"},{"line_number":94,"context_line":"    InstancePCIRequest data model to be able to hold this information."},{"line_number":95,"context_line":""},{"line_number":96,"context_line":" 2) Enable scheduling and resource tracking based on the request that can now"},{"line_number":97,"context_line":"    be for the whole device. This means extending the data model for PCIDevices"}],"source_content_type":"text/x-rst","patch_set":3,"id":"7a2fa921_decc39bd","line":94,"updated":"2015-10-08 15:21:48.000000000","message":"One of my comments in the previous patch set is that it seems to me that for PF passthrough, you can only use libvirt hostdev XML, rather than the interface XML. If that\u0027s true, I think that the spec should at least mention this, and ideally specify how to deal with it.","commit_id":"7dcb97cc79ff4ce57ec5ecd64e907803e4ddabe3"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"0bca01bc22d42ef8e07e8ae44093573ba4a2dc64","unresolved":false,"context_lines":[{"line_number":91,"context_line":"Two problems we need to solve are:"},{"line_number":92,"context_line":""},{"line_number":93,"context_line":" 1) How to enable requesting a full physical device. This means extending the"},{"line_number":94,"context_line":"    InstancePCIRequest data model to be able to hold this information."},{"line_number":95,"context_line":""},{"line_number":96,"context_line":" 2) Enable scheduling and resource tracking based on the request that can now"},{"line_number":97,"context_line":"    be for the whole device. This means extending the data model for PCIDevices"}],"source_content_type":"text/x-rst","patch_set":3,"id":"7a2fa921_647beb28","line":94,"in_reply_to":"7a2fa921_decc39bd","updated":"2015-10-09 10:03:36.000000000","message":"Fair enough  - it is mentioned in the last paragraph of this section briefly, but I will make sure to mention it in more detail so that it\u0027s clear.","commit_id":"7dcb97cc79ff4ce57ec5ecd64e907803e4ddabe3"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"42d7e7347cf800d79bb1c1931d453cd3b8340df0","unresolved":false,"context_lines":[{"line_number":55,"context_line":"that is a physical function (this is currently hard-coded in the driver). In"},{"line_number":56,"context_line":"addition, modeling of PCI device resources in Nova currently assumes flat"},{"line_number":57,"context_line":"hierarchy and resource tracking logic does not understand the relationship"},{"line_number":58,"context_line":"between different PCI devices that can be exposed to Nova."},{"line_number":59,"context_line":""},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"Use Cases"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a2fa921_59574aca","line":58,"updated":"2015-10-13 09:18:43.000000000","message":"+1 to this description.","commit_id":"d90861578b01f294c5e1cfe2526fc08cd913fc5b"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"42d7e7347cf800d79bb1c1931d453cd3b8340df0","unresolved":false,"context_lines":[{"line_number":78,"context_line":"guarantee that this always works however, due to the vast range of equipment"},{"line_number":79,"context_line":"and drivers available on the market, so we want to make sure that there is a"},{"line_number":80,"context_line":"way to tell Nova that a card is in certain configuration and cannot be assumed"},{"line_number":81,"context_line":"to be reconfigurable."},{"line_number":82,"context_line":""},{"line_number":83,"context_line":"Project Priority"},{"line_number":84,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a2fa921_b9d77e22","line":81,"updated":"2015-10-13 09:18:43.000000000","message":"This is great context.\n\nI do, however, wonder if we want more end user specific use cases adding here.","commit_id":"d90861578b01f294c5e1cfe2526fc08cd913fc5b"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"57744fbbe0848b4dcdbf3ac0ca7a9de95236cb7b","unresolved":false,"context_lines":[{"line_number":78,"context_line":"guarantee that this always works however, due to the vast range of equipment"},{"line_number":79,"context_line":"and drivers available on the market, so we want to make sure that there is a"},{"line_number":80,"context_line":"way to tell Nova that a card is in certain configuration and cannot be assumed"},{"line_number":81,"context_line":"to be reconfigurable."},{"line_number":82,"context_line":""},{"line_number":83,"context_line":"Project Priority"},{"line_number":84,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5a2ca52d_92e72d12","line":81,"in_reply_to":"7a2fa921_b9d77e22","updated":"2015-10-13 13:54:59.000000000","message":"the first sentence of the previous paragraph gives some concrete examples (promiscuous mode etc). I could also reference the other 2 specs that need the relationship between PF and VFs to be able to schedule.\n\nsince this is mostly about internal refactoring I did not want to go overboard with the specific use casees as they all mostly require some level of neutron integration.","commit_id":"d90861578b01f294c5e1cfe2526fc08cd913fc5b"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"42d7e7347cf800d79bb1c1931d453cd3b8340df0","unresolved":false,"context_lines":[{"line_number":83,"context_line":"Project Priority"},{"line_number":84,"context_line":"-----------------"},{"line_number":85,"context_line":""},{"line_number":86,"context_line":"This is part of NFV enablement and thus not considered a priority."},{"line_number":87,"context_line":""},{"line_number":88,"context_line":"Proposed change"},{"line_number":89,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a2fa921_992ba233","line":86,"updated":"2015-10-13 09:18:43.000000000","message":"We can remove this section.\n\nI quite like this as a technical debt fix, honestly.","commit_id":"d90861578b01f294c5e1cfe2526fc08cd913fc5b"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"42d7e7347cf800d79bb1c1931d453cd3b8340df0","unresolved":false,"context_lines":[{"line_number":112,"context_line":"still taking advantage of some of the functionality. This can be done by using,"},{"line_number":113,"context_line":"for example,"},{"line_number":114,"context_line":"the whitelist configuration option to make sure that a device that is meant to"},{"line_number":115,"context_line":"be treated as a PF, Nova will never see any of VFs that may be related."},{"line_number":116,"context_line":""},{"line_number":117,"context_line":"In order for PFs to work as Neutron ports, some additional work that is outside"},{"line_number":118,"context_line":"of scope of this blueprint will be needed. We aim to make internal Nova changes"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a2fa921_f972d648","line":115,"updated":"2015-10-13 09:18:43.000000000","message":"That sounds like hard work, but it should do the job.","commit_id":"d90861578b01f294c5e1cfe2526fc08cd913fc5b"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"57744fbbe0848b4dcdbf3ac0ca7a9de95236cb7b","unresolved":false,"context_lines":[{"line_number":112,"context_line":"still taking advantage of some of the functionality. This can be done by using,"},{"line_number":113,"context_line":"for example,"},{"line_number":114,"context_line":"the whitelist configuration option to make sure that a device that is meant to"},{"line_number":115,"context_line":"be treated as a PF, Nova will never see any of VFs that may be related."},{"line_number":116,"context_line":""},{"line_number":117,"context_line":"In order for PFs to work as Neutron ports, some additional work that is outside"},{"line_number":118,"context_line":"of scope of this blueprint will be needed. We aim to make internal Nova changes"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5a2ca52d_97c71f45","line":115,"in_reply_to":"7a2fa921_f972d648","updated":"2015-10-13 13:54:59.000000000","message":"Actually it\u0027s not that hard - for Intel cards with the ixgbe driver VFs can be disabled so they won\u0027t even be showing up","commit_id":"d90861578b01f294c5e1cfe2526fc08cd913fc5b"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"42d7e7347cf800d79bb1c1931d453cd3b8340df0","unresolved":false,"context_lines":[{"line_number":143,"context_line":""},{"line_number":144,"context_line":"It is likely to be implemented as a foreign key relationship to the same table,"},{"line_number":145,"context_line":"and objects support will be added, but the actual implementation discussion is"},{"line_number":146,"context_line":"better suited for the actual code proposal review."},{"line_number":147,"context_line":""},{"line_number":148,"context_line":"REST API impact"},{"line_number":149,"context_line":"---------------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a2fa921_19f8f27b","line":146,"updated":"2015-10-13 09:18:43.000000000","message":"honestly, I agree this is likely to come out in the wash.","commit_id":"d90861578b01f294c5e1cfe2526fc08cd913fc5b"},{"author":{"_account_id":6685,"name":"Baodong (Robert) Li","email":"baoli@cisco.com","username":"baoli"},"change_message_id":"1dd41f3758156fae77e9ef534a9e6928eea0ce86","unresolved":false,"context_lines":[{"line_number":91,"context_line":"Two problems we need to solve are:"},{"line_number":92,"context_line":""},{"line_number":93,"context_line":" 1) How to enable requesting a full physical device. This means extending the"},{"line_number":94,"context_line":"    InstancePCIRequest data model to be able to hold this information."},{"line_number":95,"context_line":""},{"line_number":96,"context_line":" 2) Enable scheduling and resource tracking based on the request that can now"},{"line_number":97,"context_line":"    be for the whole device. This means extending the data model for PCIDevices"}],"source_content_type":"text/x-rst","patch_set":5,"id":"5a2ca52d_4cdd6c13","line":94,"updated":"2015-10-13 17:06:19.000000000","message":"if use of \u003cinterface\u003e xml is not expected, the existing PCI Alias can support whole PF passthrough by specifying the vendor id and the product id.\n\nSuppose this whitelist on a node:\n   pci_passthrough_whitelist \u003d {\"vendor_id\":\"1137\", \"product_id\":\"0043\"}\n   pi_passthrough_whitelist \u003d {\"vendor_id\":\"1137\", \"product_id\":\u0027\u00270071\", \"physical_network\": \"hr\"}\n\nand suppose 0043 and 0071 correspondes to PF and VF respectively. Note that the above is possible right now although the first whitelist would result in noop since PF is explicitly excluded.\n\nThe user can use PCI Alias and nova flavor to request a PF or VF, and use of a direct vnic neutron port to request a VF.\n\nSo to me, how to allow the user to specify the intent is critical, and ideally should be covered in \"Proposed change\".","commit_id":"3d0b19177a08edb21226c83165daae26d036d57b"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"1958b9efe97d5babe8c807516fd531fb2a77d2bf","unresolved":false,"context_lines":[{"line_number":91,"context_line":"Two problems we need to solve are:"},{"line_number":92,"context_line":""},{"line_number":93,"context_line":" 1) How to enable requesting a full physical device. This means extending the"},{"line_number":94,"context_line":"    InstancePCIRequest data model to be able to hold this information."},{"line_number":95,"context_line":""},{"line_number":96,"context_line":" 2) Enable scheduling and resource tracking based on the request that can now"},{"line_number":97,"context_line":"    be for the whole device. This means extending the data model for PCIDevices"}],"source_content_type":"text/x-rst","patch_set":5,"id":"5a2ca52d_15e8a7b3","line":94,"in_reply_to":"5a2ca52d_4cdd6c13","updated":"2015-10-14 17:05:59.000000000","message":"OK I can mention something about this without going into how this will be later matched to pools since that\u0027s basically going into the actual implementation.","commit_id":"3d0b19177a08edb21226c83165daae26d036d57b"},{"author":{"_account_id":6685,"name":"Baodong (Robert) Li","email":"baoli@cisco.com","username":"baoli"},"change_message_id":"1dd41f3758156fae77e9ef534a9e6928eea0ce86","unresolved":false,"context_lines":[{"line_number":101,"context_line":"    way we expose the aggregate data about PCI devices to the resource tracker"},{"line_number":102,"context_line":"    (a.k.a. the PCIDeviceStats class) to be able to present PFs and their"},{"line_number":103,"context_line":"    counts, and to make sure to track the corresponding VFs that become"},{"line_number":104,"context_line":"    unavailable once the PF is claimed/used."},{"line_number":105,"context_line":""},{"line_number":106,"context_line":"In addition to the above, it is important to note that the whitelist syntax"},{"line_number":107,"context_line":"of the pci_passthrough_whitelist configuration option support for whitelisting"}],"source_content_type":"text/x-rst","patch_set":5,"id":"5a2ca52d_4f2b1eec","line":104,"updated":"2015-10-13 17:06:19.000000000","message":"Currently the PCI scheduler filter works based on PCI states pools. A pool is a dict with \"product_id\", \"vendor_id\", \"numa_node\", a number of tags if present, as keys. The relationship between PF and VF is not maintained in the pool. (on the other hand, the current implementation doesn\u0027t include PF in the pools since PF is explicitly excluded). If only the compute knows the relationship and thus rejects/accepts the request, then it has to rely on retries to fulfill the request. If the PCI stats pools need to maintain that relationship in some way, how would it be done, and would it be over complicated? So I\u0027m curious on how this can be done for the whole PF passthrough case.","commit_id":"3d0b19177a08edb21226c83165daae26d036d57b"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"1958b9efe97d5babe8c807516fd531fb2a77d2bf","unresolved":false,"context_lines":[{"line_number":101,"context_line":"    way we expose the aggregate data about PCI devices to the resource tracker"},{"line_number":102,"context_line":"    (a.k.a. the PCIDeviceStats class) to be able to present PFs and their"},{"line_number":103,"context_line":"    counts, and to make sure to track the corresponding VFs that become"},{"line_number":104,"context_line":"    unavailable once the PF is claimed/used."},{"line_number":105,"context_line":""},{"line_number":106,"context_line":"In addition to the above, it is important to note that the whitelist syntax"},{"line_number":107,"context_line":"of the pci_passthrough_whitelist configuration option support for whitelisting"}],"source_content_type":"text/x-rst","patch_set":5,"id":"5a2ca52d_df0fae2c","line":104,"in_reply_to":"5a2ca52d_4f2b1eec","updated":"2015-10-14 17:05:59.000000000","message":"That is pretty much something that is best discussed on actual patches. Some basic probing I\u0027ve done around this tells me that we will likely have a pool per PF, and an additional set of PF pools and the claiming/assignment logic would be updated to update both sets of pools.\n\nI don\u0027t know if it will be overcomplicated without actually writing the code. I don\u0027t think the spec is where we should be discussing this though.","commit_id":"3d0b19177a08edb21226c83165daae26d036d57b"},{"author":{"_account_id":6509,"name":"Roman Bogorodskiy","email":"rbogorodskiy@mirantis.com","username":"novel"},"change_message_id":"886d13d705c69a8656f280282f55888b757d87fe","unresolved":false,"context_lines":[{"line_number":101,"context_line":"    way we expose the aggregate data about PCI devices to the resource tracker"},{"line_number":102,"context_line":"    (a.k.a. the PCIDeviceStats class) to be able to present PFs and their"},{"line_number":103,"context_line":"    counts, and to make sure to track the corresponding VFs that become"},{"line_number":104,"context_line":"    unavailable once the PF is claimed/used."},{"line_number":105,"context_line":""},{"line_number":106,"context_line":"In addition to the above, it is important to note that the whitelist syntax"},{"line_number":107,"context_line":"of the pci_passthrough_whitelist configuration option support for whitelisting"}],"source_content_type":"text/x-rst","patch_set":5,"id":"5a2ca52d_f62f1af4","line":104,"in_reply_to":"5a2ca52d_df0fae2c","updated":"2015-10-15 07:38:54.000000000","message":"In the code we have to implement user controlled allocation of VFs (not in gerrit until spec / design is not finalized) we use per-PF pools like you described and it appears to work fine and doesn\u0027t involve major code modifications.","commit_id":"3d0b19177a08edb21226c83165daae26d036d57b"},{"author":{"_account_id":6685,"name":"Baodong (Robert) Li","email":"baoli@cisco.com","username":"baoli"},"change_message_id":"40d7675f392ae19231ce01946c12a4482592a47b","unresolved":false,"context_lines":[{"line_number":41,"context_line":"to manage and goes against the goal of Nova to provide on demand"},{"line_number":42,"context_line":"resources. We want to be able to give certain instances unlimited access to the"},{"line_number":43,"context_line":"port by assigning the PF to it, but revert back to using VFs when the PF is not"},{"line_number":44,"context_line":"being used, so as to ensure on-demand provisioning of available resources. This"},{"line_number":45,"context_line":"may not be possible with every SR-IOV card and their respective Linux drivers,"},{"line_number":46,"context_line":"in which case certain ports will need to be pre-provisioned as either PFs or"},{"line_number":47,"context_line":"VFs by administratior ahead of time."}],"source_content_type":"text/x-rst","patch_set":6,"id":"5a2ca52d_c50f6b37","line":44,"updated":"2015-10-16 18:13:00.000000000","message":"Continue the discussion in patch set 3, I am still a little fuzzy on how this can be done by whitelisting the devices. Maybe an example of whitelists to support the use cases would help explain it. The idea of using a physical port either as a PF or VFs at runtime worries me a bit because how you can guarantee a PF is available when one is really needed. \n\nSorry to keep coming back on this.","commit_id":"6df5e4d7ece724bbd9dbdc3dff4c1ad0c96f8025"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"83366c87cd5861da674cd23d2acdda822ae79a59","unresolved":false,"context_lines":[{"line_number":41,"context_line":"to manage and goes against the goal of Nova to provide on demand"},{"line_number":42,"context_line":"resources. We want to be able to give certain instances unlimited access to the"},{"line_number":43,"context_line":"port by assigning the PF to it, but revert back to using VFs when the PF is not"},{"line_number":44,"context_line":"being used, so as to ensure on-demand provisioning of available resources. This"},{"line_number":45,"context_line":"may not be possible with every SR-IOV card and their respective Linux drivers,"},{"line_number":46,"context_line":"in which case certain ports will need to be pre-provisioned as either PFs or"},{"line_number":47,"context_line":"VFs by administratior ahead of time."}],"source_content_type":"text/x-rst","patch_set":6,"id":"3a29b11f_7b79f7fb","line":44,"in_reply_to":"5a2ca52d_c50f6b37","updated":"2015-10-19 10:26:17.000000000","message":"OK will add an example of the whitelist config here.","commit_id":"6df5e4d7ece724bbd9dbdc3dff4c1ad0c96f8025"},{"author":{"_account_id":6685,"name":"Baodong (Robert) Li","email":"baoli@cisco.com","username":"baoli"},"change_message_id":"40d7675f392ae19231ce01946c12a4482592a47b","unresolved":false,"context_lines":[{"line_number":95,"context_line":"    the whitelist parsing logic that builds up the Spec objects probes the"},{"line_number":96,"context_line":"    system and has the information about whether a device is a PF or not, it is"},{"line_number":97,"context_line":"    enough to add a physical_function field to the PCI alias schema and the"},{"line_number":98,"context_line":"    PCIRequest object."},{"line_number":99,"context_line":""},{"line_number":100,"context_line":" 2) Enable scheduling and resource tracking based on the request that can now"},{"line_number":101,"context_line":"    be for the whole device. This means extending the data model for PCIDevices"}],"source_content_type":"text/x-rst","patch_set":6,"id":"5a2ca52d_cd6b1184","line":98,"updated":"2015-10-16 18:13:00.000000000","message":"thanks. so the intent will be declared in the PCI alias.","commit_id":"6df5e4d7ece724bbd9dbdc3dff4c1ad0c96f8025"},{"author":{"_account_id":3217,"name":"Ian Wells","username":"ijw-ubuntu"},"change_message_id":"c8c3ec38b6f1c19f9053da29ae7007524084abb9","unresolved":false,"context_lines":[{"line_number":35,"context_line":""},{"line_number":36,"context_line":"Currently it is not possible to pass through a physical function to an"},{"line_number":37,"context_line":"OpenStack instance, but some NFV applications need to have full control of the"},{"line_number":38,"context_line":"port, while others are happy with using a VF of an SR-IOV enabled card. It is"},{"line_number":39,"context_line":"beneficial to be able to do so with the same set of cards, as pre-provisioning"},{"line_number":40,"context_line":"resources on the granularity smaller than compute hosts is cumbersome"},{"line_number":41,"context_line":"to manage and goes against the goal of Nova to provide on demand"}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a29b11f_eb3fae84","line":38,"updated":"2015-10-23 01:56:05.000000000","message":"To clarify: \n\nI can pass through a whole NIC port *if and only if* the card doesn\u0027t support SRIOV?\n\nI can pass through a whole NIC port if the card supports SRIOV but no VFs have been created / SRIOV has not been enabled?","commit_id":"7277200f63efe8f036e661e65d5b27d31bf0c531"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"42321f1285035dd2f7fd2d88541851f86abc61fd","unresolved":false,"context_lines":[{"line_number":35,"context_line":""},{"line_number":36,"context_line":"Currently it is not possible to pass through a physical function to an"},{"line_number":37,"context_line":"OpenStack instance, but some NFV applications need to have full control of the"},{"line_number":38,"context_line":"port, while others are happy with using a VF of an SR-IOV enabled card. It is"},{"line_number":39,"context_line":"beneficial to be able to do so with the same set of cards, as pre-provisioning"},{"line_number":40,"context_line":"resources on the granularity smaller than compute hosts is cumbersome"},{"line_number":41,"context_line":"to manage and goes against the goal of Nova to provide on demand"}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a29b11f_0be0e3cb","line":38,"in_reply_to":"3a29b11f_eb3fae84","updated":"2015-10-23 08:14:54.000000000","message":"No, you can passthough any NIC.  If an attempt is made to passthrough a PF and some VFs already exist, then when the drive (ixgbe) is unbound fro the PF, all the VFs will be unconfigured.","commit_id":"7277200f63efe8f036e661e65d5b27d31bf0c531"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"9bebee282a4ec07668207fb102bd32a5e13750b5","unresolved":false,"context_lines":[{"line_number":44,"context_line":"being used, so as to ensure on-demand provisioning of available resources. This"},{"line_number":45,"context_line":"may not be possible with every SR-IOV card and their respective Linux drivers,"},{"line_number":46,"context_line":"in which case certain ports will need to be pre-provisioned as either PFs or"},{"line_number":47,"context_line":"VFs by administratior ahead of time."},{"line_number":48,"context_line":""},{"line_number":49,"context_line":"This in turn means that Nova would have to keep track of which VFs belong to"},{"line_number":50,"context_line":"particular PFs and make sure that this is reflected in the way resources are"}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a29b11f_479a75ba","line":47,"updated":"2015-10-22 10:33:29.000000000","message":"How does this work with cards that do support it. eg if we intend to attach a PF to a guest, i imagine we need to talk to the kernel to disable the VFs before assigning the PF, because the guest may then configure \u0026 use VFs itself and you don\u0027t want 2 kernels seeing VFs at the same time.","commit_id":"7277200f63efe8f036e661e65d5b27d31bf0c531"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"316d5fc03e5bec382f19be37c76d93d2e12cc9ba","unresolved":false,"context_lines":[{"line_number":44,"context_line":"being used, so as to ensure on-demand provisioning of available resources. This"},{"line_number":45,"context_line":"may not be possible with every SR-IOV card and their respective Linux drivers,"},{"line_number":46,"context_line":"in which case certain ports will need to be pre-provisioned as either PFs or"},{"line_number":47,"context_line":"VFs by administratior ahead of time."},{"line_number":48,"context_line":""},{"line_number":49,"context_line":"This in turn means that Nova would have to keep track of which VFs belong to"},{"line_number":50,"context_line":"particular PFs and make sure that this is reflected in the way resources are"}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a29b11f_dd31745b","line":47,"in_reply_to":"3a29b11f_479a75ba","updated":"2015-10-22 11:36:59.000000000","message":"I would imagine most drivers support it. ixgbe driver would remove all VFs (unless they are being used) when the device is removed from the driver (which happens when it\u0027s passed to libvirt with hostdev managed\u003dyes).\n\nhttps://github.com/torvalds/linux/blob/master/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c#L9026\n\nthis is I would imagine the only sane thing to do by a driver.\n\nNB - we will be tracking this as resources in nova so assigning a PF would render VFs unassignable from Nova\u0027s POV regardless, and the opposite would be true - any VF assigned would mean a PF is not available.","commit_id":"7277200f63efe8f036e661e65d5b27d31bf0c531"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"42321f1285035dd2f7fd2d88541851f86abc61fd","unresolved":false,"context_lines":[{"line_number":44,"context_line":"being used, so as to ensure on-demand provisioning of available resources. This"},{"line_number":45,"context_line":"may not be possible with every SR-IOV card and their respective Linux drivers,"},{"line_number":46,"context_line":"in which case certain ports will need to be pre-provisioned as either PFs or"},{"line_number":47,"context_line":"VFs by administratior ahead of time."},{"line_number":48,"context_line":""},{"line_number":49,"context_line":"This in turn means that Nova would have to keep track of which VFs belong to"},{"line_number":50,"context_line":"particular PFs and make sure that this is reflected in the way resources are"}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a29b11f_2bbcbfb9","line":47,"in_reply_to":"3a29b11f_4b0b42d6","updated":"2015-10-23 08:14:54.000000000","message":"That can\u0027t happen because whenever a PCI device is assigned to a guest, it is unbound from the original driver (ixgbe) and bound to pci-stub instead. The pci-stub driver ensures no other host driver can do anything with that device for as long as its assigned to the guest.  So when the guest programs VFs only the guest sees them","commit_id":"7277200f63efe8f036e661e65d5b27d31bf0c531"},{"author":{"_account_id":3217,"name":"Ian Wells","username":"ijw-ubuntu"},"change_message_id":"c8c3ec38b6f1c19f9053da29ae7007524084abb9","unresolved":false,"context_lines":[{"line_number":44,"context_line":"being used, so as to ensure on-demand provisioning of available resources. This"},{"line_number":45,"context_line":"may not be possible with every SR-IOV card and their respective Linux drivers,"},{"line_number":46,"context_line":"in which case certain ports will need to be pre-provisioned as either PFs or"},{"line_number":47,"context_line":"VFs by administratior ahead of time."},{"line_number":48,"context_line":""},{"line_number":49,"context_line":"This in turn means that Nova would have to keep track of which VFs belong to"},{"line_number":50,"context_line":"particular PFs and make sure that this is reflected in the way resources are"}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a29b11f_4b0b42d6","line":47,"in_reply_to":"3a29b11f_dd31745b","updated":"2015-10-23 01:56:05.000000000","message":"Again, I\u0027m concerned what happens here when the VM reprograms the PF to make new / different VFs.  They will be given to the host, and bound to (say) ixgbe as a first preference.  (And you can\u0027t guarantee ixgbe is no longer in the kernel cos it\u0027s not unlikely you\u0027re using an ixgbe interface for your management Openstack network as well.)\n\nI don\u0027t know that there\u0027s anything in vfio that would know what to do with them and I don\u0027t think vfio gets first refusal anyway.","commit_id":"7277200f63efe8f036e661e65d5b27d31bf0c531"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"4e82abe04c246926350ad77a9054a9002bba9424","unresolved":false,"context_lines":[{"line_number":44,"context_line":"being used, so as to ensure on-demand provisioning of available resources. This"},{"line_number":45,"context_line":"may not be possible with every SR-IOV card and their respective Linux drivers,"},{"line_number":46,"context_line":"in which case certain ports will need to be pre-provisioned as either PFs or"},{"line_number":47,"context_line":"VFs by administratior ahead of time."},{"line_number":48,"context_line":""},{"line_number":49,"context_line":"This in turn means that Nova would have to keep track of which VFs belong to"},{"line_number":50,"context_line":"particular PFs and make sure that this is reflected in the way resources are"}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a29b11f_01a22699","line":47,"in_reply_to":"3a29b11f_dd31745b","updated":"2015-10-22 12:47:49.000000000","message":"Ok, so libvirt\u0027s logic is good enough to deal with this.\n\nOne thing to beware of is that when nova reports available PCI devices for the compute host table, we query libvirt to get this list.  When when you assign a PF to a guest, all the VFs will disappear, and so Nova should not even report the existence of VFs at all.","commit_id":"7277200f63efe8f036e661e65d5b27d31bf0c531"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"9bebee282a4ec07668207fb102bd32a5e13750b5","unresolved":false,"context_lines":[{"line_number":51,"context_line":"tracked (so even a single VF being used means the related PF is unavailable and"},{"line_number":52,"context_line":"vice versa, if a PF is being used, all of it\u0027s VFs are marked as used)."},{"line_number":53,"context_line":""},{"line_number":54,"context_line":"The only driver that supports PCI pass-through (libvirt) filters out any device"},{"line_number":55,"context_line":"that is a physical function (this is currently hard-coded in the driver). In"},{"line_number":56,"context_line":"addition, modeling of PCI device resources in Nova currently assumes flat"},{"line_number":57,"context_line":"hierarchy and resource tracking logic does not understand the relationship"}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a29b11f_6740d134","line":54,"updated":"2015-10-22 10:33:29.000000000","message":"The XenAPI driver in Nova supports PCI passthrough too.","commit_id":"7277200f63efe8f036e661e65d5b27d31bf0c531"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"316d5fc03e5bec382f19be37c76d93d2e12cc9ba","unresolved":false,"context_lines":[{"line_number":51,"context_line":"tracked (so even a single VF being used means the related PF is unavailable and"},{"line_number":52,"context_line":"vice versa, if a PF is being used, all of it\u0027s VFs are marked as used)."},{"line_number":53,"context_line":""},{"line_number":54,"context_line":"The only driver that supports PCI pass-through (libvirt) filters out any device"},{"line_number":55,"context_line":"that is a physical function (this is currently hard-coded in the driver). In"},{"line_number":56,"context_line":"addition, modeling of PCI device resources in Nova currently assumes flat"},{"line_number":57,"context_line":"hierarchy and resource tracking logic does not understand the relationship"}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a29b11f_3d43c8b3","line":54,"in_reply_to":"3a29b11f_6740d134","updated":"2015-10-22 11:36:59.000000000","message":"Fair enough - none of the changes are driver specific so I will just update this sentence.","commit_id":"7277200f63efe8f036e661e65d5b27d31bf0c531"},{"author":{"_account_id":3217,"name":"Ian Wells","username":"ijw-ubuntu"},"change_message_id":"c8c3ec38b6f1c19f9053da29ae7007524084abb9","unresolved":false,"context_lines":[{"line_number":52,"context_line":"vice versa, if a PF is being used, all of it\u0027s VFs are marked as used)."},{"line_number":53,"context_line":""},{"line_number":54,"context_line":"The only driver that supports PCI pass-through (libvirt) filters out any device"},{"line_number":55,"context_line":"that is a physical function (this is currently hard-coded in the driver). In"},{"line_number":56,"context_line":"addition, modeling of PCI device resources in Nova currently assumes flat"},{"line_number":57,"context_line":"hierarchy and resource tracking logic does not understand the relationship"},{"line_number":58,"context_line":"between different PCI devices that can be exposed to Nova."}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a29b11f_0b28ea31","line":55,"updated":"2015-10-23 01:56:05.000000000","message":"Again - PF, or potential to be a PF?","commit_id":"7277200f63efe8f036e661e65d5b27d31bf0c531"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"0ee3add7ad75ca82a321aef643c36600fbdd7f12","unresolved":false,"context_lines":[{"line_number":52,"context_line":"vice versa, if a PF is being used, all of it\u0027s VFs are marked as used)."},{"line_number":53,"context_line":""},{"line_number":54,"context_line":"The only driver that supports PCI pass-through (libvirt) filters out any device"},{"line_number":55,"context_line":"that is a physical function (this is currently hard-coded in the driver). In"},{"line_number":56,"context_line":"addition, modeling of PCI device resources in Nova currently assumes flat"},{"line_number":57,"context_line":"hierarchy and resource tracking logic does not understand the relationship"},{"line_number":58,"context_line":"between different PCI devices that can be exposed to Nova."}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a29b11f_b0131a0d","line":55,"in_reply_to":"3a29b11f_0b28ea31","updated":"2015-10-26 15:06:31.000000000","message":"That\u0027s a good point - I actually had to go look at the code.\n\nIt seems that both will be filtered out but I would have to confirm as I don\u0027t have access to my test box.\n\nWill make it clearer in the text.","commit_id":"7277200f63efe8f036e661e65d5b27d31bf0c531"},{"author":{"_account_id":3217,"name":"Ian Wells","username":"ijw-ubuntu"},"change_message_id":"c8c3ec38b6f1c19f9053da29ae7007524084abb9","unresolved":false,"context_lines":[{"line_number":64,"context_line":"Certain NFV workloads may need to have the full control of the physical device,"},{"line_number":65,"context_line":"in order to use some of the functionality not available to the VF (an example"},{"line_number":66,"context_line":"is the need to set the port into promiscuous mode, which may be a requirement"},{"line_number":67,"context_line":"for certain types of VNFs) or to exclusively use the full bandwidth of the"},{"line_number":68,"context_line":"port. However, due to the dynamic nature of the elastic cloud, and the promise"},{"line_number":69,"context_line":"of Nova to we do not wish to have to pre-provision certain SR-IOV cards to be"},{"line_number":70,"context_line":"used as PFs as this defeats the promise of the infrastructure management tool"}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a29b11f_4bd90256","line":67,"updated":"2015-10-23 01:56:05.000000000","message":"Lack of promiscuous mode is not a reason to use PFs.  \n\nThe problem you\u0027re talking about is a feature of certain Intel NICs, which have antispoofing filters on when you\u0027re using VFs and not when you\u0027re using the whole device (filters which, I presume, you can\u0027t disable via the PF, given this point keeps coming up).\n\n(Also, it\u0027s not that you can\u0027t put them in promiscuous mode, it\u0027s that you won\u0027t see all the packets when you do ;)","commit_id":"7277200f63efe8f036e661e65d5b27d31bf0c531"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"0ee3add7ad75ca82a321aef643c36600fbdd7f12","unresolved":false,"context_lines":[{"line_number":64,"context_line":"Certain NFV workloads may need to have the full control of the physical device,"},{"line_number":65,"context_line":"in order to use some of the functionality not available to the VF (an example"},{"line_number":66,"context_line":"is the need to set the port into promiscuous mode, which may be a requirement"},{"line_number":67,"context_line":"for certain types of VNFs) or to exclusively use the full bandwidth of the"},{"line_number":68,"context_line":"port. However, due to the dynamic nature of the elastic cloud, and the promise"},{"line_number":69,"context_line":"of Nova to we do not wish to have to pre-provision certain SR-IOV cards to be"},{"line_number":70,"context_line":"used as PFs as this defeats the promise of the infrastructure management tool"}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a29b11f_1003eeb9","line":67,"in_reply_to":"3a29b11f_4bd90256","updated":"2015-10-26 15:06:31.000000000","message":"I will update this bit to make it more clear","commit_id":"7277200f63efe8f036e661e65d5b27d31bf0c531"},{"author":{"_account_id":3217,"name":"Ian Wells","username":"ijw-ubuntu"},"change_message_id":"c8c3ec38b6f1c19f9053da29ae7007524084abb9","unresolved":false,"context_lines":[{"line_number":68,"context_line":"port. However, due to the dynamic nature of the elastic cloud, and the promise"},{"line_number":69,"context_line":"of Nova to we do not wish to have to pre-provision certain SR-IOV cards to be"},{"line_number":70,"context_line":"used as PFs as this defeats the promise of the infrastructure management tool"},{"line_number":71,"context_line":"that allows for quick re-purposing of resources that Nova brings."},{"line_number":72,"context_line":""},{"line_number":73,"context_line":"Modern SR-IOV enabled cards along with their drivers usually allow for such"},{"line_number":74,"context_line":"reconfiguration to be done on the fly, so once the passthrough of the PF is no"}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a29b11f_2ba4c6b7","line":71,"updated":"2015-10-23 01:56:05.000000000","message":"You know, while I appreciate this logic, I do wonder if we should have dealt with whole-device Neutron mediated passthrough before we started down whole-device vs SRIOV scheduling, programming or whatever.  But that\u0027s not a reason to reject this spec and it may be something we can do in parallel.","commit_id":"7277200f63efe8f036e661e65d5b27d31bf0c531"},{"author":{"_account_id":5319,"name":"Sam Stoelinga","email":"sammiestoel@gmail.com","username":"sammiestoel"},"change_message_id":"641b2a74bdb331c5b3c7995e3cd73e02805514b2","unresolved":false,"context_lines":[{"line_number":68,"context_line":"port. However, due to the dynamic nature of the elastic cloud, and the promise"},{"line_number":69,"context_line":"of Nova to we do not wish to have to pre-provision certain SR-IOV cards to be"},{"line_number":70,"context_line":"used as PFs as this defeats the promise of the infrastructure management tool"},{"line_number":71,"context_line":"that allows for quick re-purposing of resources that Nova brings."},{"line_number":72,"context_line":""},{"line_number":73,"context_line":"Modern SR-IOV enabled cards along with their drivers usually allow for such"},{"line_number":74,"context_line":"reconfiguration to be done on the fly, so once the passthrough of the PF is no"}],"source_content_type":"text/x-rst","patch_set":7,"id":"1a26ad4f_6072a4e7","line":71,"in_reply_to":"3a29b11f_2ba4c6b7","updated":"2015-10-28 07:25:50.000000000","message":"I\u0027m not sure if we are on the same page, but I agree that we should not care whether a device is SRIOV device or not. An example would be creating an NFV compute node with 10 physcial NICs and each VM gets a physical NIC assigned to use via PCI passthrough, whether this device is a PF or not doesn\u0027t matter. If it\u0027s an PF then we should make sure that there weren\u0027t any VFs in use by a VM.","commit_id":"7277200f63efe8f036e661e65d5b27d31bf0c531"},{"author":{"_account_id":3217,"name":"Ian Wells","username":"ijw-ubuntu"},"change_message_id":"c8c3ec38b6f1c19f9053da29ae7007524084abb9","unresolved":false,"context_lines":[{"line_number":74,"context_line":"reconfiguration to be done on the fly, so once the passthrough of the PF is no"},{"line_number":75,"context_line":"longer needed on a specific host (either the instance using it got moved or"},{"line_number":76,"context_line":"deleted), it depends on the Linux driver for the specific card, if VFs will be"},{"line_number":77,"context_line":"restored once the card is returned to the host OS. It is not possible to"},{"line_number":78,"context_line":"guarantee that this always works however, due to the vast range of equipment"},{"line_number":79,"context_line":"and drivers available on the market, so we want to make sure that there is a"},{"line_number":80,"context_line":"way to tell Nova that a card is in certain configuration and cannot be assumed"}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a29b11f_eb97ae37","line":77,"updated":"2015-10-23 01:56:05.000000000","message":"It never technically leaves the host OS, it\u0027s more \u0027when it is rebound to the ethernet driver from the vfio driver\u0027, but that\u0027s really just a wording quibble.","commit_id":"7277200f63efe8f036e661e65d5b27d31bf0c531"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"0ee3add7ad75ca82a321aef643c36600fbdd7f12","unresolved":false,"context_lines":[{"line_number":74,"context_line":"reconfiguration to be done on the fly, so once the passthrough of the PF is no"},{"line_number":75,"context_line":"longer needed on a specific host (either the instance using it got moved or"},{"line_number":76,"context_line":"deleted), it depends on the Linux driver for the specific card, if VFs will be"},{"line_number":77,"context_line":"restored once the card is returned to the host OS. It is not possible to"},{"line_number":78,"context_line":"guarantee that this always works however, due to the vast range of equipment"},{"line_number":79,"context_line":"and drivers available on the market, so we want to make sure that there is a"},{"line_number":80,"context_line":"way to tell Nova that a card is in certain configuration and cannot be assumed"}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a29b11f_701d820e","line":77,"in_reply_to":"3a29b11f_eb97ae37","updated":"2015-10-26 15:06:31.000000000","message":"Will make the sentence use more precise language - thanks!","commit_id":"7277200f63efe8f036e661e65d5b27d31bf0c531"},{"author":{"_account_id":6685,"name":"Baodong (Robert) Li","email":"baoli@cisco.com","username":"baoli"},"change_message_id":"b302a01005b1c9b432c6c73ea2756f24f5031f24","unresolved":false,"context_lines":[{"line_number":123,"context_line":"example - specifying a PF and all of it\u0027s VFs - it would be enough to do"},{"line_number":124,"context_line":"(on my server)::"},{"line_number":125,"context_line":""},{"line_number":126,"context_line":"    pci_passthrough_whitelist \u003d {\"vendor_id\":\"1520\",\"product_id\":\"8086\"}"},{"line_number":127,"context_line":""},{"line_number":128,"context_line":"Which would make sure that both the PF and all of the enabled VFs. Filtering"},{"line_number":129,"context_line":"out the PF would currently be slightly trickier as the admin would need to"}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a29b11f_453a63e8","line":126,"updated":"2015-10-22 15:32:44.000000000","message":"What I have seen so far is that the PFs and VFs have different product (or device) ids. PF: 8086:10fb, VF: 8086:10ed\n\n$ lspci -nn | grep -i ethernet | grep -i intel\n\n10:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)\n\n10:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)\n\n10:10.0 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)\n\n10:10.1 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)\n\n10:11.0 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)\n\n10:11.1 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)\n\nAlso, I thought that vendor_id is 8086, product_id 1520\n\nThe existing semantics is that if a PF\u0027s ADDRESS is specified, all of its VFs will be included (but not the PF itself). But if it\u0027s just the vendor_id and product_id that are specified, then only the devices with the vendor_id:product_id are accounted. I am not aware of any change that enables the semantic you mentioned here. It could be a recent change that I missed.","commit_id":"7277200f63efe8f036e661e65d5b27d31bf0c531"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"0ee3add7ad75ca82a321aef643c36600fbdd7f12","unresolved":false,"context_lines":[{"line_number":123,"context_line":"example - specifying a PF and all of it\u0027s VFs - it would be enough to do"},{"line_number":124,"context_line":"(on my server)::"},{"line_number":125,"context_line":""},{"line_number":126,"context_line":"    pci_passthrough_whitelist \u003d {\"vendor_id\":\"1520\",\"product_id\":\"8086\"}"},{"line_number":127,"context_line":""},{"line_number":128,"context_line":"Which would make sure that both the PF and all of the enabled VFs. Filtering"},{"line_number":129,"context_line":"out the PF would currently be slightly trickier as the admin would need to"}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a29b11f_533ad0ce","line":126,"in_reply_to":"3a29b11f_453a63e8","updated":"2015-10-26 15:06:31.000000000","message":"Hmmm let me look into this and update the spec here (likely after the summit).\n\nthis does not seem like something that we need to discuss on the spec though. At the end of the day we are talking about matching bits of text, hardly a problem that needs design discussion in a spec.\n\nWhat we may want to do is have better documentation for this once we\u0027re agreed on the approach (via code review) and patches are merged.","commit_id":"7277200f63efe8f036e661e65d5b27d31bf0c531"},{"author":{"_account_id":6685,"name":"Baodong (Robert) Li","email":"baoli@cisco.com","username":"baoli"},"change_message_id":"b302a01005b1c9b432c6c73ea2756f24f5031f24","unresolved":false,"context_lines":[{"line_number":155,"context_line":"                                 {\"address\":\"*:08:11:*\"},"},{"line_number":156,"context_line":"                                 {\"address\":\"*:08:12:*\"},"},{"line_number":157,"context_line":"                                 {\"address\":\"*:08:13:*\"}]"},{"line_number":158,"context_line":""},{"line_number":159,"context_line":""},{"line_number":160,"context_line":"potentially more complicated if we need to exclude further devices. Another BP"},{"line_number":161,"context_line":"[3]_ that will enhance the whitelist syntax with regexes will make this"}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a29b11f_3edc1445","line":158,"updated":"2015-10-22 15:32:44.000000000","message":"I\u0027m still not quite clear from reading above. If it\u0027s the whole PF passthrough, does the admin only specify the PF\u0027s vendor_id:product_id in the whitelist?","commit_id":"7277200f63efe8f036e661e65d5b27d31bf0c531"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"9bebee282a4ec07668207fb102bd32a5e13750b5","unresolved":false,"context_lines":[{"line_number":168,"context_line":"since there will be no Neutron support"},{"line_number":169,"context_line":"at first, the only way to assign such a device would be using the \u003chostdev\u003e"},{"line_number":170,"context_line":"element, and no support for \u003cinterface\u003e is in scope for this blueprint."},{"line_number":171,"context_line":""},{"line_number":172,"context_line":"Alternatives"},{"line_number":173,"context_line":"------------"},{"line_number":174,"context_line":""}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a29b11f_27e2d92f","line":171,"updated":"2015-10-22 10:33:29.000000000","message":"I\u0027m not seeing info here on how we re-configure the host OS to disable the VFs when assigning a PF.","commit_id":"7277200f63efe8f036e661e65d5b27d31bf0c531"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"316d5fc03e5bec382f19be37c76d93d2e12cc9ba","unresolved":false,"context_lines":[{"line_number":168,"context_line":"since there will be no Neutron support"},{"line_number":169,"context_line":"at first, the only way to assign such a device would be using the \u003chostdev\u003e"},{"line_number":170,"context_line":"element, and no support for \u003cinterface\u003e is in scope for this blueprint."},{"line_number":171,"context_line":""},{"line_number":172,"context_line":"Alternatives"},{"line_number":173,"context_line":"------------"},{"line_number":174,"context_line":""}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a29b11f_5d53246b","line":171,"in_reply_to":"3a29b11f_27e2d92f","updated":"2015-10-22 11:36:59.000000000","message":"Described above I thin - It\u0027s good to notice the opposite is not true, VFs will not be re-configured on the PF being given back to the PCI driver. This is easily worked around by udev I think.","commit_id":"7277200f63efe8f036e661e65d5b27d31bf0c531"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"4e82abe04c246926350ad77a9054a9002bba9424","unresolved":false,"context_lines":[{"line_number":168,"context_line":"since there will be no Neutron support"},{"line_number":169,"context_line":"at first, the only way to assign such a device would be using the \u003chostdev\u003e"},{"line_number":170,"context_line":"element, and no support for \u003cinterface\u003e is in scope for this blueprint."},{"line_number":171,"context_line":""},{"line_number":172,"context_line":"Alternatives"},{"line_number":173,"context_line":"------------"},{"line_number":174,"context_line":""}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a29b11f_810ab688","line":171,"in_reply_to":"3a29b11f_5d53246b","updated":"2015-10-22 12:47:49.000000000","message":"We should probably make a note about this. There is some host OS initial setup that the admin does when provisioning a compute host, where they enable some number of VFs. We need to make sure we return to the original state the admin setup. This might be just a documentation task for extra setup steps","commit_id":"7277200f63efe8f036e661e65d5b27d31bf0c531"},{"author":{"_account_id":17973,"name":"Ludovic Beliveau","email":"lbeliveau@gmail.com","username":"lbeliveau"},"change_message_id":"009353401540c060c782e82629da85a6a1425104","unresolved":false,"context_lines":[{"line_number":19,"context_line":"as a full device (PF - physical function)."},{"line_number":20,"context_line":""},{"line_number":21,"context_line":"Nova\u0027s current handling exposes only virtual functions as resources that can"},{"line_number":22,"context_line":"be requested by instances - and this is the most common use case by far."},{"line_number":23,"context_line":"However with the rise of the requirements to virtualize network applications,"},{"line_number":24,"context_line":"it can be necessary to give instances full control over the port and not just a"},{"line_number":25,"context_line":"single virtual function."}],"source_content_type":"text/x-rst","patch_set":8,"id":"da85f559_97562ccd","line":22,"updated":"2015-11-12 22:05:58.000000000","message":"I\u0027ve never had problem allocating a pci passthrough device (not SR-IOV/VF) to a guest and booting it  using usual pci_alias and pci_passthrough_whitelist.  Not sure what is the issue here or maybe it needs clarification.\n\nIn this case, the pci passthrough is exposed as a normal PCI device.  So I don\u0027t understand why it says that it only \"exposes virtual functions\" ?","commit_id":"866f306bcfe46b3f9433009ce76731c304278e2d"},{"author":{"_account_id":17973,"name":"Ludovic Beliveau","email":"lbeliveau@gmail.com","username":"lbeliveau"},"change_message_id":"009353401540c060c782e82629da85a6a1425104","unresolved":false,"context_lines":[{"line_number":34,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":35,"context_line":""},{"line_number":36,"context_line":"Currently it is not possible to pass through a physical function to an"},{"line_number":37,"context_line":"OpenStack instance, but some NFV applications need to have full control of the"},{"line_number":38,"context_line":"port, while others are happy with using a VF of an SR-IOV enabled card. It is"},{"line_number":39,"context_line":"beneficial to be able to do so with the same set of cards, as pre-provisioning"},{"line_number":40,"context_line":"resources on the granularity smaller than compute hosts is cumbersome"}],"source_content_type":"text/x-rst","patch_set":8,"id":"da85f559_b7f6d0f7","line":37,"updated":"2015-11-12 22:05:58.000000000","message":"Again, I\u0027ve never had problem allocating a pci passthrough to a guest ...\n\nAre you refering to the fact that a NIC pci passthrough can\u0027t be configured with a network (via neutron port-create --binding:vnic-type direct) like it is for a NIC SR-IOV device ?","commit_id":"866f306bcfe46b3f9433009ce76731c304278e2d"},{"author":{"_account_id":19212,"name":"Antonio López","email":"antonio.lopezgracia@telefonica.com","username":"alogra"},"change_message_id":"13199302aca4d29601115456a1b2705da3c30459","unresolved":false,"context_lines":[{"line_number":34,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":35,"context_line":""},{"line_number":36,"context_line":"Currently it is not possible to pass through a physical function to an"},{"line_number":37,"context_line":"OpenStack instance, but some NFV applications need to have full control of the"},{"line_number":38,"context_line":"port, while others are happy with using a VF of an SR-IOV enabled card. It is"},{"line_number":39,"context_line":"beneficial to be able to do so with the same set of cards, as pre-provisioning"},{"line_number":40,"context_line":"resources on the granularity smaller than compute hosts is cumbersome"}],"source_content_type":"text/x-rst","patch_set":8,"id":"ba8a016a_431c36c1","line":37,"in_reply_to":"da85f559_b7f6d0f7","updated":"2015-11-17 16:18:46.000000000","message":"I think you are right. The final goal is to have passthrough ports exposed in Neutron like a SR-IOV device, so the underlay connections through an external switch -or any other automated provision operation- can be fired when a guest is booted with passthrough ports attached.","commit_id":"866f306bcfe46b3f9433009ce76731c304278e2d"},{"author":{"_account_id":17973,"name":"Ludovic Beliveau","email":"lbeliveau@gmail.com","username":"lbeliveau"},"change_message_id":"009353401540c060c782e82629da85a6a1425104","unresolved":false,"context_lines":[{"line_number":41,"context_line":"to manage and goes against the goal of Nova to provide on demand"},{"line_number":42,"context_line":"resources. We want to be able to give certain instances unlimited access to the"},{"line_number":43,"context_line":"port by assigning the PF to it, but revert back to using VFs when the PF is not"},{"line_number":44,"context_line":"being used, so as to ensure on-demand provisioning of available resources. This"},{"line_number":45,"context_line":"may not be possible with every SR-IOV card and their respective Linux drivers,"},{"line_number":46,"context_line":"in which case certain ports will need to be pre-provisioned as either PFs or"},{"line_number":47,"context_line":"VFs by administratior ahead of time."}],"source_content_type":"text/x-rst","patch_set":8,"id":"da85f559_7778682c","line":44,"updated":"2015-11-12 22:05:58.000000000","message":"If the real goal of this feature is to support dynamic reconfiguration of the PCI device (PF to VF and vice versa), then I think this should be specified in the title and description (assuming that pci passthrough does work currently).","commit_id":"866f306bcfe46b3f9433009ce76731c304278e2d"},{"author":{"_account_id":19212,"name":"Antonio López","email":"antonio.lopezgracia@telefonica.com","username":"alogra"},"change_message_id":"13199302aca4d29601115456a1b2705da3c30459","unresolved":false,"context_lines":[{"line_number":50,"context_line":"particular PFs and make sure that this is reflected in the way resources are"},{"line_number":51,"context_line":"tracked (so even a single VF being used means the related PF is unavailable and"},{"line_number":52,"context_line":"vice versa, if a PF is being used, all of it\u0027s VFs are marked as used)."},{"line_number":53,"context_line":""},{"line_number":54,"context_line":"PCI device management code in Nova currently filters out any"},{"line_number":55,"context_line":"device that is a physical function (this is currently hard-coded). In"},{"line_number":56,"context_line":"addition, modeling of PCI device resources in Nova currently assumes flat"}],"source_content_type":"text/x-rst","patch_set":8,"id":"ba8a016a_031cee75","line":53,"updated":"2015-11-17 16:18:46.000000000","message":"It would be great to track also the bandwidth of the VFs: if a VF is allocated with all the bandwidth of the underlying PF all other VFs defined in the host over the same PF should be marked as unavailable. \n\nEach VF allocated should substract its declared bandwidth from the total PF bandwidth, so if a new VF is requested with a bandwidth greater than the remaining bandwidth in the PF, this new VF should not be allocated.","commit_id":"866f306bcfe46b3f9433009ce76731c304278e2d"},{"author":{"_account_id":17973,"name":"Ludovic Beliveau","email":"lbeliveau@gmail.com","username":"lbeliveau"},"change_message_id":"009353401540c060c782e82629da85a6a1425104","unresolved":false,"context_lines":[{"line_number":113,"context_line":"Currently if a whitelist entry"},{"line_number":114,"context_line":"specifies an address or a devname of a PF, the matching code will make sure"},{"line_number":115,"context_line":"any of the VFs match. This behavior, combined with allowing a device that is a"},{"line_number":116,"context_line":"PF to be tracked by nova (by removing the hard-coded check that skips any PFs)"},{"line_number":117,"context_line":"should be sufficient to allow most of the flexibility administrators need."},{"line_number":118,"context_line":"As it is not sufficient for a device to be whitelisted to be requestable by"},{"line_number":119,"context_line":"users (it needs to either have an alias that is specified on the flavor),"}],"source_content_type":"text/x-rst","patch_set":8,"id":"da85f559_8b7361e7","line":116,"updated":"2015-11-12 22:05:58.000000000","message":"Which hard-coded check ?","commit_id":"866f306bcfe46b3f9433009ce76731c304278e2d"},{"author":{"_account_id":19212,"name":"Antonio López","email":"antonio.lopezgracia@telefonica.com","username":"alogra"},"change_message_id":"13199302aca4d29601115456a1b2705da3c30459","unresolved":false,"context_lines":[{"line_number":132,"context_line":"device is unbound from the host driver."},{"line_number":133,"context_line":""},{"line_number":134,"context_line":"In order for PFs to work as Neutron ports, some additional work that is outside"},{"line_number":135,"context_line":"of scope of this blueprint will be needed. We aim to make internal Nova changes"},{"line_number":136,"context_line":"that are needed the focus here and defer on the integration work to a future"},{"line_number":137,"context_line":"(possibly cross-project) blueprint. For the libvirt driver, this means that,"},{"line_number":138,"context_line":"since there will be no Neutron support"}],"source_content_type":"text/x-rst","patch_set":8,"id":"ba8a016a_f4d39e5d","line":135,"updated":"2015-11-17 16:18:46.000000000","message":"Any idea of when will be scoped to expose PFs as Neutron ports? That is a very important requirement for NFV workloads and automated orchestration.","commit_id":"866f306bcfe46b3f9433009ce76731c304278e2d"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"6cab78352c55437ade2f419b0b52731976db14e3","unresolved":false,"context_lines":[{"line_number":137,"context_line":"(possibly cross-project) blueprint. For the libvirt driver, this means that,"},{"line_number":138,"context_line":"since there will be no Neutron support"},{"line_number":139,"context_line":"at first, the only way to assign such a device would be using the \u003chostdev\u003e"},{"line_number":140,"context_line":"element, and no support for \u003cinterface\u003e is in scope for this blueprint."},{"line_number":141,"context_line":""},{"line_number":142,"context_line":"Alternatives"},{"line_number":143,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":8,"id":"da85f559_dddf5738","line":140,"updated":"2015-11-11 10:50:55.000000000","message":"Nit: more detail in here than I like in a spec really, but I don\u0027t disagree with any of it, so lets keep it.","commit_id":"866f306bcfe46b3f9433009ce76731c304278e2d"},{"author":{"_account_id":19212,"name":"Antonio López","email":"antonio.lopezgracia@telefonica.com","username":"alogra"},"change_message_id":"13199302aca4d29601115456a1b2705da3c30459","unresolved":false,"context_lines":[{"line_number":157,"context_line":"to (through the use of `extra_info` free-form field) it may be necessary to add"},{"line_number":158,"context_line":"a more \"query friendly\" relationship, that will allow us to answer the question"},{"line_number":159,"context_line":"\"given a PCI device record that is a PF, which VF records does it contain\"."},{"line_number":160,"context_line":""},{"line_number":161,"context_line":"It is likely to be implemented as a foreign key relationship to the same table,"},{"line_number":162,"context_line":"and objects support will be added, but the actual implementation discussion is"},{"line_number":163,"context_line":"better suited for the actual code proposal review."}],"source_content_type":"text/x-rst","patch_set":8,"id":"ba8a016a_54214a4c","line":160,"updated":"2015-11-17 16:18:46.000000000","message":"I think total BW of the PF and BW of any allocated VF should be also added to the DDBB in order to be able to manage correctly SRIOV ports","commit_id":"866f306bcfe46b3f9433009ce76731c304278e2d"},{"author":{"_account_id":17973,"name":"Ludovic Beliveau","email":"lbeliveau@gmail.com","username":"lbeliveau"},"change_message_id":"009353401540c060c782e82629da85a6a1425104","unresolved":false,"context_lines":[{"line_number":235,"context_line":"* Change the PCIDeviceStats class to expose PFs in it\u0027s pools, and change the"},{"line_number":236,"context_line":"  claiming/consuming logic to claim appropriate amounts of VFs when a PF is"},{"line_number":237,"context_line":"  consumed or claimed. Once this work item is complete, all of the scheduling"},{"line_number":238,"context_line":"  and resource tracking logic will be aware of the PF constraint."},{"line_number":239,"context_line":""},{"line_number":240,"context_line":"* Add support for specifying the PF requirement through the pci_alias"},{"line_number":241,"context_line":"  configuration options, so that it can be requested through flavor"}],"source_content_type":"text/x-rst","patch_set":8,"id":"da85f559_66951487","line":238,"updated":"2015-11-12 22:05:58.000000000","message":"Abstracting the PF-VF relationship, it could be easy at the moment for the scheduler to differentiate between a PF and a VF.\n\nPciDevice already exposes the device type (dev_type can be type-PCI or type-VF).\n\nPciDeviceStats could be easily modified to specify \"dev_type\" in the pool_keys.  Therefore this field would be automatically exported to the scehduler.  See https://github.com/openstack/nova/blob/master/nova/pci/stats.py#L56","commit_id":"866f306bcfe46b3f9433009ce76731c304278e2d"},{"author":{"_account_id":17973,"name":"Ludovic Beliveau","email":"lbeliveau@gmail.com","username":"lbeliveau"},"change_message_id":"009353401540c060c782e82629da85a6a1425104","unresolved":false,"context_lines":[{"line_number":239,"context_line":""},{"line_number":240,"context_line":"* Add support for specifying the PF requirement through the pci_alias"},{"line_number":241,"context_line":"  configuration options, so that it can be requested through flavor"},{"line_number":242,"context_line":"  extra-specs."},{"line_number":243,"context_line":""},{"line_number":244,"context_line":""},{"line_number":245,"context_line":"Dependencies"}],"source_content_type":"text/x-rst","patch_set":8,"id":"da85f559_860fd82d","line":242,"updated":"2015-11-12 22:05:58.000000000","message":"Do we want to extend the pci_alias to allow specifying a physical network (similar with what is done with NIC SR-IOV) ?","commit_id":"866f306bcfe46b3f9433009ce76731c304278e2d"}]}