)]}'
{"specs/ocata/approved/share-pci-between-numa-nodes.rst":[{"author":{"_account_id":17973,"name":"Ludovic Beliveau","email":"lbeliveau@gmail.com","username":"lbeliveau"},"change_message_id":"340f8ff4c6f34da987c2709494985a3df5638ad9","unresolved":false,"context_lines":[{"line_number":43,"context_line":"Proposed change"},{"line_number":44,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":45,"context_line":""},{"line_number":46,"context_line":"Add new config option \u0027share_pci_between_numa_nodes\u0027 in default section of nova"},{"line_number":47,"context_line":"config. This option will be used by scheduler\u0027s NUMATopologyFilter which will"},{"line_number":48,"context_line":"decide on which NUMA nodes VM can be booted. Default value of this option"},{"line_number":49,"context_line":"will be \u0027False\u0027. It means that Nova will boot VMs with PCI devices *only* on"}],"source_content_type":"text/x-rst","patch_set":1,"id":"fa7ab95a_f4ad5989","line":46,"updated":"2016-08-26 13:12:06.000000000","message":"Your proposition doesn\u0027t allow to have granular control of this feature.  Maybe some guests cares about PCI numa locality, some others don\u0027t.  For this reason I think we should provide a per guest config.  Maybe in the flavor extra spec.  What do you think ?","commit_id":"afb631e8a2f0e42a56ef0788fe1341f18f9b0035"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"4c149de45d3a40b4b8f683dc77e383e2129a76fd","unresolved":false,"context_lines":[{"line_number":43,"context_line":"Proposed change"},{"line_number":44,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":45,"context_line":""},{"line_number":46,"context_line":"Add new config option \u0027share_pci_between_numa_nodes\u0027 in default section of nova"},{"line_number":47,"context_line":"config. This option will be used by scheduler\u0027s NUMATopologyFilter which will"},{"line_number":48,"context_line":"decide on which NUMA nodes VM can be booted. Default value of this option"},{"line_number":49,"context_line":"will be \u0027False\u0027. It means that Nova will boot VMs with PCI devices *only* on"}],"source_content_type":"text/x-rst","patch_set":1,"id":"fa7ab95a_2a12f8b4","line":46,"in_reply_to":"fa7ab95a_0a8e1478","updated":"2016-08-26 13:36:14.000000000","message":"if yes, I think it\u0027s good.  it will add more flexibility","commit_id":"afb631e8a2f0e42a56ef0788fe1341f18f9b0035"},{"author":{"_account_id":17973,"name":"Ludovic Beliveau","email":"lbeliveau@gmail.com","username":"lbeliveau"},"change_message_id":"9af43747036c5dd4082a134df9d40263211097cd","unresolved":false,"context_lines":[{"line_number":43,"context_line":"Proposed change"},{"line_number":44,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":45,"context_line":""},{"line_number":46,"context_line":"Add new config option \u0027share_pci_between_numa_nodes\u0027 in default section of nova"},{"line_number":47,"context_line":"config. This option will be used by scheduler\u0027s NUMATopologyFilter which will"},{"line_number":48,"context_line":"decide on which NUMA nodes VM can be booted. Default value of this option"},{"line_number":49,"context_line":"will be \u0027False\u0027. It means that Nova will boot VMs with PCI devices *only* on"}],"source_content_type":"text/x-rst","patch_set":1,"id":"fa7ab95a_0c69d1be","line":46,"in_reply_to":"fa7ab95a_2a12f8b4","updated":"2016-08-26 13:42:13.000000000","message":"Exactly.  Some guest/application have strict requirement on performance and need to specify that they need such numa locality and nova needs to schedule the guest base on this criteria.  Some others don\u0027t care ...","commit_id":"afb631e8a2f0e42a56ef0788fe1341f18f9b0035"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"24e1091f38805be9ab3bff088b3172786049b910","unresolved":false,"context_lines":[{"line_number":43,"context_line":"Proposed change"},{"line_number":44,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":45,"context_line":""},{"line_number":46,"context_line":"Add new config option \u0027share_pci_between_numa_nodes\u0027 in default section of nova"},{"line_number":47,"context_line":"config. This option will be used by scheduler\u0027s NUMATopologyFilter which will"},{"line_number":48,"context_line":"decide on which NUMA nodes VM can be booted. Default value of this option"},{"line_number":49,"context_line":"will be \u0027False\u0027. It means that Nova will boot VMs with PCI devices *only* on"}],"source_content_type":"text/x-rst","patch_set":1,"id":"fa7ab95a_0a8e1478","line":46,"in_reply_to":"fa7ab95a_f4ad5989","updated":"2016-08-26 13:31:43.000000000","message":"You mean forget about new config option and use some flavor property \u0027ignoring numa\u0027 instead of it?","commit_id":"afb631e8a2f0e42a56ef0788fe1341f18f9b0035"},{"author":{"_account_id":17973,"name":"Ludovic Beliveau","email":"lbeliveau@gmail.com","username":"lbeliveau"},"change_message_id":"340f8ff4c6f34da987c2709494985a3df5638ad9","unresolved":false,"context_lines":[{"line_number":50,"context_line":"NUMA nodes associated with these PCI devices (current behaviour)."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"The \u0027True\u0027 value will mean that scheduler will ignore association of PCI"},{"line_number":53,"context_line":"devices and NUMA nodes."},{"line_number":54,"context_line":""},{"line_number":55,"context_line":"Alternatives"},{"line_number":56,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"fa7ab95a_5417e5a2","line":53,"updated":"2016-08-26 13:12:06.000000000","message":"In this case should we still try to have PCI numa locality (meaning PCI and vcpu are sharing the same numa) first, then if it fails we can fall back to ignoring the numa.","commit_id":"afb631e8a2f0e42a56ef0788fe1341f18f9b0035"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"24e1091f38805be9ab3bff088b3172786049b910","unresolved":false,"context_lines":[{"line_number":50,"context_line":"NUMA nodes associated with these PCI devices (current behaviour)."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"The \u0027True\u0027 value will mean that scheduler will ignore association of PCI"},{"line_number":53,"context_line":"devices and NUMA nodes."},{"line_number":54,"context_line":""},{"line_number":55,"context_line":"Alternatives"},{"line_number":56,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"fa7ab95a_4f7e6ee5","line":53,"in_reply_to":"fa7ab95a_5417e5a2","updated":"2016-08-26 13:31:43.000000000","message":"Yes, It\u0027s a good Idea. I wanted to implement it, but unfortunately forget to mention it in spec","commit_id":"afb631e8a2f0e42a56ef0788fe1341f18f9b0035"},{"author":{"_account_id":17973,"name":"Ludovic Beliveau","email":"lbeliveau@gmail.com","username":"lbeliveau"},"change_message_id":"340f8ff4c6f34da987c2709494985a3df5638ad9","unresolved":false,"context_lines":[{"line_number":118,"context_line":"----------"},{"line_number":119,"context_line":""},{"line_number":120,"context_line":"* Add option \u0027share_pci_between_numa_nodes\u0027 to Nova config"},{"line_number":121,"context_line":"* Use this opting to change scheduler behavior"},{"line_number":122,"context_line":""},{"line_number":123,"context_line":"Dependencies"},{"line_number":124,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":1,"id":"fa7ab95a_3460a135","line":121,"updated":"2016-08-26 13:12:06.000000000","message":"Need user doc update.","commit_id":"afb631e8a2f0e42a56ef0788fe1341f18f9b0035"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"24e1091f38805be9ab3bff088b3172786049b910","unresolved":false,"context_lines":[{"line_number":118,"context_line":"----------"},{"line_number":119,"context_line":""},{"line_number":120,"context_line":"* Add option \u0027share_pci_between_numa_nodes\u0027 to Nova config"},{"line_number":121,"context_line":"* Use this opting to change scheduler behavior"},{"line_number":122,"context_line":""},{"line_number":123,"context_line":"Dependencies"},{"line_number":124,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":1,"id":"fa7ab95a_af6d3a34","line":121,"in_reply_to":"fa7ab95a_3460a135","updated":"2016-08-26 13:31:43.000000000","message":"sure","commit_id":"afb631e8a2f0e42a56ef0788fe1341f18f9b0035"},{"author":{"_account_id":6772,"name":"Stephen Gordon","email":"sgordon@redhat.com","username":"sgordon"},"change_message_id":"da161939295330eddf97b7a0564e798c0f72b929","unresolved":false,"context_lines":[{"line_number":47,"context_line":"values:"},{"line_number":48,"context_line":""},{"line_number":49,"context_line":" * associated"},{"line_number":50,"context_line":" * shared"},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"This spec will be used by scheduler\u0027s NUMATopologyFilter which will"},{"line_number":53,"context_line":"decide on which NUMA nodes VM can be booted. Default value of this option"}],"source_content_type":"text/x-rst","patch_set":2,"id":"fa7ab95a_9b6a27b2","line":50,"updated":"2016-08-29 10:57:01.000000000","message":"I don\u0027t think these values really reflect the meaning very clear, I suspect \"soft\" and \"hard\" might be simpler and more accurately convey the meaning.","commit_id":"be17997e8ce1de060464b06cc562cd9be0d72936"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"8e6a1db1f050439cca6d1dce3c97264c66c778e0","unresolved":false,"context_lines":[{"line_number":47,"context_line":"values:"},{"line_number":48,"context_line":""},{"line_number":49,"context_line":" * associated"},{"line_number":50,"context_line":" * shared"},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"This spec will be used by scheduler\u0027s NUMATopologyFilter which will"},{"line_number":53,"context_line":"decide on which NUMA nodes VM can be booted. Default value of this option"}],"source_content_type":"text/x-rst","patch_set":2,"id":"fa7ab95a_e76ab080","line":50,"in_reply_to":"fa7ab95a_9b6a27b2","updated":"2016-08-29 12:10:28.000000000","message":"ok","commit_id":"be17997e8ce1de060464b06cc562cd9be0d72936"},{"author":{"_account_id":6772,"name":"Stephen Gordon","email":"sgordon@redhat.com","username":"sgordon"},"change_message_id":"da161939295330eddf97b7a0564e798c0f72b929","unresolved":false,"context_lines":[{"line_number":57,"context_line":"The \u0027shared\u0027 value will mean that scheduler firstly will try to choose a NUMA"},{"line_number":58,"context_line":"node associated with requested PCI device but if it failed, scheduler will try"},{"line_number":59,"context_line":"to choose NUMA node again ignoring association of NUMA nodes and PCI devices."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"Alternatives"},{"line_number":62,"context_line":"------------"},{"line_number":63,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"fa7ab95a_5bf17f00","line":60,"updated":"2016-08-29 10:57:01.000000000","message":"In addition to ensure that \"associated\" (or whatever the language ends up being) has a higher chance of success I still think there is a need for the filter to weight requests that *don\u0027t* request a device towards NUMA nodes that don\u0027t have them.","commit_id":"be17997e8ce1de060464b06cc562cd9be0d72936"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"8e6a1db1f050439cca6d1dce3c97264c66c778e0","unresolved":false,"context_lines":[{"line_number":57,"context_line":"The \u0027shared\u0027 value will mean that scheduler firstly will try to choose a NUMA"},{"line_number":58,"context_line":"node associated with requested PCI device but if it failed, scheduler will try"},{"line_number":59,"context_line":"to choose NUMA node again ignoring association of NUMA nodes and PCI devices."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"Alternatives"},{"line_number":62,"context_line":"------------"},{"line_number":63,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"fa7ab95a_e79d5067","line":60,"in_reply_to":"fa7ab95a_5bf17f00","updated":"2016-08-29 12:10:28.000000000","message":"actually  \"associated\" has a *lower* chance of success because in this case nova will try to boot vms only on NUMA nodes associated with PCI devices.","commit_id":"be17997e8ce1de060464b06cc562cd9be0d72936"},{"author":{"_account_id":6772,"name":"Stephen Gordon","email":"sgordon@redhat.com","username":"sgordon"},"change_message_id":"8a8bf9c533505a5b7eef17f4bae575958853ce97","unresolved":false,"context_lines":[{"line_number":57,"context_line":"The \u0027shared\u0027 value will mean that scheduler firstly will try to choose a NUMA"},{"line_number":58,"context_line":"node associated with requested PCI device but if it failed, scheduler will try"},{"line_number":59,"context_line":"to choose NUMA node again ignoring association of NUMA nodes and PCI devices."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"Alternatives"},{"line_number":62,"context_line":"------------"},{"line_number":63,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"fa7ab95a_2a0e91e5","line":60,"in_reply_to":"fa7ab95a_e79d5067","updated":"2016-08-31 09:28:38.000000000","message":"I think you missed my point, by weighting requests that *don\u0027t* require PCIe devices at all towards nodes that don\u0027t have them you implicitly would increase the chances of a request for an \"associated\" PCIe device to succeed.","commit_id":"be17997e8ce1de060464b06cc562cd9be0d72936"},{"author":{"_account_id":6772,"name":"Stephen Gordon","email":"sgordon@redhat.com","username":"sgordon"},"change_message_id":"da161939295330eddf97b7a0564e798c0f72b929","unresolved":false,"context_lines":[{"line_number":123,"context_line":"Work Items"},{"line_number":124,"context_line":"----------"},{"line_number":125,"context_line":""},{"line_number":126,"context_line":"* Add option \u0027share_pci_between_numa_nodes\u0027 to Nova config"},{"line_number":127,"context_line":"* Use this opting to change scheduler behavior"},{"line_number":128,"context_line":"* Update user docs"},{"line_number":129,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"fa7ab95a_9bc6a798","line":126,"updated":"2016-08-29 10:57:01.000000000","message":"This looks like it\u0027s left over from an earlier patchset.","commit_id":"be17997e8ce1de060464b06cc562cd9be0d72936"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"8e6a1db1f050439cca6d1dce3c97264c66c778e0","unresolved":false,"context_lines":[{"line_number":123,"context_line":"Work Items"},{"line_number":124,"context_line":"----------"},{"line_number":125,"context_line":""},{"line_number":126,"context_line":"* Add option \u0027share_pci_between_numa_nodes\u0027 to Nova config"},{"line_number":127,"context_line":"* Use this opting to change scheduler behavior"},{"line_number":128,"context_line":"* Update user docs"},{"line_number":129,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"fa7ab95a_478fdc29","line":126,"in_reply_to":"fa7ab95a_9bc6a798","updated":"2016-08-29 12:10:28.000000000","message":"yep","commit_id":"be17997e8ce1de060464b06cc562cd9be0d72936"},{"author":{"_account_id":17973,"name":"Ludovic Beliveau","email":"lbeliveau@gmail.com","username":"lbeliveau"},"change_message_id":"a30c1260cad76c250489fa942847be7924fed9e0","unresolved":false,"context_lines":[{"line_number":14,"context_line":"scheduling algorithm allows to boot VM with PCI devices *only* on NUMA nodes"},{"line_number":15,"context_line":"associated with these PCI devices. Before that nova booted VMs with PCI devices"},{"line_number":16,"context_line":"without checking NUMA nodes. But in some cases such hardcoded behaviour makes"},{"line_number":17,"context_line":"problems."},{"line_number":18,"context_line":""},{"line_number":19,"context_line":"Problem description"},{"line_number":20,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":3,"id":"fa7ab95a_0d2e87fa","line":17,"updated":"2016-08-30 12:47:20.000000000","message":"I would maybe change this last sentence and explain why this can cause problems (mainly PCI resources that can be assigned to guest).","commit_id":"764703ac3a017e7eb5d548a84fcc10413d771553"},{"author":{"_account_id":17973,"name":"Ludovic Beliveau","email":"lbeliveau@gmail.com","username":"lbeliveau"},"change_message_id":"a30c1260cad76c250489fa942847be7924fed9e0","unresolved":false,"context_lines":[{"line_number":27,"context_line":"example SR-IOV card associated with first NUMA node) he or she would be able to"},{"line_number":28,"context_line":"boot VMs with SR-IOV ports only on the first NUMA node. In this case user"},{"line_number":29,"context_line":"cannot use half of CPUs and half of RAM because these resources are placed on"},{"line_number":30,"context_line":"second NUMA node. Half of compute host is unused."},{"line_number":31,"context_line":""},{"line_number":32,"context_line":"User should be able to choose scheduling behaviour. In described case user"},{"line_number":33,"context_line":"should be able to boot VMs on different NUMA nodes, even if it makes"}],"source_content_type":"text/x-rst","patch_set":3,"id":"fa7ab95a_ed4883b5","line":30,"updated":"2016-08-30 12:47:20.000000000","message":"s/ Half of compute host is unused/ Half of this compute host\u0027s PCI resources are unused/","commit_id":"764703ac3a017e7eb5d548a84fcc10413d771553"},{"author":{"_account_id":8768,"name":"Chris Friesen","email":"chris.friesen@windriver.com","username":"cbf123"},"change_message_id":"2790ad0348c5adace6126e699b434d907f05ac2c","unresolved":false,"context_lines":[{"line_number":46,"context_line":"Add new flavor extra spec \u0027hw:pci_and_numa_policy\u0027. It will has one of two"},{"line_number":47,"context_line":"values:"},{"line_number":48,"context_line":""},{"line_number":49,"context_line":" * hard"},{"line_number":50,"context_line":" * soft"},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"This spec will be used by scheduler\u0027s NUMATopologyFilter which will"},{"line_number":53,"context_line":"decide on which NUMA nodes VM can be booted. Default value of this option"}],"source_content_type":"text/x-rst","patch_set":3,"id":"fa7ab95a_0a75ca62","line":50,"range":{"start_line":49,"start_character":4,"end_line":50,"end_character":7},"updated":"2016-08-29 16:03:01.000000000","message":"Do we have similar flags for other config options?  If so we should ensure the naming is consistent.","commit_id":"764703ac3a017e7eb5d548a84fcc10413d771553"},{"author":{"_account_id":17973,"name":"Ludovic Beliveau","email":"lbeliveau@gmail.com","username":"lbeliveau"},"change_message_id":"a30c1260cad76c250489fa942847be7924fed9e0","unresolved":false,"context_lines":[{"line_number":46,"context_line":"Add new flavor extra spec \u0027hw:pci_and_numa_policy\u0027. It will has one of two"},{"line_number":47,"context_line":"values:"},{"line_number":48,"context_line":""},{"line_number":49,"context_line":" * hard"},{"line_number":50,"context_line":" * soft"},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"This spec will be used by scheduler\u0027s NUMATopologyFilter which will"},{"line_number":53,"context_line":"decide on which NUMA nodes VM can be booted. Default value of this option"}],"source_content_type":"text/x-rst","patch_set":3,"id":"fa7ab95a_ad759bcb","line":50,"range":{"start_line":49,"start_character":4,"end_line":50,"end_character":7},"in_reply_to":"fa7ab95a_0a75ca62","updated":"2016-08-30 12:47:20.000000000","message":"Hard and soft could be confusing.  Maybe strict and besteffort ?","commit_id":"764703ac3a017e7eb5d548a84fcc10413d771553"},{"author":{"_account_id":8768,"name":"Chris Friesen","email":"chris.friesen@windriver.com","username":"cbf123"},"change_message_id":"bcc0f218e9a8dc72cd7ae5936dc1d41d5bca31e6","unresolved":false,"context_lines":[{"line_number":46,"context_line":"Add new flavor extra spec \u0027hw:pci_and_numa_policy\u0027. It will has one of two"},{"line_number":47,"context_line":"values:"},{"line_number":48,"context_line":""},{"line_number":49,"context_line":" * hard"},{"line_number":50,"context_line":" * soft"},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"This spec will be used by scheduler\u0027s NUMATopologyFilter which will"},{"line_number":53,"context_line":"decide on which NUMA nodes VM can be booted. Default value of this option"}],"source_content_type":"text/x-rst","patch_set":3,"id":"fa7ab95a_f4755932","line":50,"range":{"start_line":49,"start_character":4,"end_line":50,"end_character":7},"in_reply_to":"fa7ab95a_6a2419c6","updated":"2016-08-31 14:29:12.000000000","message":"I think the \"shared/dedicated\" terms are suboptimal, they don\u0027t really make any sense from the point of view of the instance.\n\nWe do have the \"soft-affinity\" server group policy, and the soft reboot of the instance, so there\u0027s some precedent for the soft/hard terminology.\n\nIf we\u0027re making changes, I think \"hw:pci_numa_affinity\" would make more sense than the proposed \"hw:pci_and_numa_policy\".  Then we could maybe follow the example of the CPU thread allocation policy and use \"prefer\" and \"require\" to indicate whether numa affinity is preferred (ie soft) or required (ie hard).","commit_id":"764703ac3a017e7eb5d548a84fcc10413d771553"},{"author":{"_account_id":6772,"name":"Stephen Gordon","email":"sgordon@redhat.com","username":"sgordon"},"change_message_id":"5e9d1eaf0fb6b4f232f17254a5c2d975540fee07","unresolved":false,"context_lines":[{"line_number":46,"context_line":"Add new flavor extra spec \u0027hw:pci_and_numa_policy\u0027. It will has one of two"},{"line_number":47,"context_line":"values:"},{"line_number":48,"context_line":""},{"line_number":49,"context_line":" * hard"},{"line_number":50,"context_line":" * soft"},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"This spec will be used by scheduler\u0027s NUMATopologyFilter which will"},{"line_number":53,"context_line":"decide on which NUMA nodes VM can be booted. Default value of this option"}],"source_content_type":"text/x-rst","patch_set":3,"id":"fa7ab95a_6a2419c6","line":50,"range":{"start_line":49,"start_character":4,"end_line":50,"end_character":7},"in_reply_to":"fa7ab95a_ad759bcb","updated":"2016-08-31 09:40:24.000000000","message":"I don\u0027t think strict and besteffort are any \"less bad\" than soft and hard (terms which align with how we handle similar logic for the server group policy names) nor what was originally here. The only other example I can think of are the values for hw:cpu_policy which are shared and dedicated. This might also be appropriate here?","commit_id":"764703ac3a017e7eb5d548a84fcc10413d771553"},{"author":{"_account_id":6772,"name":"Stephen Gordon","email":"sgordon@redhat.com","username":"sgordon"},"change_message_id":"395645a85a219d98b71f10595865f40e5f39e7b5","unresolved":false,"context_lines":[{"line_number":46,"context_line":"Add new flavor extra spec \u0027hw:pci_and_numa_policy\u0027. It will has one of two"},{"line_number":47,"context_line":"values:"},{"line_number":48,"context_line":""},{"line_number":49,"context_line":" * hard"},{"line_number":50,"context_line":" * soft"},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"This spec will be used by scheduler\u0027s NUMATopologyFilter which will"},{"line_number":53,"context_line":"decide on which NUMA nodes VM can be booted. Default value of this option"}],"source_content_type":"text/x-rst","patch_set":3,"id":"fa7ab95a_da76703d","line":50,"range":{"start_line":49,"start_character":4,"end_line":50,"end_character":7},"in_reply_to":"fa7ab95a_f4755932","updated":"2016-09-01 15:40:25.000000000","message":"Yes, that approach makes sense to me.","commit_id":"764703ac3a017e7eb5d548a84fcc10413d771553"},{"author":{"_account_id":17973,"name":"Ludovic Beliveau","email":"lbeliveau@gmail.com","username":"lbeliveau"},"change_message_id":"a30c1260cad76c250489fa942847be7924fed9e0","unresolved":false,"context_lines":[{"line_number":57,"context_line":"The \u0027soft\u0027 value will mean that scheduler firstly will try to choose a NUMA"},{"line_number":58,"context_line":"node associated with requested PCI device but if it failed, scheduler will try"},{"line_number":59,"context_line":"to choose NUMA node again ignoring association of NUMA nodes and PCI devices."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"Alternatives"},{"line_number":62,"context_line":"------------"},{"line_number":63,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"fa7ab95a_ad0afb23","line":60,"updated":"2016-08-30 12:47:20.000000000","message":"I would go in greater length at how this could be implemented so that effort is well understood.","commit_id":"764703ac3a017e7eb5d548a84fcc10413d771553"},{"author":{"_account_id":17973,"name":"Ludovic Beliveau","email":"lbeliveau@gmail.com","username":"lbeliveau"},"change_message_id":"a30c1260cad76c250489fa942847be7924fed9e0","unresolved":false,"context_lines":[{"line_number":124,"context_line":"----------"},{"line_number":125,"context_line":""},{"line_number":126,"context_line":"* Add new spec to the flavor"},{"line_number":127,"context_line":"* Use this spec to change scheduler behavior"},{"line_number":128,"context_line":"* Update user docs"},{"line_number":129,"context_line":""},{"line_number":130,"context_line":"Dependencies"}],"source_content_type":"text/x-rst","patch_set":3,"id":"fa7ab95a_3f29b28a","line":127,"updated":"2016-08-30 12:47:20.000000000","message":"PciTracker might also requires changes for consuming the request.","commit_id":"764703ac3a017e7eb5d548a84fcc10413d771553"},{"author":{"_account_id":7461,"name":"Paul Murray","email":"pmurray@hpe.com","username":"ptm"},"change_message_id":"7b14f6f6d52f3dbacfe61af846c0511732546b9d","unresolved":false,"context_lines":[{"line_number":11,"context_line":"https://blueprints.launchpad.net/nova/+spec/share-pci-between-numa-nodes"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"In Juno release \"I/O based NUMA scheduling\" spec was implemented [1]. This"},{"line_number":14,"context_line":"scheduling algorithm allows to boot VM with PCI devices *only* on NUMA nodes"},{"line_number":15,"context_line":"associated with these PCI devices. Before that nova booted VMs with PCI devices"},{"line_number":16,"context_line":"without checking NUMA nodes. But such hardcoded behaviour makes problems if"},{"line_number":17,"context_line":"not each NUMA node has its own PCI device. In this case Nova wouldn\u0027t allow to"},{"line_number":18,"context_line":"boot a VM on NUMA nodes without PCI devices."}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa7ab95a_b0bb560e","line":15,"range":{"start_line":14,"start_character":56,"end_line":15,"end_character":33},"updated":"2016-09-02 14:27:03.000000000","message":"I think it actually insists that the pci devices are associated with *one* of the VMs numa nodes. E.g.: if a VM has two numa nodes and one pci device, one of the numa nodes has to be co-located with the pci device; the other numa node can go anywhere.","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"491bb0a497f6380241c4c82dca6b7b0703172cdc","unresolved":false,"context_lines":[{"line_number":11,"context_line":"https://blueprints.launchpad.net/nova/+spec/share-pci-between-numa-nodes"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"In Juno release \"I/O based NUMA scheduling\" spec was implemented [1]. This"},{"line_number":14,"context_line":"scheduling algorithm allows to boot VM with PCI devices *only* on NUMA nodes"},{"line_number":15,"context_line":"associated with these PCI devices. Before that nova booted VMs with PCI devices"},{"line_number":16,"context_line":"without checking NUMA nodes. But such hardcoded behaviour makes problems if"},{"line_number":17,"context_line":"not each NUMA node has its own PCI device. In this case Nova wouldn\u0027t allow to"},{"line_number":18,"context_line":"boot a VM on NUMA nodes without PCI devices."}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a8ec9b2_842bf8ea","line":15,"range":{"start_line":14,"start_character":56,"end_line":15,"end_character":33},"in_reply_to":"fa7ab95a_b0bb560e","updated":"2016-09-14 14:20:20.000000000","message":"Done","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"f5457a10f36dcc699f41c4c51900db5973b1f749","unresolved":false,"context_lines":[{"line_number":13,"context_line":"In Juno release \"I/O based NUMA scheduling\" spec was implemented [1]. This"},{"line_number":14,"context_line":"scheduling algorithm allows to boot VM with PCI devices *only* on NUMA nodes"},{"line_number":15,"context_line":"associated with these PCI devices. Before that nova booted VMs with PCI devices"},{"line_number":16,"context_line":"without checking NUMA nodes. But such hardcoded behaviour makes problems if"},{"line_number":17,"context_line":"not each NUMA node has its own PCI device. In this case Nova wouldn\u0027t allow to"},{"line_number":18,"context_line":"boot a VM on NUMA nodes without PCI devices."},{"line_number":19,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa7ab95a_d29c946c","line":16,"range":{"start_line":16,"start_character":29,"end_line":16,"end_character":32},"updated":"2016-09-01 16:28:24.000000000","message":"However,","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"f5457a10f36dcc699f41c4c51900db5973b1f749","unresolved":false,"context_lines":[{"line_number":13,"context_line":"In Juno release \"I/O based NUMA scheduling\" spec was implemented [1]. This"},{"line_number":14,"context_line":"scheduling algorithm allows to boot VM with PCI devices *only* on NUMA nodes"},{"line_number":15,"context_line":"associated with these PCI devices. Before that nova booted VMs with PCI devices"},{"line_number":16,"context_line":"without checking NUMA nodes. But such hardcoded behaviour makes problems if"},{"line_number":17,"context_line":"not each NUMA node has its own PCI device. In this case Nova wouldn\u0027t allow to"},{"line_number":18,"context_line":"boot a VM on NUMA nodes without PCI devices."},{"line_number":19,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa7ab95a_32c06050","line":16,"range":{"start_line":16,"start_character":58,"end_line":16,"end_character":63},"updated":"2016-09-01 16:28:24.000000000","message":"causes","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"491bb0a497f6380241c4c82dca6b7b0703172cdc","unresolved":false,"context_lines":[{"line_number":13,"context_line":"In Juno release \"I/O based NUMA scheduling\" spec was implemented [1]. This"},{"line_number":14,"context_line":"scheduling algorithm allows to boot VM with PCI devices *only* on NUMA nodes"},{"line_number":15,"context_line":"associated with these PCI devices. Before that nova booted VMs with PCI devices"},{"line_number":16,"context_line":"without checking NUMA nodes. But such hardcoded behaviour makes problems if"},{"line_number":17,"context_line":"not each NUMA node has its own PCI device. In this case Nova wouldn\u0027t allow to"},{"line_number":18,"context_line":"boot a VM on NUMA nodes without PCI devices."},{"line_number":19,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a8ec9b2_c4350047","line":16,"range":{"start_line":16,"start_character":58,"end_line":16,"end_character":63},"in_reply_to":"fa7ab95a_32c06050","updated":"2016-09-14 14:20:20.000000000","message":"Done","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"f5457a10f36dcc699f41c4c51900db5973b1f749","unresolved":false,"context_lines":[{"line_number":14,"context_line":"scheduling algorithm allows to boot VM with PCI devices *only* on NUMA nodes"},{"line_number":15,"context_line":"associated with these PCI devices. Before that nova booted VMs with PCI devices"},{"line_number":16,"context_line":"without checking NUMA nodes. But such hardcoded behaviour makes problems if"},{"line_number":17,"context_line":"not each NUMA node has its own PCI device. In this case Nova wouldn\u0027t allow to"},{"line_number":18,"context_line":"boot a VM on NUMA nodes without PCI devices."},{"line_number":19,"context_line":""},{"line_number":20,"context_line":"Problem description"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa7ab95a_d2c5745c","line":17,"range":{"start_line":17,"start_character":4,"end_line":17,"end_character":8},"updated":"2016-09-01 16:28:24.000000000","message":"every","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"491bb0a497f6380241c4c82dca6b7b0703172cdc","unresolved":false,"context_lines":[{"line_number":14,"context_line":"scheduling algorithm allows to boot VM with PCI devices *only* on NUMA nodes"},{"line_number":15,"context_line":"associated with these PCI devices. Before that nova booted VMs with PCI devices"},{"line_number":16,"context_line":"without checking NUMA nodes. But such hardcoded behaviour makes problems if"},{"line_number":17,"context_line":"not each NUMA node has its own PCI device. In this case Nova wouldn\u0027t allow to"},{"line_number":18,"context_line":"boot a VM on NUMA nodes without PCI devices."},{"line_number":19,"context_line":""},{"line_number":20,"context_line":"Problem description"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a8ec9b2_e43afc35","line":17,"range":{"start_line":17,"start_character":4,"end_line":17,"end_character":8},"in_reply_to":"fa7ab95a_d2c5745c","updated":"2016-09-14 14:20:20.000000000","message":"Done","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":7461,"name":"Paul Murray","email":"pmurray@hpe.com","username":"ptm"},"change_message_id":"7b14f6f6d52f3dbacfe61af846c0511732546b9d","unresolved":false,"context_lines":[{"line_number":20,"context_line":"Problem description"},{"line_number":21,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":22,"context_line":""},{"line_number":23,"context_line":"Booting VMs with PCI devices *only* on NUMA nodes associated with these PCI"},{"line_number":24,"context_line":"devices is good for performance. This ensures there is no cross NUMA node"},{"line_number":25,"context_line":"memory traffic."},{"line_number":26,"context_line":""},{"line_number":27,"context_line":"But if a user has environment with 2 NUMA nodes and only 1 PCI device (for"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa7ab95a_b0c19631","line":24,"range":{"start_line":23,"start_character":29,"end_line":24,"end_character":7},"updated":"2016-09-02 14:27:03.000000000","message":"Same as above: \"Booting VMs with PCI devices only if at least one of the NUMA nodes is associated with the PCI devices...\"","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"491bb0a497f6380241c4c82dca6b7b0703172cdc","unresolved":false,"context_lines":[{"line_number":20,"context_line":"Problem description"},{"line_number":21,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":22,"context_line":""},{"line_number":23,"context_line":"Booting VMs with PCI devices *only* on NUMA nodes associated with these PCI"},{"line_number":24,"context_line":"devices is good for performance. This ensures there is no cross NUMA node"},{"line_number":25,"context_line":"memory traffic."},{"line_number":26,"context_line":""},{"line_number":27,"context_line":"But if a user has environment with 2 NUMA nodes and only 1 PCI device (for"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a8ec9b2_444ad0c6","line":24,"range":{"start_line":23,"start_character":29,"end_line":24,"end_character":7},"in_reply_to":"fa7ab95a_b0c19631","updated":"2016-09-14 14:20:20.000000000","message":"Done","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":7461,"name":"Paul Murray","email":"pmurray@hpe.com","username":"ptm"},"change_message_id":"7b14f6f6d52f3dbacfe61af846c0511732546b9d","unresolved":false,"context_lines":[{"line_number":24,"context_line":"devices is good for performance. This ensures there is no cross NUMA node"},{"line_number":25,"context_line":"memory traffic."},{"line_number":26,"context_line":""},{"line_number":27,"context_line":"But if a user has environment with 2 NUMA nodes and only 1 PCI device (for"},{"line_number":28,"context_line":"example SR-IOV card associated with first NUMA node) he or she would be able to"},{"line_number":29,"context_line":"boot VMs with SR-IOV ports only on the first NUMA node. In this case user"},{"line_number":30,"context_line":"cannot use half of CPUs and half of RAM because these resources are placed on"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa7ab95a_f0795efd","line":27,"range":{"start_line":27,"start_character":18,"end_line":27,"end_character":69},"updated":"2016-09-02 14:27:03.000000000","message":"Same as above - booting VMs with two numa nodes and 1 pci device works well in this setup.","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"491bb0a497f6380241c4c82dca6b7b0703172cdc","unresolved":false,"context_lines":[{"line_number":24,"context_line":"devices is good for performance. This ensures there is no cross NUMA node"},{"line_number":25,"context_line":"memory traffic."},{"line_number":26,"context_line":""},{"line_number":27,"context_line":"But if a user has environment with 2 NUMA nodes and only 1 PCI device (for"},{"line_number":28,"context_line":"example SR-IOV card associated with first NUMA node) he or she would be able to"},{"line_number":29,"context_line":"boot VMs with SR-IOV ports only on the first NUMA node. In this case user"},{"line_number":30,"context_line":"cannot use half of CPUs and half of RAM because these resources are placed on"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a8ec9b2_a4363401","line":27,"range":{"start_line":27,"start_character":18,"end_line":27,"end_character":69},"in_reply_to":"fa7ab95a_f0795efd","updated":"2016-09-14 14:20:20.000000000","message":"Done","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"f5457a10f36dcc699f41c4c51900db5973b1f749","unresolved":false,"context_lines":[{"line_number":38,"context_line":"Use Cases"},{"line_number":39,"context_line":"---------"},{"line_number":40,"context_line":""},{"line_number":41,"context_line":"Users with environments in which number of PCI devices is less than NUMA nodes"},{"line_number":42,"context_line":"will be able to use all of their hardware resources (CPUs, RAM)."},{"line_number":43,"context_line":""},{"line_number":44,"context_line":"Proposed change"},{"line_number":45,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa7ab95a_75a7320b","line":42,"range":{"start_line":41,"start_character":0,"end_line":42,"end_character":64},"updated":"2016-09-01 16:28:24.000000000","message":"Not quite. You could have infinite PCI devices and two nodes, but if all these devices are in slots affinitized to node 0 then scheduling will still fail. Try something like:\n\n    Users with hosts where PCI devices are divided unequally\n    between different NUMA nodes.","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"491bb0a497f6380241c4c82dca6b7b0703172cdc","unresolved":false,"context_lines":[{"line_number":38,"context_line":"Use Cases"},{"line_number":39,"context_line":"---------"},{"line_number":40,"context_line":""},{"line_number":41,"context_line":"Users with environments in which number of PCI devices is less than NUMA nodes"},{"line_number":42,"context_line":"will be able to use all of their hardware resources (CPUs, RAM)."},{"line_number":43,"context_line":""},{"line_number":44,"context_line":"Proposed change"},{"line_number":45,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a8ec9b2_7fff5901","line":42,"range":{"start_line":41,"start_character":0,"end_line":42,"end_character":64},"in_reply_to":"fa7ab95a_75a7320b","updated":"2016-09-14 14:20:20.000000000","message":"Done","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"c9333295807678587c15708a92aa2db0fcd5053f","unresolved":false,"context_lines":[{"line_number":44,"context_line":"Proposed change"},{"line_number":45,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":46,"context_line":""},{"line_number":47,"context_line":"Add new flavor extra spec \u0027hw:pci_numa_affinity\u0027. It will has one of two"},{"line_number":48,"context_line":"values:"},{"line_number":49,"context_line":""},{"line_number":50,"context_line":" * require"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa7ab95a_d5c80177","line":47,"range":{"start_line":47,"start_character":30,"end_line":47,"end_character":47},"updated":"2016-09-01 15:56:52.000000000","message":"May I suggest \u0027hw:pci_numa_policy\u0027? This is consistent with the \u0027cpu_thread_policy\u0027 extra spec (and, less importantly, the \u0027cpu_policy\u0027 extra spec)","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"c9333295807678587c15708a92aa2db0fcd5053f","unresolved":false,"context_lines":[{"line_number":44,"context_line":"Proposed change"},{"line_number":45,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":46,"context_line":""},{"line_number":47,"context_line":"Add new flavor extra spec \u0027hw:pci_numa_affinity\u0027. It will has one of two"},{"line_number":48,"context_line":"values:"},{"line_number":49,"context_line":""},{"line_number":50,"context_line":" * require"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa7ab95a_956d9934","line":47,"range":{"start_line":47,"start_character":1,"end_line":47,"end_character":25},"updated":"2016-09-01 15:56:52.000000000","message":"What about image metadata?","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"491bb0a497f6380241c4c82dca6b7b0703172cdc","unresolved":false,"context_lines":[{"line_number":44,"context_line":"Proposed change"},{"line_number":45,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":46,"context_line":""},{"line_number":47,"context_line":"Add new flavor extra spec \u0027hw:pci_numa_affinity\u0027. It will has one of two"},{"line_number":48,"context_line":"values:"},{"line_number":49,"context_line":""},{"line_number":50,"context_line":" * require"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a8ec9b2_0ae8d1e1","line":47,"range":{"start_line":47,"start_character":1,"end_line":47,"end_character":25},"in_reply_to":"fa7ab95a_05892226","updated":"2016-09-14 14:20:20.000000000","message":"Done","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":17973,"name":"Ludovic Beliveau","email":"lbeliveau@gmail.com","username":"lbeliveau"},"change_message_id":"8537479e0e99fe338115b27fdbadaee97a88af06","unresolved":false,"context_lines":[{"line_number":44,"context_line":"Proposed change"},{"line_number":45,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":46,"context_line":""},{"line_number":47,"context_line":"Add new flavor extra spec \u0027hw:pci_numa_affinity\u0027. It will has one of two"},{"line_number":48,"context_line":"values:"},{"line_number":49,"context_line":""},{"line_number":50,"context_line":" * require"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa7ab95a_b2b1c591","line":47,"range":{"start_line":47,"start_character":1,"end_line":47,"end_character":25},"in_reply_to":"fa7ab95a_956d9934","updated":"2016-09-01 20:12:12.000000000","message":"That`s a possibility, maybe it should be mentioned in the alternatives.  Personally, flavor has finer grained control over such a policy.  Problem with image metadata is that an operator might end up with multiple identical images but with different metadata.","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"c2d2d5608e03196805710133320f06ae439a2734","unresolved":false,"context_lines":[{"line_number":44,"context_line":"Proposed change"},{"line_number":45,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":46,"context_line":""},{"line_number":47,"context_line":"Add new flavor extra spec \u0027hw:pci_numa_affinity\u0027. It will has one of two"},{"line_number":48,"context_line":"values:"},{"line_number":49,"context_line":""},{"line_number":50,"context_line":" * require"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa7ab95a_fd406761","line":47,"range":{"start_line":47,"start_character":1,"end_line":47,"end_character":25},"in_reply_to":"fa7ab95a_b2b1c591","updated":"2016-09-02 09:47:49.000000000","message":"\u003e That\u0027s a possibility, maybe it should be mentioned in the alternatives.  Personally, flavor has finer grained control over such a policy.  Problem with image metadata is that an operator might end up with multiple identical images but with different metadata.\n\nIt\u0027s not so much an alternative as an extension. Both cpu_policy and cpu_thread_policy support definition by both flavor extra specs and image metadata. The latter takes precedence in the case of a conflict. We should probably do the same here.","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":7461,"name":"Paul Murray","email":"pmurray@hpe.com","username":"ptm"},"change_message_id":"7b14f6f6d52f3dbacfe61af846c0511732546b9d","unresolved":false,"context_lines":[{"line_number":44,"context_line":"Proposed change"},{"line_number":45,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":46,"context_line":""},{"line_number":47,"context_line":"Add new flavor extra spec \u0027hw:pci_numa_affinity\u0027. It will has one of two"},{"line_number":48,"context_line":"values:"},{"line_number":49,"context_line":""},{"line_number":50,"context_line":" * require"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa7ab95a_05892226","line":47,"range":{"start_line":47,"start_character":1,"end_line":47,"end_character":25},"in_reply_to":"fa7ab95a_bfcc0c36","updated":"2016-09-02 14:27:03.000000000","message":"Agree on image properties. pci_numa_policy is ok.","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":8768,"name":"Chris Friesen","email":"chris.friesen@windriver.com","username":"cbf123"},"change_message_id":"040b7bfff8b7ca18367ce7640d194fa8a79d5255","unresolved":false,"context_lines":[{"line_number":44,"context_line":"Proposed change"},{"line_number":45,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":46,"context_line":""},{"line_number":47,"context_line":"Add new flavor extra spec \u0027hw:pci_numa_affinity\u0027. It will has one of two"},{"line_number":48,"context_line":"values:"},{"line_number":49,"context_line":""},{"line_number":50,"context_line":" * require"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa7ab95a_cd694ac0","line":47,"range":{"start_line":47,"start_character":30,"end_line":47,"end_character":47},"in_reply_to":"fa7ab95a_d5c80177","updated":"2016-09-01 20:04:40.000000000","message":"I\u0027m fine with that.","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":17973,"name":"Ludovic Beliveau","email":"lbeliveau@gmail.com","username":"lbeliveau"},"change_message_id":"de8053333402949e351917dc4c4d6f0e0d507070","unresolved":false,"context_lines":[{"line_number":44,"context_line":"Proposed change"},{"line_number":45,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":46,"context_line":""},{"line_number":47,"context_line":"Add new flavor extra spec \u0027hw:pci_numa_affinity\u0027. It will has one of two"},{"line_number":48,"context_line":"values:"},{"line_number":49,"context_line":""},{"line_number":50,"context_line":" * require"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa7ab95a_bfcc0c36","line":47,"range":{"start_line":47,"start_character":1,"end_line":47,"end_character":25},"in_reply_to":"fa7ab95a_fd406761","updated":"2016-09-02 12:27:16.000000000","message":"Good point.  Agreed.","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"c9333295807678587c15708a92aa2db0fcd5053f","unresolved":false,"context_lines":[{"line_number":47,"context_line":"Add new flavor extra spec \u0027hw:pci_numa_affinity\u0027. It will has one of two"},{"line_number":48,"context_line":"values:"},{"line_number":49,"context_line":""},{"line_number":50,"context_line":" * require"},{"line_number":51,"context_line":" * prefer"},{"line_number":52,"context_line":""},{"line_number":53,"context_line":"This spec will be used by to decide on which NUMA nodes VM can be booted."},{"line_number":54,"context_line":"Default value of this option will be \u0027require\u0027. It means that Nova will boot"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa7ab95a_95967992","line":51,"range":{"start_line":50,"start_character":0,"end_line":51,"end_character":9},"updated":"2016-09-01 15:56:52.000000000","message":"nit: Can we use \u0027required\u0027 and \u0027prefered\u0027? This is more consistent with things like \u0027cpu_policy\u0027 and \u0027mem_policy\u0027 (though not \u0027cpu_thread_policy\u0027 - my mistake :()","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"491bb0a497f6380241c4c82dca6b7b0703172cdc","unresolved":false,"context_lines":[{"line_number":47,"context_line":"Add new flavor extra spec \u0027hw:pci_numa_affinity\u0027. It will has one of two"},{"line_number":48,"context_line":"values:"},{"line_number":49,"context_line":""},{"line_number":50,"context_line":" * require"},{"line_number":51,"context_line":" * prefer"},{"line_number":52,"context_line":""},{"line_number":53,"context_line":"This spec will be used by to decide on which NUMA nodes VM can be booted."},{"line_number":54,"context_line":"Default value of this option will be \u0027require\u0027. It means that Nova will boot"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a8ec9b2_cadd0944","line":51,"range":{"start_line":50,"start_character":0,"end_line":51,"end_character":9},"in_reply_to":"fa7ab95a_4d9c5ac0","updated":"2016-09-14 14:20:20.000000000","message":"Done","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":8768,"name":"Chris Friesen","email":"chris.friesen@windriver.com","username":"cbf123"},"change_message_id":"040b7bfff8b7ca18367ce7640d194fa8a79d5255","unresolved":false,"context_lines":[{"line_number":47,"context_line":"Add new flavor extra spec \u0027hw:pci_numa_affinity\u0027. It will has one of two"},{"line_number":48,"context_line":"values:"},{"line_number":49,"context_line":""},{"line_number":50,"context_line":" * require"},{"line_number":51,"context_line":" * prefer"},{"line_number":52,"context_line":""},{"line_number":53,"context_line":"This spec will be used by to decide on which NUMA nodes VM can be booted."},{"line_number":54,"context_line":"Default value of this option will be \u0027require\u0027. It means that Nova will boot"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa7ab95a_4d9c5ac0","line":51,"range":{"start_line":50,"start_character":0,"end_line":51,"end_character":9},"in_reply_to":"fa7ab95a_95967992","updated":"2016-09-01 20:04:40.000000000","message":"agreed","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"c9333295807678587c15708a92aa2db0fcd5053f","unresolved":false,"context_lines":[{"line_number":51,"context_line":" * prefer"},{"line_number":52,"context_line":""},{"line_number":53,"context_line":"This spec will be used by to decide on which NUMA nodes VM can be booted."},{"line_number":54,"context_line":"Default value of this option will be \u0027require\u0027. It means that Nova will boot"},{"line_number":55,"context_line":"VMs with PCI devices *only* on NUMA nodes associated with these PCI devices"},{"line_number":56,"context_line":"(current behaviour)."},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"The \u0027prefer\u0027 value will mean that scheduler will choose a compute host"},{"line_number":59,"context_line":"ignoring association of NUMA nodes and PCI devices. To optimise performance of"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa7ab95a_b5f45dcf","line":56,"range":{"start_line":54,"start_character":48,"end_line":56,"end_character":20},"updated":"2016-09-01 15:56:52.000000000","message":"This isn\u0027t exactly true. As pointed out in the below spec, if a NUMA topology isn\u0027t provided then PCI devices are shared between all nodes.\n\n    https://review.openstack.org/#/c/334419/3/specs/ocata/approved/modify-pci-without-numa-node-info-treatment.rst\n\nUnless you plan to change the default behavior (please don\u0027t) then you should reword this or add an additional \u0027strict\u0027 policy","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"c2d2d5608e03196805710133320f06ae439a2734","unresolved":false,"context_lines":[{"line_number":51,"context_line":" * prefer"},{"line_number":52,"context_line":""},{"line_number":53,"context_line":"This spec will be used by to decide on which NUMA nodes VM can be booted."},{"line_number":54,"context_line":"Default value of this option will be \u0027require\u0027. It means that Nova will boot"},{"line_number":55,"context_line":"VMs with PCI devices *only* on NUMA nodes associated with these PCI devices"},{"line_number":56,"context_line":"(current behaviour)."},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"The \u0027prefer\u0027 value will mean that scheduler will choose a compute host"},{"line_number":59,"context_line":"ignoring association of NUMA nodes and PCI devices. To optimise performance of"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa7ab95a_881ab76d","line":56,"range":{"start_line":54,"start_character":48,"end_line":56,"end_character":20},"in_reply_to":"fa7ab95a_4dd45a1f","updated":"2016-09-02 09:47:49.000000000","message":"\u003e Currently, if you have one vcpu, nova will fail if the vpcu and the PCI devices are not co-located.\n\nNot quite. Some hosts do not report correct NUMA affinities for PCI devices. When this happens, PCI devices are shared across all NUMA nodes. It\u0027s rare but it does happen.","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":17973,"name":"Ludovic Beliveau","email":"lbeliveau@gmail.com","username":"lbeliveau"},"change_message_id":"de8053333402949e351917dc4c4d6f0e0d507070","unresolved":false,"context_lines":[{"line_number":51,"context_line":" * prefer"},{"line_number":52,"context_line":""},{"line_number":53,"context_line":"This spec will be used by to decide on which NUMA nodes VM can be booted."},{"line_number":54,"context_line":"Default value of this option will be \u0027require\u0027. It means that Nova will boot"},{"line_number":55,"context_line":"VMs with PCI devices *only* on NUMA nodes associated with these PCI devices"},{"line_number":56,"context_line":"(current behaviour)."},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"The \u0027prefer\u0027 value will mean that scheduler will choose a compute host"},{"line_number":59,"context_line":"ignoring association of NUMA nodes and PCI devices. To optimise performance of"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa7ab95a_df9b9015","line":56,"range":{"start_line":54,"start_character":48,"end_line":56,"end_character":20},"in_reply_to":"fa7ab95a_881ab76d","updated":"2016-09-02 12:27:16.000000000","message":"In this case, you are right.","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":17973,"name":"Ludovic Beliveau","email":"lbeliveau@gmail.com","username":"lbeliveau"},"change_message_id":"8537479e0e99fe338115b27fdbadaee97a88af06","unresolved":false,"context_lines":[{"line_number":51,"context_line":" * prefer"},{"line_number":52,"context_line":""},{"line_number":53,"context_line":"This spec will be used by to decide on which NUMA nodes VM can be booted."},{"line_number":54,"context_line":"Default value of this option will be \u0027require\u0027. It means that Nova will boot"},{"line_number":55,"context_line":"VMs with PCI devices *only* on NUMA nodes associated with these PCI devices"},{"line_number":56,"context_line":"(current behaviour)."},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"The \u0027prefer\u0027 value will mean that scheduler will choose a compute host"},{"line_number":59,"context_line":"ignoring association of NUMA nodes and PCI devices. To optimise performance of"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa7ab95a_4dd45a1f","line":56,"range":{"start_line":54,"start_character":48,"end_line":56,"end_character":20},"in_reply_to":"fa7ab95a_b5f45dcf","updated":"2016-09-01 20:12:12.000000000","message":"Currently, if you have one vcpu, nova will fail if the vpcu and the PCI devices are not co-located.\n\nUnless you have more than one vcpu, it just check that the PCI devices are sharing at least one of the numa node with the vcpu.\n\nSee https://github.com/openstack/nova/blob/master/nova/pci/stats.py#L214","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"491bb0a497f6380241c4c82dca6b7b0703172cdc","unresolved":false,"context_lines":[{"line_number":51,"context_line":" * prefer"},{"line_number":52,"context_line":""},{"line_number":53,"context_line":"This spec will be used by to decide on which NUMA nodes VM can be booted."},{"line_number":54,"context_line":"Default value of this option will be \u0027require\u0027. It means that Nova will boot"},{"line_number":55,"context_line":"VMs with PCI devices *only* on NUMA nodes associated with these PCI devices"},{"line_number":56,"context_line":"(current behaviour)."},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"The \u0027prefer\u0027 value will mean that scheduler will choose a compute host"},{"line_number":59,"context_line":"ignoring association of NUMA nodes and PCI devices. To optimise performance of"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a8ec9b2_05de026d","line":56,"range":{"start_line":54,"start_character":48,"end_line":56,"end_character":20},"in_reply_to":"fa7ab95a_df9b9015","updated":"2016-09-14 14:20:20.000000000","message":"Done","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"c9333295807678587c15708a92aa2db0fcd5053f","unresolved":false,"context_lines":[{"line_number":64,"context_line":"Alternatives"},{"line_number":65,"context_line":"------------"},{"line_number":66,"context_line":""},{"line_number":67,"context_line":"Left current scheduler behaviour and buy more PCI devices to give each NUMA"},{"line_number":68,"context_line":"node its own PCI device."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"Data model impact"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa7ab95a_d576413d","line":67,"range":{"start_line":67,"start_character":0,"end_line":67,"end_character":4},"updated":"2016-09-01 15:56:52.000000000","message":"Retain","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"491bb0a497f6380241c4c82dca6b7b0703172cdc","unresolved":false,"context_lines":[{"line_number":64,"context_line":"Alternatives"},{"line_number":65,"context_line":"------------"},{"line_number":66,"context_line":""},{"line_number":67,"context_line":"Left current scheduler behaviour and buy more PCI devices to give each NUMA"},{"line_number":68,"context_line":"node its own PCI device."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"Data model impact"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a8ec9b2_0adc51e3","line":67,"range":{"start_line":67,"start_character":0,"end_line":67,"end_character":4},"in_reply_to":"fa7ab95a_d576413d","updated":"2016-09-14 14:20:20.000000000","message":"Done","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"c9333295807678587c15708a92aa2db0fcd5053f","unresolved":false,"context_lines":[{"line_number":65,"context_line":"------------"},{"line_number":66,"context_line":""},{"line_number":67,"context_line":"Left current scheduler behaviour and buy more PCI devices to give each NUMA"},{"line_number":68,"context_line":"node its own PCI device."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"Data model impact"},{"line_number":71,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa7ab95a_b59e1d52","line":68,"updated":"2016-09-01 15:56:52.000000000","message":"You could also ensure that PCI devices are only attached to instances that absolutely need them: they are expensive, after all :)","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"c9333295807678587c15708a92aa2db0fcd5053f","unresolved":false,"context_lines":[{"line_number":70,"context_line":"Data model impact"},{"line_number":71,"context_line":"-----------------"},{"line_number":72,"context_line":""},{"line_number":73,"context_line":"New field \u0027pci_numa_affinity\u0027 will be added to the InstanceNUMACell object."},{"line_number":74,"context_line":"Fortunately this object is storing as JSON in DB. So we need no DB migrations"},{"line_number":75,"context_line":"to add new field to this object."},{"line_number":76,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa7ab95a_f5bfc5a7","line":73,"range":{"start_line":73,"start_character":11,"end_line":73,"end_character":28},"updated":"2016-09-01 15:56:52.000000000","message":"*pci_numa_policy?","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"491bb0a497f6380241c4c82dca6b7b0703172cdc","unresolved":false,"context_lines":[{"line_number":70,"context_line":"Data model impact"},{"line_number":71,"context_line":"-----------------"},{"line_number":72,"context_line":""},{"line_number":73,"context_line":"New field \u0027pci_numa_affinity\u0027 will be added to the InstanceNUMACell object."},{"line_number":74,"context_line":"Fortunately this object is storing as JSON in DB. So we need no DB migrations"},{"line_number":75,"context_line":"to add new field to this object."},{"line_number":76,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a8ec9b2_4aa019a0","line":73,"range":{"start_line":73,"start_character":11,"end_line":73,"end_character":28},"in_reply_to":"fa7ab95a_10f40259","updated":"2016-09-14 14:20:20.000000000","message":"lets name it as spec \"pci_numa_policy\"","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":7461,"name":"Paul Murray","email":"pmurray@hpe.com","username":"ptm"},"change_message_id":"7b14f6f6d52f3dbacfe61af846c0511732546b9d","unresolved":false,"context_lines":[{"line_number":70,"context_line":"Data model impact"},{"line_number":71,"context_line":"-----------------"},{"line_number":72,"context_line":""},{"line_number":73,"context_line":"New field \u0027pci_numa_affinity\u0027 will be added to the InstanceNUMACell object."},{"line_number":74,"context_line":"Fortunately this object is storing as JSON in DB. So we need no DB migrations"},{"line_number":75,"context_line":"to add new field to this object."},{"line_number":76,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa7ab95a_10f40259","line":73,"range":{"start_line":73,"start_character":11,"end_line":73,"end_character":28},"in_reply_to":"fa7ab95a_f5bfc5a7","updated":"2016-09-02 14:27:03.000000000","message":"This is the numa cell - we could call this pci_affinity_policy?","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":17973,"name":"Ludovic Beliveau","email":"lbeliveau@gmail.com","username":"lbeliveau"},"change_message_id":"6ed5e0df6212823bd90a941f52e89776035d387e","unresolved":false,"context_lines":[{"line_number":72,"context_line":""},{"line_number":73,"context_line":"New field \u0027pci_numa_affinity\u0027 will be added to the InstanceNUMACell object."},{"line_number":74,"context_line":"Fortunately this object is storing as JSON in DB. So we need no DB migrations"},{"line_number":75,"context_line":"to add new field to this object."},{"line_number":76,"context_line":""},{"line_number":77,"context_line":"REST API impact"},{"line_number":78,"context_line":"---------------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa7ab95a_a4f8b17d","line":75,"updated":"2016-09-01 15:13:01.000000000","message":"Not a bad idea to put the pci_numa_affinity in the InstanceNUMACell.  I\u0027m wondering if it would make more sense to put it in the InstancePCIRequest.  Not sure if the numa cell should be aware of this ...  What do you think ?","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"c9333295807678587c15708a92aa2db0fcd5053f","unresolved":false,"context_lines":[{"line_number":71,"context_line":"-----------------"},{"line_number":72,"context_line":""},{"line_number":73,"context_line":"New field \u0027pci_numa_affinity\u0027 will be added to the InstanceNUMACell object."},{"line_number":74,"context_line":"Fortunately this object is storing as JSON in DB. So we need no DB migrations"},{"line_number":75,"context_line":"to add new field to this object."},{"line_number":76,"context_line":""},{"line_number":77,"context_line":"REST API impact"},{"line_number":78,"context_line":"---------------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa7ab95a_151269b4","line":75,"range":{"start_line":74,"start_character":50,"end_line":75,"end_character":32},"updated":"2016-09-01 15:56:52.000000000","message":"sidenote: Yet! Jay will have something to say about this in Ocata, I would imagine.","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":7461,"name":"Paul Murray","email":"pmurray@hpe.com","username":"ptm"},"change_message_id":"7b14f6f6d52f3dbacfe61af846c0511732546b9d","unresolved":false,"context_lines":[{"line_number":71,"context_line":"-----------------"},{"line_number":72,"context_line":""},{"line_number":73,"context_line":"New field \u0027pci_numa_affinity\u0027 will be added to the InstanceNUMACell object."},{"line_number":74,"context_line":"Fortunately this object is storing as JSON in DB. So we need no DB migrations"},{"line_number":75,"context_line":"to add new field to this object."},{"line_number":76,"context_line":""},{"line_number":77,"context_line":"REST API impact"},{"line_number":78,"context_line":"---------------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa7ab95a_cbf66b5c","line":75,"range":{"start_line":74,"start_character":50,"end_line":75,"end_character":32},"in_reply_to":"fa7ab95a_151269b4","updated":"2016-09-02 14:27:03.000000000","message":"Putting the flag in InstanceNUMACell is a good idea. It is also consistent with cpu_policy and cpu_thread_policy. Even if all cells have the same value. Note also that InstanceNUMATopology has a property cpu_pinning_requested, you could do that if needed. I can imagine this being modified in the future to indicate that one particular numa cell should be close to a pci device (I\u0027m not suggesting that\u0027s needed for this spec though).","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"491bb0a497f6380241c4c82dca6b7b0703172cdc","unresolved":false,"context_lines":[{"line_number":72,"context_line":""},{"line_number":73,"context_line":"New field \u0027pci_numa_affinity\u0027 will be added to the InstanceNUMACell object."},{"line_number":74,"context_line":"Fortunately this object is storing as JSON in DB. So we need no DB migrations"},{"line_number":75,"context_line":"to add new field to this object."},{"line_number":76,"context_line":""},{"line_number":77,"context_line":"REST API impact"},{"line_number":78,"context_line":"---------------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a8ec9b2_0555224f","line":75,"in_reply_to":"fa7ab95a_a4f8b17d","updated":"2016-09-14 14:20:20.000000000","message":"I think adding it to InstancePCIRequest will cause duplication of information.","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"c9333295807678587c15708a92aa2db0fcd5053f","unresolved":false,"context_lines":[{"line_number":108,"context_line":"Other deployer impact"},{"line_number":109,"context_line":"---------------------"},{"line_number":110,"context_line":""},{"line_number":111,"context_line":"To use this feature the deployer must use HW that is capable of reporting"},{"line_number":112,"context_line":"NUMA related info to the OS."},{"line_number":113,"context_line":""},{"line_number":114,"context_line":"Developer impact"},{"line_number":115,"context_line":"----------------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa7ab95a_75d495bc","line":112,"range":{"start_line":111,"start_character":0,"end_line":112,"end_character":28},"updated":"2016-09-01 15:56:52.000000000","message":"Are you sure? Again, the below spec suggests that this would work anyway and, at best, the new parameter would have no effect.\n\n    https://review.openstack.org/#/c/334419/3/specs/ocata/approved/modify-pci-without-numa-node-info-treatment.rst","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"491bb0a497f6380241c4c82dca6b7b0703172cdc","unresolved":false,"context_lines":[{"line_number":108,"context_line":"Other deployer impact"},{"line_number":109,"context_line":"---------------------"},{"line_number":110,"context_line":""},{"line_number":111,"context_line":"To use this feature the deployer must use HW that is capable of reporting"},{"line_number":112,"context_line":"NUMA related info to the OS."},{"line_number":113,"context_line":""},{"line_number":114,"context_line":"Developer impact"},{"line_number":115,"context_line":"----------------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a8ec9b2_aacc3d36","line":112,"range":{"start_line":111,"start_character":0,"end_line":112,"end_character":28},"in_reply_to":"fa7ab95a_75d495bc","updated":"2016-09-14 14:20:20.000000000","message":"Done","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"c9333295807678587c15708a92aa2db0fcd5053f","unresolved":false,"context_lines":[{"line_number":123,"context_line":"-----------"},{"line_number":124,"context_line":""},{"line_number":125,"context_line":"Primary assignee:"},{"line_number":126,"context_line":"    Sergey Nikitin (snikitin)"},{"line_number":127,"context_line":""},{"line_number":128,"context_line":"Work Items"},{"line_number":129,"context_line":"----------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa7ab95a_159509e3","line":126,"range":{"start_line":126,"start_character":0,"end_line":126,"end_character":29},"updated":"2016-09-01 15:56:52.000000000","message":"I\u0027d like to help with the documentation side of this, if you wanted a hand","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"491bb0a497f6380241c4c82dca6b7b0703172cdc","unresolved":false,"context_lines":[{"line_number":123,"context_line":"-----------"},{"line_number":124,"context_line":""},{"line_number":125,"context_line":"Primary assignee:"},{"line_number":126,"context_line":"    Sergey Nikitin (snikitin)"},{"line_number":127,"context_line":""},{"line_number":128,"context_line":"Work Items"},{"line_number":129,"context_line":"----------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a8ec9b2_c5ddfa43","line":126,"range":{"start_line":126,"start_character":0,"end_line":126,"end_character":29},"in_reply_to":"fa7ab95a_159509e3","updated":"2016-09-14 14:20:20.000000000","message":"sure. Would be great!","commit_id":"350ad4567ac89f93f5b49cde8a06f1024c3f2350"},{"author":{"_account_id":17973,"name":"Ludovic Beliveau","email":"lbeliveau@gmail.com","username":"lbeliveau"},"change_message_id":"664ce066beba9d809045a89385c1c64e43d4facf","unresolved":false,"context_lines":[{"line_number":65,"context_line":"------------"},{"line_number":66,"context_line":""},{"line_number":67,"context_line":"Retain current scheduler behaviour and buy more PCI devices to give each NUMA"},{"line_number":68,"context_line":"node its own PCI device."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"Data model impact"},{"line_number":71,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":5,"id":"7a8ec9b2_8aabf3d5","line":68,"updated":"2016-09-14 16:48:29.000000000","message":"IMO this is not an alternative solution in the sense that you don\u0027t provide another way to implement this feature.\n\nSo I would put none here and move this phrase in the problem description.","commit_id":"d2017a31fee217e022b0735b64fa23d0939633aa"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"b0ac344a2ccb6f138a2d515862d902356eecddd6","unresolved":false,"context_lines":[{"line_number":65,"context_line":"------------"},{"line_number":66,"context_line":""},{"line_number":67,"context_line":"Retain current scheduler behaviour and buy more PCI devices to give each NUMA"},{"line_number":68,"context_line":"node its own PCI device."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"Data model impact"},{"line_number":71,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":5,"id":"7a8ec9b2_6b091fdd","line":68,"in_reply_to":"7a8ec9b2_559c0769","updated":"2016-09-15 12:26:44.000000000","message":"It\u0027ll certainly need a spec as its a significant change in behaviour, and there\u0027s been many other changes suggested in this area, to the extent that we likely need a configurable placement policy. eg some people prefer spreading across all nodes, instead of filling nodes","commit_id":"d2017a31fee217e022b0735b64fa23d0939633aa"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"95a9cb6d81888a7beaab6223e249a840e7266461","unresolved":false,"context_lines":[{"line_number":65,"context_line":"------------"},{"line_number":66,"context_line":""},{"line_number":67,"context_line":"Retain current scheduler behaviour and buy more PCI devices to give each NUMA"},{"line_number":68,"context_line":"node its own PCI device."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"Data model impact"},{"line_number":71,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":5,"id":"7a8ec9b2_559c0769","line":68,"in_reply_to":"7a8ec9b2_657df81b","updated":"2016-09-15 12:12:55.000000000","message":"Yes, this is a good alternative but i doesn\u0027t fully solve the problem. For example you have PCI only on first NUMA node. And you will boot VMs without PCI on other NUMA nodes. But if your first NUMA node will be full of VMs with PCI, you wouldn\u0027t be able to boot any new VM with PCI. \n\nI think your idea is good and we need to implement it, but I\u0027m not sure that it need a spec. We will just optimize placement behavior. No API changes, no user side changes, no config changes.","commit_id":"d2017a31fee217e022b0735b64fa23d0939633aa"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"ef88aa7b22a0018fcb0b84a891289786dd7f60ce","unresolved":false,"context_lines":[{"line_number":65,"context_line":"------------"},{"line_number":66,"context_line":""},{"line_number":67,"context_line":"Retain current scheduler behaviour and buy more PCI devices to give each NUMA"},{"line_number":68,"context_line":"node its own PCI device."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"Data model impact"},{"line_number":71,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":5,"id":"7a8ec9b2_cb5f933f","line":68,"in_reply_to":"7a8ec9b2_6b091fdd","updated":"2016-09-15 12:35:47.000000000","message":"ok. agree with you","commit_id":"d2017a31fee217e022b0735b64fa23d0939633aa"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"6159440bf9677611a7457987722399cc117e93c1","unresolved":false,"context_lines":[{"line_number":65,"context_line":"------------"},{"line_number":66,"context_line":""},{"line_number":67,"context_line":"Retain current scheduler behaviour and buy more PCI devices to give each NUMA"},{"line_number":68,"context_line":"node its own PCI device."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"Data model impact"},{"line_number":71,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":5,"id":"7a8ec9b2_657df81b","line":68,"in_reply_to":"7a8ec9b2_8aabf3d5","updated":"2016-09-14 16:59:45.000000000","message":"There is a slightly more serious alternative. Currently Nova packs guests onto NUMA nodes, filling the first node before considering the second node, then third, etc.\n\nIf your PCI device is on the first node and you\u0027re running guests which don\u0027t use PCI devices, then you might fill up the first node with guests that don\u0027t require PCI devices. When you finally need a guest with PCI device, only the second node is free.\n\nSo an alternative is to define a new guest placement policy such that if the guest doesn\u0027t require PCI devices, the scheduler will try to avoid placing it on a node which has PCI devices. This means that the nodes with PCI devices are much much more likely to be available when needed, at which point this spec is less important.\n\nThat said, I actually think we want both enhancements - this spec, and also the improved VM placement to avoid PCI device nodes","commit_id":"d2017a31fee217e022b0735b64fa23d0939633aa"},{"author":{"_account_id":17973,"name":"Ludovic Beliveau","email":"lbeliveau@gmail.com","username":"lbeliveau"},"change_message_id":"664ce066beba9d809045a89385c1c64e43d4facf","unresolved":false,"context_lines":[{"line_number":132,"context_line":""},{"line_number":133,"context_line":"* Add new spec to the flavor"},{"line_number":134,"context_line":"* Add new field to the InstanceNUMACell object"},{"line_number":135,"context_line":"* Change the process of NUMA node choosing, considering new policy"},{"line_number":136,"context_line":"* Update user docs"},{"line_number":137,"context_line":""},{"line_number":138,"context_line":"Dependencies"}],"source_content_type":"text/x-rst","patch_set":5,"id":"7a8ec9b2_4a5e4b77","line":135,"updated":"2016-09-14 16:48:29.000000000","message":"Can you go in more details as to what this implies ?  It would be nice to have a kind of walk through on what will happen when a user boot a guest with pci_numa_affinity\u003dprefer (scheduler, compute, ...).","commit_id":"d2017a31fee217e022b0735b64fa23d0939633aa"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"95a9cb6d81888a7beaab6223e249a840e7266461","unresolved":false,"context_lines":[{"line_number":132,"context_line":""},{"line_number":133,"context_line":"* Add new spec to the flavor"},{"line_number":134,"context_line":"* Add new field to the InstanceNUMACell object"},{"line_number":135,"context_line":"* Change the process of NUMA node choosing, considering new policy"},{"line_number":136,"context_line":"* Update user docs"},{"line_number":137,"context_line":""},{"line_number":138,"context_line":"Dependencies"}],"source_content_type":"text/x-rst","patch_set":5,"id":"7a8ec9b2_1586ffb8","line":135,"in_reply_to":"7a8ec9b2_4a5e4b77","updated":"2016-09-15 12:12:55.000000000","message":"ok, will explain in \u0027Proposed change\u0027","commit_id":"d2017a31fee217e022b0735b64fa23d0939633aa"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"75ea811f078dde5f47435710f4644180d0b811c1","unresolved":false,"context_lines":[{"line_number":45,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":46,"context_line":""},{"line_number":47,"context_line":"Add new flavor extra spec \u0027hw:pci_numa_policy\u0027 and image metadata"},{"line_number":48,"context_line":"\u0027hw_pci_numa_policy\u0027. They will has one of two values:"},{"line_number":49,"context_line":""},{"line_number":50,"context_line":" * required"},{"line_number":51,"context_line":" * prefered"}],"source_content_type":"text/x-rst","patch_set":6,"id":"7a8ec9b2_a7b4a8d4","line":48,"range":{"start_line":48,"start_character":32,"end_line":48,"end_character":35},"updated":"2016-09-19 13:20:35.000000000","message":"nit: have","commit_id":"f01a843be725a065307843e10465f3ab809ab368"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"9fc2dce910ecf2a82a6e4a4a7d16c50f2bdc89d5","unresolved":false,"context_lines":[{"line_number":45,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":46,"context_line":""},{"line_number":47,"context_line":"Add new flavor extra spec \u0027hw:pci_numa_policy\u0027 and image metadata"},{"line_number":48,"context_line":"\u0027hw_pci_numa_policy\u0027. They will has one of two values:"},{"line_number":49,"context_line":""},{"line_number":50,"context_line":" * required"},{"line_number":51,"context_line":" * prefered"}],"source_content_type":"text/x-rst","patch_set":6,"id":"7a8ec9b2_4ac3e08a","line":48,"range":{"start_line":48,"start_character":32,"end_line":48,"end_character":35},"in_reply_to":"7a8ec9b2_a7b4a8d4","updated":"2016-09-20 15:30:10.000000000","message":"Done","commit_id":"f01a843be725a065307843e10465f3ab809ab368"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"75ea811f078dde5f47435710f4644180d0b811c1","unresolved":false,"context_lines":[{"line_number":48,"context_line":"\u0027hw_pci_numa_policy\u0027. They will has one of two values:"},{"line_number":49,"context_line":""},{"line_number":50,"context_line":" * required"},{"line_number":51,"context_line":" * prefered"},{"line_number":52,"context_line":""},{"line_number":53,"context_line":"This spec will be used by to decide on which NUMA nodes VM can be booted."},{"line_number":54,"context_line":"Default value of this option will be \u0027required\u0027. It means that if host NUMA"}],"source_content_type":"text/x-rst","patch_set":6,"id":"7a8ec9b2_a72f48a6","line":51,"range":{"start_line":51,"start_character":3,"end_line":51,"end_character":11},"updated":"2016-09-19 13:20:35.000000000","message":"preferred (two \u0027r\u0027s)","commit_id":"f01a843be725a065307843e10465f3ab809ab368"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"9fc2dce910ecf2a82a6e4a4a7d16c50f2bdc89d5","unresolved":false,"context_lines":[{"line_number":48,"context_line":"\u0027hw_pci_numa_policy\u0027. They will has one of two values:"},{"line_number":49,"context_line":""},{"line_number":50,"context_line":" * required"},{"line_number":51,"context_line":" * prefered"},{"line_number":52,"context_line":""},{"line_number":53,"context_line":"This spec will be used by to decide on which NUMA nodes VM can be booted."},{"line_number":54,"context_line":"Default value of this option will be \u0027required\u0027. It means that if host NUMA"}],"source_content_type":"text/x-rst","patch_set":6,"id":"7a8ec9b2_6ac09c7d","line":51,"range":{"start_line":51,"start_character":3,"end_line":51,"end_character":11},"in_reply_to":"7a8ec9b2_a72f48a6","updated":"2016-09-20 15:30:10.000000000","message":"Done","commit_id":"f01a843be725a065307843e10465f3ab809ab368"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"75ea811f078dde5f47435710f4644180d0b811c1","unresolved":false,"context_lines":[{"line_number":66,"context_line":"try to choose a NUMA node associated with requested PCI device but if it failed"},{"line_number":67,"context_line":"nova compute will try to choose NUMA node again ignoring association of NUMA"},{"line_number":68,"context_line":"nodes and PCI devices."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"Alternatives"},{"line_number":71,"context_line":"------------"},{"line_number":72,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"7a8ec9b2_0748fc86","line":69,"updated":"2016-09-19 13:20:35.000000000","message":"You should call out what happens if there\u0027s a conflict between image and flavor metadata. I would recommend the following:\n\n    flavor    image     outcome\n    --------- --------- ---------\n    preferred required  allow\n    required  preferred exception","commit_id":"f01a843be725a065307843e10465f3ab809ab368"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"9fc2dce910ecf2a82a6e4a4a7d16c50f2bdc89d5","unresolved":false,"context_lines":[{"line_number":66,"context_line":"try to choose a NUMA node associated with requested PCI device but if it failed"},{"line_number":67,"context_line":"nova compute will try to choose NUMA node again ignoring association of NUMA"},{"line_number":68,"context_line":"nodes and PCI devices."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"Alternatives"},{"line_number":71,"context_line":"------------"},{"line_number":72,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"7a8ec9b2_8a44c5b7","line":69,"in_reply_to":"7a8ec9b2_0748fc86","updated":"2016-09-20 15:30:10.000000000","message":"I\u0027m a little bit confused about this. Obviously the default value for flavor and image metadata will be None. We interpret it like \u0027required\u0027 policy.\n\nSo we have 4 different cases:\n\n* if both image and flavor metadata will have \u0027None\u0027 value, we will use \u0027required\u0027 policy.\n\n* if flavor *or* image metadata (not both) will have not \u0027None\u0027 value, we will use this not \u0027None\u0027 value.\n\n* if flavor *and* image metadata have equal not \u0027None\u0027 value obviously we will use this value\n\n* flavor and image metadata will have not equal and not \u0027None\u0027 values. Here we have conflict.\n\n\n***Note***: \u0027required\u0027 metadata value is not quite default value because the default value is \u0027None\u0027.    \u0027required\u0027 value means that user *consciously* added this value to the flavor/image. It meas that he wants a performance VM.\n\n\nSo after this note I think we have three options:\n\n\n1. we will use \u0027required\u0027 policy, because this policy means \"We need a performance VM!\", and if even one of flavor or image metadata wants a performance VM, we will got a performance VM.\n2. We will raise an exception. \n3. We will do like Stephen proposed.\n\nI prefer the second option to do not confuse our users. I think such case is a classical \u0027Bad request\u0027 case because the request has a contradictions and we need to raise an exception. What do you think?","commit_id":"f01a843be725a065307843e10465f3ab809ab368"},{"author":{"_account_id":8768,"name":"Chris Friesen","email":"chris.friesen@windriver.com","username":"cbf123"},"change_message_id":"87262d1a9bd185a955712fcd584b088a04542308","unresolved":false,"context_lines":[{"line_number":66,"context_line":"try to choose a NUMA node associated with requested PCI device but if it failed"},{"line_number":67,"context_line":"nova compute will try to choose NUMA node again ignoring association of NUMA"},{"line_number":68,"context_line":"nodes and PCI devices."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"Alternatives"},{"line_number":71,"context_line":"------------"},{"line_number":72,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"7a8ec9b2_07e4d8dd","line":69,"in_reply_to":"7a8ec9b2_7c32a011","updated":"2016-09-21 19:21:35.000000000","message":"The question is what counts as a \"subset\".  Is \"preferred\" a subset of \"required\", or is it the other way around?\n\nOne could argue that we should allow the user to make scheduling less restrictive, not more.  So if the flavor specifies \"preferred\" then the tenant can\u0027t specify \"required\" because it would make scheduling more restrictive.  But if the flavor specifies \"required\" we could allow the image to specify \"preferred\" since it makes scheduling less restrictive.\n\nThis sort of holds true for dedicated CPUs, where if the flavor specifies \"shared\" then the image isn\u0027t allowed to specify \"dedicated\".  However if the flavor sets \"dedicated\" then you get \"dedicated\" no matter what the image sets, which is arguably a bit odd.","commit_id":"f01a843be725a065307843e10465f3ab809ab368"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"ffda6203713c79f7cfe801bd59aac0482b9a7f56","unresolved":false,"context_lines":[{"line_number":66,"context_line":"try to choose a NUMA node associated with requested PCI device but if it failed"},{"line_number":67,"context_line":"nova compute will try to choose NUMA node again ignoring association of NUMA"},{"line_number":68,"context_line":"nodes and PCI devices."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"Alternatives"},{"line_number":71,"context_line":"------------"},{"line_number":72,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"7a8ec9b2_7c32a011","line":69,"in_reply_to":"7a8ec9b2_7c9c60c0","updated":"2016-09-21 15:52:56.000000000","message":"The way we\u0027ve dealt with this for other NUMA related properties is that, if set, the flavour takes priority. The image is used only if flavour is not set, or if the image setting is a subset of what is already allowed by the flavour setting.\n\nThe rationale is that the cloud admin should be able to lock down what the tenant can do by setting a flavour property.","commit_id":"f01a843be725a065307843e10465f3ab809ab368"},{"author":{"_account_id":8768,"name":"Chris Friesen","email":"chris.friesen@windriver.com","username":"cbf123"},"change_message_id":"77da80c61c8e9ecb50701edf8b892e4a740a3eb3","unresolved":false,"context_lines":[{"line_number":66,"context_line":"try to choose a NUMA node associated with requested PCI device but if it failed"},{"line_number":67,"context_line":"nova compute will try to choose NUMA node again ignoring association of NUMA"},{"line_number":68,"context_line":"nodes and PCI devices."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"Alternatives"},{"line_number":71,"context_line":"------------"},{"line_number":72,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"7a8ec9b2_f3ad449c","line":69,"in_reply_to":"7a8ec9b2_8a44c5b7","updated":"2016-09-20 16:15:45.000000000","message":"Since the default is to require affinity, setting \"preferred\" is equivalent to explicitly saying \"this image/flavor cares more about running *somewhere* than it does about performance\".\n\nAs such, I think it would almost make sense to do the opposite of your number 1....if either of image or flavor explicitly sets \"preferred\", then we go with \"preferred\".\n\nIt seems backwards, I know...","commit_id":"f01a843be725a065307843e10465f3ab809ab368"},{"author":{"_account_id":8768,"name":"Chris Friesen","email":"chris.friesen@windriver.com","username":"cbf123"},"change_message_id":"8543d864a99ac49299da432a2c11d398ddc6da4a","unresolved":false,"context_lines":[{"line_number":66,"context_line":"try to choose a NUMA node associated with requested PCI device but if it failed"},{"line_number":67,"context_line":"nova compute will try to choose NUMA node again ignoring association of NUMA"},{"line_number":68,"context_line":"nodes and PCI devices."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"Alternatives"},{"line_number":71,"context_line":"------------"},{"line_number":72,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"7a8ec9b2_7c9c60c0","line":69,"in_reply_to":"7a8ec9b2_c403e740","updated":"2016-09-21 15:43:24.000000000","message":"The image can be created by \"normal\" users, but the flavor is set by the admin.  So it\u0027s not necessarily the same people.\n\nI don\u0027t think there\u0027s a single right answer here.  We could prioritize the flavor since it\u0027s set by the admin.  We could raise an exception in all cases of disagreement.  We could prioritize \"required\" or \"preferred\".  They\u0027re all reasonable choices.","commit_id":"f01a843be725a065307843e10465f3ab809ab368"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"c1a1efac4f8ad1c8cf552b102cd698dbfa1407ed","unresolved":false,"context_lines":[{"line_number":66,"context_line":"try to choose a NUMA node associated with requested PCI device but if it failed"},{"line_number":67,"context_line":"nova compute will try to choose NUMA node again ignoring association of NUMA"},{"line_number":68,"context_line":"nodes and PCI devices."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"Alternatives"},{"line_number":71,"context_line":"------------"},{"line_number":72,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"7a8ec9b2_c403e740","line":69,"in_reply_to":"7a8ec9b2_f3ad449c","updated":"2016-09-21 13:08:58.000000000","message":"Yes, maybe we could do so. But I want to repeat my note. in 90% of cases user wouldn\u0027t know/use this flavor/image metadata. Only if user wouldn\u0027t be able to boot a VM, he/she will try to use \u0027preferred\u0027 value to boot VM \u0027somewhere\u0027. The *only* reason to set value to \u0027required\u0027 is to make sure that in any case VM will have strict PCI-NUMA association. In all other cases user will use \u0027preferred\u0027 or \u0027None\u0027 values.\n\n\nI just want to say that value \u0027preferred\u0027 in flavor and \u0027required\u0027 in image means that user **consciously** adeed \u0027required\u0027 to get a performance, and **consciously** added \u0027preferred\u0027 to boot VM some where. This is a contradiction and we must raise an exception.\n\n\nIn normal workflow user will try to boot VM without any metadata. If it will be booted - that\u0027s OK. If it will be failed - user will try to use \u0027preferred\u0027. Or user will always use \u0027preferred\u0027 if he knows that his env doesn\u0027t have enough PCI devices.","commit_id":"f01a843be725a065307843e10465f3ab809ab368"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"75ea811f078dde5f47435710f4644180d0b811c1","unresolved":false,"context_lines":[{"line_number":111,"context_line":"on NUMA nodes with associated PCI devices. But with current scheduler behavior"},{"line_number":112,"context_line":"NUMA nodes without associated PCI devices wouldn\u0027t be used at all."},{"line_number":113,"context_line":""},{"line_number":114,"context_line":"If option will have \u0027required\u0027 value Nova behaviour will be the same as"},{"line_number":115,"context_line":"current behaviour."},{"line_number":116,"context_line":""},{"line_number":117,"context_line":"Other deployer impact"}],"source_content_type":"text/x-rst","patch_set":6,"id":"7a8ec9b2_c715349c","line":114,"range":{"start_line":114,"start_character":37,"end_line":114,"end_character":41},"updated":"2016-09-19 13:20:35.000000000","message":"nit: doc guidelines say to use \u0027nova\u0027 (lowercase). Same for any other occurrences in here","commit_id":"f01a843be725a065307843e10465f3ab809ab368"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"9fc2dce910ecf2a82a6e4a4a7d16c50f2bdc89d5","unresolved":false,"context_lines":[{"line_number":111,"context_line":"on NUMA nodes with associated PCI devices. But with current scheduler behavior"},{"line_number":112,"context_line":"NUMA nodes without associated PCI devices wouldn\u0027t be used at all."},{"line_number":113,"context_line":""},{"line_number":114,"context_line":"If option will have \u0027required\u0027 value Nova behaviour will be the same as"},{"line_number":115,"context_line":"current behaviour."},{"line_number":116,"context_line":""},{"line_number":117,"context_line":"Other deployer impact"}],"source_content_type":"text/x-rst","patch_set":6,"id":"7a8ec9b2_ca3530a9","line":114,"range":{"start_line":114,"start_character":37,"end_line":114,"end_character":41},"in_reply_to":"7a8ec9b2_c715349c","updated":"2016-09-20 15:30:10.000000000","message":"Done","commit_id":"f01a843be725a065307843e10465f3ab809ab368"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"75ea811f078dde5f47435710f4644180d0b811c1","unresolved":false,"context_lines":[{"line_number":122,"context_line":"Developer impact"},{"line_number":123,"context_line":"----------------"},{"line_number":124,"context_line":""},{"line_number":125,"context_line":"None"},{"line_number":126,"context_line":""},{"line_number":127,"context_line":"Implementation"},{"line_number":128,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":6,"id":"7a8ec9b2_0702bc6f","line":125,"range":{"start_line":125,"start_character":0,"end_line":125,"end_character":4},"updated":"2016-09-19 13:20:35.000000000","message":"Actually, I guess this will only be implemented for libvirt? If so, other hypervisors will need to be extended to support this","commit_id":"f01a843be725a065307843e10465f3ab809ab368"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"9fc2dce910ecf2a82a6e4a4a7d16c50f2bdc89d5","unresolved":false,"context_lines":[{"line_number":122,"context_line":"Developer impact"},{"line_number":123,"context_line":"----------------"},{"line_number":124,"context_line":""},{"line_number":125,"context_line":"None"},{"line_number":126,"context_line":""},{"line_number":127,"context_line":"Implementation"},{"line_number":128,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":6,"id":"7a8ec9b2_b95d7317","line":125,"range":{"start_line":125,"start_character":0,"end_line":125,"end_character":4},"in_reply_to":"7a8ec9b2_0702bc6f","updated":"2016-09-20 15:30:10.000000000","message":"Mostly we will change virt/hardware.py module. We wouldn\u0027t change anything in hypervisors drivers. \n\nAlso as I see VMWare doesn\u0027t support NUMA at all. XenAPI has very limited NUMA support and this support works only with Windows VMs.\n\nhttps://wiki.openstack.org/wiki/VirtDriverGuestCPUMemoryPlacement#Technology_availability\n\nSo I guess at this moment we can\u0027t extend other hypervisor drivers to support this feature.","commit_id":"f01a843be725a065307843e10465f3ab809ab368"},{"author":{"_account_id":17973,"name":"Ludovic Beliveau","email":"lbeliveau@gmail.com","username":"lbeliveau"},"change_message_id":"aabe919b944df362ca99548f7b5c9ab3ef2e716f","unresolved":false,"context_lines":[{"line_number":141,"context_line":""},{"line_number":142,"context_line":"* Add new spec to the flavor"},{"line_number":143,"context_line":"* Add new field to the InstanceNUMACell object"},{"line_number":144,"context_line":"* Change the process of NUMA node choosing, considering new policy"},{"line_number":145,"context_line":"* Update user docs"},{"line_number":146,"context_line":""},{"line_number":147,"context_line":"Dependencies"}],"source_content_type":"text/x-rst","patch_set":6,"id":"7a8ec9b2_ae0af123","line":144,"updated":"2016-09-17 17:18:34.000000000","message":"Yeah, I had a bit more time to think about how this could be implemented, and you are right, mostly all changes in the scheduling logic will be in numa_fit_instance_to_host().","commit_id":"f01a843be725a065307843e10465f3ab809ab368"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"c21e6d61e1be6e53056b8d837c9532d934af4c3b","unresolved":false,"context_lines":[{"line_number":47,"context_line":"Add new flavor extra spec \u0027hw:pci_numa_policy\u0027 and image metadata"},{"line_number":48,"context_line":"\u0027hw_pci_numa_policy\u0027. They will have one of two values:"},{"line_number":49,"context_line":""},{"line_number":50,"context_line":" * required"},{"line_number":51,"context_line":" * preferred"},{"line_number":52,"context_line":""},{"line_number":53,"context_line":"This spec will be used by to decide on which NUMA nodes VM can be booted."},{"line_number":54,"context_line":"Default value of this option will be \u0027required\u0027. It means that if host NUMA"}],"source_content_type":"text/x-rst","patch_set":7,"id":"5a8bc5a2_23878513","line":51,"range":{"start_line":50,"start_character":0,"end_line":51,"end_character":12},"updated":"2016-09-28 12:29:43.000000000","message":"I had a look at another spec [1] and it seems require isn\u0027t strictly true - if NUMA topology information isn\u0027t exposed then a PCI device is set as belonging to all nodes. Maybe we should add a third policy where a NUMA topology *must* be available?\n\n[1] https://review.openstack.org/#/c/334419/","commit_id":"54b81c3ba0a911d5df48943169c34e59258d2be0"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"e5b1b0c4f206090ef6d0276a0c6e002948fc53f2","unresolved":false,"context_lines":[{"line_number":47,"context_line":"Add new flavor extra spec \u0027hw:pci_numa_policy\u0027 and image metadata"},{"line_number":48,"context_line":"\u0027hw_pci_numa_policy\u0027. They will have one of two values:"},{"line_number":49,"context_line":""},{"line_number":50,"context_line":" * required"},{"line_number":51,"context_line":" * preferred"},{"line_number":52,"context_line":""},{"line_number":53,"context_line":"This spec will be used by to decide on which NUMA nodes VM can be booted."},{"line_number":54,"context_line":"Default value of this option will be \u0027required\u0027. It means that if host NUMA"}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a98d1f4_c26286ad","line":51,"range":{"start_line":50,"start_character":0,"end_line":51,"end_character":12},"in_reply_to":"5a8bc5a2_23878513","updated":"2016-09-28 13:02:32.000000000","message":"Very good idea! We just need to come up name of the new policy :)","commit_id":"54b81c3ba0a911d5df48943169c34e59258d2be0"},{"author":{"_account_id":22307,"name":"Carlos Conde","email":"carconper@gmail.com"},"change_message_id":"f3d4d539782c2f74c83ef96b56757f4a4b9da00e","unresolved":false,"context_lines":[{"line_number":69,"context_line":""},{"line_number":70,"context_line":"If both of image and flavor property is not set (equals \u0027None\u0027) \u0027required\u0027"},{"line_number":71,"context_line":"policy will be used. If one of image *or* flavor property is not set (equals"},{"line_number":72,"context_line":"\u0027None\u0027) but the other property is set then will be used the value of"},{"line_number":73,"context_line":"the property which was set. In a case of conflicts between flavor and image"},{"line_number":74,"context_line":"properties (both properties are set and they are not equal) an exception will"},{"line_number":75,"context_line":"be raised."},{"line_number":76,"context_line":""}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a98d1f4_c0ac17f0","line":73,"range":{"start_line":72,"start_character":38,"end_line":73,"end_character":26},"updated":"2016-09-29 03:38:19.000000000","message":"... the value of the the property which was set will be used.","commit_id":"54b81c3ba0a911d5df48943169c34e59258d2be0"},{"author":{"_account_id":22307,"name":"Carlos Conde","email":"carconper@gmail.com"},"change_message_id":"f3d4d539782c2f74c83ef96b56757f4a4b9da00e","unresolved":false,"context_lines":[{"line_number":79,"context_line":""},{"line_number":80,"context_line":"Now we have no implemented alternatives. But we can change placement behavior"},{"line_number":81,"context_line":"to do *not* boot VMs which need no PCI devices on NUMA nodes with PCI devices"},{"line_number":82,"context_line":"if possible. Unfortunately it wouldn\u0027t fully solve the problem. User wouldn\u0027t"},{"line_number":83,"context_line":"be able to boot VM with PCI device if all NUMA nodes with PCI devices will be"},{"line_number":84,"context_line":"used by VMs which require PCI even NUMA nodes without PCI devices will be free."},{"line_number":85,"context_line":""},{"line_number":86,"context_line":"Data model impact"},{"line_number":87,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a98d1f4_e06cb39e","line":84,"range":{"start_line":82,"start_character":64,"end_line":84,"end_character":78},"updated":"2016-09-29 03:38:19.000000000","message":"I may be missing something but IMO this sentence needs rewriting. I don\u0027t really get what you are trying to explain. Maybe it would be useful to use a couple of sentences, or some commas to separate ideas?","commit_id":"54b81c3ba0a911d5df48943169c34e59258d2be0"},{"author":{"_account_id":22307,"name":"Carlos Conde","email":"carconper@gmail.com"},"change_message_id":"f3d4d539782c2f74c83ef96b56757f4a4b9da00e","unresolved":false,"context_lines":[{"line_number":87,"context_line":"-----------------"},{"line_number":88,"context_line":""},{"line_number":89,"context_line":"New field \u0027pci_numa_policy\u0027 will be added to the InstanceNUMACell object."},{"line_number":90,"context_line":"Fortunately this object is storing as JSON in DB. So we need no DB migrations"},{"line_number":91,"context_line":"to add new field to this object."},{"line_number":92,"context_line":""},{"line_number":93,"context_line":"REST API impact"}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a98d1f4_a0dd4b72","line":90,"range":{"start_line":90,"start_character":27,"end_line":90,"end_character":34},"updated":"2016-09-29 03:38:19.000000000","message":"stored","commit_id":"54b81c3ba0a911d5df48943169c34e59258d2be0"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"ccb034d58c749bcff9ae2c22fddb74bce68edf64","unresolved":false,"context_lines":[{"line_number":5,"context_line":" http://creativecommons.org/licenses/by/3.0/legalcode"},{"line_number":6,"context_line":""},{"line_number":7,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":8,"context_line":"I/O (PCIe) sharing between NUMA nodes"},{"line_number":9,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/nova/+spec/share-pci-between-numa-nodes"}],"source_content_type":"text/x-rst","patch_set":8,"id":"3a98d1f4_a0d9a785","line":8,"range":{"start_line":8,"start_character":0,"end_line":8,"end_character":37},"updated":"2016-09-29 16:40:45.000000000","message":"Perhaps a new name might be helpful now? How about\n\n    PCI NUMA Policies\n\n?","commit_id":"41e6d65d88e6c84abef36f912c7777f20d4084f6"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"177f90eac22de191dda2f8b0ae1f03f83367070c","unresolved":false,"context_lines":[{"line_number":5,"context_line":" http://creativecommons.org/licenses/by/3.0/legalcode"},{"line_number":6,"context_line":""},{"line_number":7,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":8,"context_line":"I/O (PCIe) sharing between NUMA nodes"},{"line_number":9,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/nova/+spec/share-pci-between-numa-nodes"}],"source_content_type":"text/x-rst","patch_set":8,"id":"3a98d1f4_93476b76","line":8,"range":{"start_line":8,"start_character":0,"end_line":8,"end_character":37},"in_reply_to":"3a98d1f4_a0d9a785","updated":"2017-02-07 08:00:03.000000000","message":"Yep. need to be updated","commit_id":"41e6d65d88e6c84abef36f912c7777f20d4084f6"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"ccb034d58c749bcff9ae2c22fddb74bce68edf64","unresolved":false,"context_lines":[{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/nova/+spec/share-pci-between-numa-nodes"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"In Juno release \"I/O based NUMA scheduling\" spec was implemented [1]. This"},{"line_number":14,"context_line":"scheduling algorithm allows to boot VM with PCI devices *only* if at least one"},{"line_number":15,"context_line":"of the NUMA nodes is associated with the PCI devices. Before that nova booted"},{"line_number":16,"context_line":"VMs with PCI devices without checking NUMA nodes. However, such hardcoded"}],"source_content_type":"text/x-rst","patch_set":8,"id":"3a98d1f4_40b543ec","line":13,"range":{"start_line":13,"start_character":65,"end_line":13,"end_character":68},"updated":"2016-09-29 16:40:45.000000000","message":"This isn\u0027t coming out correctly in the built version. I think this needs some work.\n\n[1] http://docs-draft.openstack.org/40/361140/8/check/gate-nova-specs-docs-ubuntu-xenial/94779cf//doc/build/html/specs/ocata/approved/share-pci-between-numa-nodes.html","commit_id":"41e6d65d88e6c84abef36f912c7777f20d4084f6"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"177f90eac22de191dda2f8b0ae1f03f83367070c","unresolved":false,"context_lines":[{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/nova/+spec/share-pci-between-numa-nodes"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"In Juno release \"I/O based NUMA scheduling\" spec was implemented [1]. This"},{"line_number":14,"context_line":"scheduling algorithm allows to boot VM with PCI devices *only* if at least one"},{"line_number":15,"context_line":"of the NUMA nodes is associated with the PCI devices. Before that nova booted"},{"line_number":16,"context_line":"VMs with PCI devices without checking NUMA nodes. However, such hardcoded"}],"source_content_type":"text/x-rst","patch_set":8,"id":"3a98d1f4_7342df69","line":13,"range":{"start_line":13,"start_character":65,"end_line":13,"end_character":68},"in_reply_to":"3a98d1f4_40b543ec","updated":"2017-02-07 08:00:03.000000000","message":"Done","commit_id":"41e6d65d88e6c84abef36f912c7777f20d4084f6"},{"author":{"_account_id":17973,"name":"Ludovic Beliveau","email":"lbeliveau@gmail.com","username":"lbeliveau"},"change_message_id":"7ade1f9426481036f05dc8afbcb4510432341baa","unresolved":false,"context_lines":[{"line_number":24,"context_line":"associated with the PCI devices is good for performance. This ensures there is"},{"line_number":25,"context_line":"no cross NUMA node memory traffic."},{"line_number":26,"context_line":""},{"line_number":27,"context_line":"But if a user has environment with 2 NUMA nodes and only 1 PCI device (for"},{"line_number":28,"context_line":"example SR-IOV card associated with first NUMA node) he or she would be able to"},{"line_number":29,"context_line":"boot VMs with *one* NUMA node and SR-IOV ports only on the first NUMA node."},{"line_number":30,"context_line":"In this case user cannot use half of CPUs and half of RAM because these"}],"source_content_type":"text/x-rst","patch_set":9,"id":"3a98d1f4_a50c0fee","line":27,"updated":"2016-09-30 16:37:41.000000000","message":"nit: s/has environment/has an environment/","commit_id":"884d295d4abb353a27d6ee6e3fb03e50b0487f0c"},{"author":{"_account_id":17973,"name":"Ludovic Beliveau","email":"lbeliveau@gmail.com","username":"lbeliveau"},"change_message_id":"7ade1f9426481036f05dc8afbcb4510432341baa","unresolved":false,"context_lines":[{"line_number":28,"context_line":"example SR-IOV card associated with first NUMA node) he or she would be able to"},{"line_number":29,"context_line":"boot VMs with *one* NUMA node and SR-IOV ports only on the first NUMA node."},{"line_number":30,"context_line":"In this case user cannot use half of CPUs and half of RAM because these"},{"line_number":31,"context_line":"resources are placed on second NUMA node. In described case user should be able"},{"line_number":32,"context_line":"to boot VMs on different NUMA nodes, even if it makes performance on second"},{"line_number":33,"context_line":"NUMA node worse."},{"line_number":34,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"3a98d1f4_a5332faa","line":31,"updated":"2016-09-30 16:37:41.000000000","message":"s/In described case/In this case/","commit_id":"884d295d4abb353a27d6ee6e3fb03e50b0487f0c"},{"author":{"_account_id":17973,"name":"Ludovic Beliveau","email":"lbeliveau@gmail.com","username":"lbeliveau"},"change_message_id":"7ade1f9426481036f05dc8afbcb4510432341baa","unresolved":false,"context_lines":[{"line_number":35,"context_line":"Also current behavior is not a best performance solution. If some compute nodes"},{"line_number":36,"context_line":"don\u0027t report NUMA node info for pci devices, then these pci devices will be"},{"line_number":37,"context_line":"regarded as \"belong to all numa node\". It can lead to a problem that pci"},{"line_number":38,"context_line":"devices are not on the numa node which CPU\\MEM on. In this way, the real"},{"line_number":39,"context_line":"purpose of I/O (PCIe) Based NUMA Scheduling is not reached. In this case we can"},{"line_number":40,"context_line":"deny using of such PCI devices to get the best performance."},{"line_number":41,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"3a98d1f4_45f0abc9","line":38,"updated":"2016-09-30 16:37:41.000000000","message":"s/which CPU\\MEM on/which the CP/MEM is on/","commit_id":"884d295d4abb353a27d6ee6e3fb03e50b0487f0c"},{"author":{"_account_id":17973,"name":"Ludovic Beliveau","email":"lbeliveau@gmail.com","username":"lbeliveau"},"change_message_id":"7ade1f9426481036f05dc8afbcb4510432341baa","unresolved":false,"context_lines":[{"line_number":50,"context_line":"different NUMA nodes will be able to use all of their hardware"},{"line_number":51,"context_line":"resources (CPUs, RAM). Also if some compute nodes don\u0027t report NUMA node info"},{"line_number":52,"context_line":"for pci devices, users will be sure that they use only PCI devices placed on"},{"line_number":53,"context_line":"the NUMA node which CPU\\MEM on."},{"line_number":54,"context_line":""},{"line_number":55,"context_line":"Proposed change"},{"line_number":56,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":9,"id":"3a98d1f4_6572074b","line":53,"updated":"2016-09-30 16:37:41.000000000","message":"This is not a good description of the use case per se.  This is more of a description of the problem.","commit_id":"884d295d4abb353a27d6ee6e3fb03e50b0487f0c"},{"author":{"_account_id":17973,"name":"Ludovic Beliveau","email":"lbeliveau@gmail.com","username":"lbeliveau"},"change_message_id":"7ade1f9426481036f05dc8afbcb4510432341baa","unresolved":false,"context_lines":[{"line_number":67,"context_line":"topology is provided nova will boot VMs with PCI devices if at least one"},{"line_number":68,"context_line":"of the NUMA nodes associated with these PCI devices. However, if NUMA node info"},{"line_number":69,"context_line":"for pci devices isn\u0027t provided these PCI devices will be used too. This is the"},{"line_number":70,"context_line":"current behaviour."},{"line_number":71,"context_line":""},{"line_number":72,"context_line":"The \u0027required\u0027 value will mean that nova will boot VMs with PCI devices *only*"},{"line_number":73,"context_line":"if at least one of the NUMA nodes associated with these PCI devices. It means"}],"source_content_type":"text/x-rst","patch_set":9,"id":"3a98d1f4_e50ff785","line":70,"updated":"2016-09-30 16:37:41.000000000","message":"Please rephrase.","commit_id":"884d295d4abb353a27d6ee6e3fb03e50b0487f0c"},{"author":{"_account_id":17973,"name":"Ludovic Beliveau","email":"lbeliveau@gmail.com","username":"lbeliveau"},"change_message_id":"c07d0d41232994a53040083be3425d292b446b61","unresolved":false,"context_lines":[{"line_number":47,"context_line":""},{"line_number":48,"context_line":"Users will get more control over scheduling of VMs with PCI devices. They would"},{"line_number":49,"context_line":"be able to choose between the performance of such VMs and maximum number of"},{"line_number":50,"context_line":"booted VMs."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"Proposed change"},{"line_number":53,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":10,"id":"3a98d1f4_90e1a49d","line":50,"updated":"2016-10-04 12:41:38.000000000","message":"Use case should be something like this: A user wants to boot a guest with PCI devices and numa co-locality of those devices with the vcpus is not important for him (even if performance will be less) ...","commit_id":"82d6971d62efc877dedd5c9f1aab3f694fb02c34"},{"author":{"_account_id":17973,"name":"Ludovic Beliveau","email":"lbeliveau@gmail.com","username":"lbeliveau"},"change_message_id":"3399d962655d0d0d9e9beefb09a29d78298c5688","unresolved":false,"context_lines":[{"line_number":65,"context_line":"  devices on said hosts \"belong to all NUMA nodes\". The ``preferred`` value"},{"line_number":66,"context_line":"  will mean that nova will boot VMs with PCI devices if:"},{"line_number":67,"context_line":""},{"line_number":68,"context_line":"  * the PCI device is associated with least one NUMA nodes on which the"},{"line_number":69,"context_line":"    instance will be booted"},{"line_number":70,"context_line":"  * there is no information about PCI-NUMA association"},{"line_number":71,"context_line":""}],"source_content_type":"text/x-rst","patch_set":12,"id":"1a95cdbc_331453f4","line":68,"updated":"2016-10-04 17:19:08.000000000","message":"nit: s/with least/with at least/","commit_id":"29a10e2cf1b08419e4bb179e414feea15239f422"},{"author":{"_account_id":12171,"name":"Moshe Levi","email":"moshele@nvidia.com","username":"moshele"},"change_message_id":"7b6791ad1f243e0405f20e2348b7c2ae7fe0707c","unresolved":false,"context_lines":[{"line_number":193,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":194,"context_line":""},{"line_number":195,"context_line":"This feature will not add a new scheduling filter, but it will change the"},{"line_number":196,"context_line":"behaviour of NUMATopologyFilter. We should add documentation to describe new"},{"line_number":197,"context_line":"config option."},{"line_number":198,"context_line":""},{"line_number":199,"context_line":"References"},{"line_number":200,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":13,"id":"1a95cdbc_a5ec86c8","line":197,"range":{"start_line":196,"start_character":32,"end_line":197,"end_character":14},"updated":"2016-10-05 05:48:38.000000000","message":"what config option? is not the new option \"pci_numa_policy\" is in the flavor extra spec?","commit_id":"d4ce56c16b5fe5775fc6e529bf4e9aecf8bb474e"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"1459a011a320d3488080a377df257fdeb219c4a1","unresolved":false,"context_lines":[{"line_number":193,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":194,"context_line":""},{"line_number":195,"context_line":"This feature will not add a new scheduling filter, but it will change the"},{"line_number":196,"context_line":"behaviour of NUMATopologyFilter. We should add documentation to describe new"},{"line_number":197,"context_line":"config option."},{"line_number":198,"context_line":""},{"line_number":199,"context_line":"References"},{"line_number":200,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":13,"id":"1a95cdbc_d35eb844","line":197,"range":{"start_line":196,"start_character":32,"end_line":197,"end_character":14},"in_reply_to":"1a95cdbc_a5ec86c8","updated":"2016-10-05 07:48:00.000000000","message":"forgot to change it :(","commit_id":"d4ce56c16b5fe5775fc6e529bf4e9aecf8bb474e"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"702c6e48fe7cbd3a0cc367c345d6f2f564ef0842","unresolved":false,"context_lines":[{"line_number":14,"context_line":"scheduling algorithm allows users to boot VM with PCI devices *only* if at"},{"line_number":15,"context_line":"least one of the NUMA nodes is associated with the PCI devices. Before this,"},{"line_number":16,"context_line":"nova booted VMs with PCI devices without checking NUMA nodes. However, such"},{"line_number":17,"context_line":"hardcoded behaviour causes problems if not every NUMA node has its own PCI"},{"line_number":18,"context_line":"device. In this case nova wouldn\u0027t allow to boot a VM on NUMA nodes without PCI"},{"line_number":19,"context_line":"devices."},{"line_number":20,"context_line":""}],"source_content_type":"text/x-rst","patch_set":14,"id":"3a71b18c_8183b2cb","line":17,"range":{"start_line":17,"start_character":0,"end_line":17,"end_character":19},"updated":"2016-12-09 16:17:29.000000000","message":"the original juno spec was intended to provide the same pci numa affinity policy discussed below but it was not implented in kilo due review and time constrains. at the end of the kilo cycle the developer who was leading the work on this work left our team which is why this was never added liberty.","commit_id":"46061d3c524473c9d383bd7e2b90dba6cda280ca"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"f7cfab35ab7328788b4270a77af53cc33df90951","unresolved":false,"context_lines":[{"line_number":14,"context_line":"scheduling algorithm allows users to boot VM with PCI devices *only* if at"},{"line_number":15,"context_line":"least one of the NUMA nodes is associated with the PCI devices. Before this,"},{"line_number":16,"context_line":"nova booted VMs with PCI devices without checking NUMA nodes. However, such"},{"line_number":17,"context_line":"hardcoded behaviour causes problems if not every NUMA node has its own PCI"},{"line_number":18,"context_line":"device. In this case nova wouldn\u0027t allow to boot a VM on NUMA nodes without PCI"},{"line_number":19,"context_line":"devices."},{"line_number":20,"context_line":""}],"source_content_type":"text/x-rst","patch_set":14,"id":"1a430d35_ec93d54d","line":17,"range":{"start_line":17,"start_character":0,"end_line":17,"end_character":19},"in_reply_to":"3a71b18c_8183b2cb","updated":"2017-02-03 13:30:14.000000000","message":"thanks for information!","commit_id":"46061d3c524473c9d383bd7e2b90dba6cda280ca"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"702c6e48fe7cbd3a0cc367c345d6f2f564ef0842","unresolved":false,"context_lines":[{"line_number":30,"context_line":"boot VMs with *one* NUMA node and SR-IOV ports only on the first NUMA node. In"},{"line_number":31,"context_line":"this case, the user cannot use half of the CPUs and RAM because these resources"},{"line_number":32,"context_line":"are placed on second NUMA node. The user should be able to boot VMs on"},{"line_number":33,"context_line":"different NUMA nodes, even if it makes performance on second NUMA node worse."},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"In addition, the current behavior doesn\u0027t always provide the best performance"},{"line_number":36,"context_line":"solution. If some compute nodes don\u0027t report NUMA node info for PCI devices,"}],"source_content_type":"text/x-rst","patch_set":14,"id":"3a71b18c_8c41290e","line":33,"range":{"start_line":33,"start_character":22,"end_line":33,"end_character":77},"updated":"2016-12-09 16:17:29.000000000","message":"actully it makes the performace of both numanodes worse as both sockets will contend for bandwith on the same set of pcie lanes form the first socket. \n\nit also requires the the first socket to process traffic from socket 2 via that qpi bus. there is deicated hardware for this but it does increase the power consumtion on the first socket which raises the temperature and lower the maxium susstained clock rate when turbo boost is enabled as a result\nof reaching termal and power limits sooner. \n\nfortunetly this is a minimal effect but my point is that the perfromace impact is not configed to workloads on the remote sockets.","commit_id":"46061d3c524473c9d383bd7e2b90dba6cda280ca"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"f7cfab35ab7328788b4270a77af53cc33df90951","unresolved":false,"context_lines":[{"line_number":30,"context_line":"boot VMs with *one* NUMA node and SR-IOV ports only on the first NUMA node. In"},{"line_number":31,"context_line":"this case, the user cannot use half of the CPUs and RAM because these resources"},{"line_number":32,"context_line":"are placed on second NUMA node. The user should be able to boot VMs on"},{"line_number":33,"context_line":"different NUMA nodes, even if it makes performance on second NUMA node worse."},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"In addition, the current behavior doesn\u0027t always provide the best performance"},{"line_number":36,"context_line":"solution. If some compute nodes don\u0027t report NUMA node info for PCI devices,"}],"source_content_type":"text/x-rst","patch_set":14,"id":"1a430d35_4cca2951","line":33,"range":{"start_line":33,"start_character":22,"end_line":33,"end_character":77},"in_reply_to":"3a71b18c_8c41290e","updated":"2017-02-03 13:30:14.000000000","message":"Done","commit_id":"46061d3c524473c9d383bd7e2b90dba6cda280ca"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"702c6e48fe7cbd3a0cc367c345d6f2f564ef0842","unresolved":false,"context_lines":[{"line_number":33,"context_line":"different NUMA nodes, even if it makes performance on second NUMA node worse."},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"In addition, the current behavior doesn\u0027t always provide the best performance"},{"line_number":36,"context_line":"solution. If some compute nodes don\u0027t report NUMA node info for PCI devices,"},{"line_number":37,"context_line":"then these PCI devices will be regarded as \"belonging to all NUMA node\". This"},{"line_number":38,"context_line":"can lead to a situation whereby PCI devices are not on the NUMA node which the"},{"line_number":39,"context_line":"CPU\\RAM is on. In this way, the real purpose of \"I/O (PCIe) based NUMA"}],"source_content_type":"text/x-rst","patch_set":14,"id":"3a71b18c_a7b7def3","line":36,"range":{"start_line":36,"start_character":10,"end_line":36,"end_character":75},"updated":"2016-12-09 16:17:29.000000000","message":"i feel i should also point out that this is not an all or nothing case.\n\nit is common for host to report nunma affinity for some devics and no numa infomation for others. as such pci numa report really is not a host specific atibute its a devices specficif attribute which may or may not be populated per deivce on a host.","commit_id":"46061d3c524473c9d383bd7e2b90dba6cda280ca"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"f7cfab35ab7328788b4270a77af53cc33df90951","unresolved":false,"context_lines":[{"line_number":33,"context_line":"different NUMA nodes, even if it makes performance on second NUMA node worse."},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"In addition, the current behavior doesn\u0027t always provide the best performance"},{"line_number":36,"context_line":"solution. If some compute nodes don\u0027t report NUMA node info for PCI devices,"},{"line_number":37,"context_line":"then these PCI devices will be regarded as \"belonging to all NUMA node\". This"},{"line_number":38,"context_line":"can lead to a situation whereby PCI devices are not on the NUMA node which the"},{"line_number":39,"context_line":"CPU\\RAM is on. In this way, the real purpose of \"I/O (PCIe) based NUMA"}],"source_content_type":"text/x-rst","patch_set":14,"id":"1a430d35_46b045de","line":36,"range":{"start_line":36,"start_character":10,"end_line":36,"end_character":75},"in_reply_to":"3a71b18c_a7b7def3","updated":"2017-02-03 13:30:14.000000000","message":"Done","commit_id":"46061d3c524473c9d383bd7e2b90dba6cda280ca"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"702c6e48fe7cbd3a0cc367c345d6f2f564ef0842","unresolved":false,"context_lines":[{"line_number":34,"context_line":""},{"line_number":35,"context_line":"In addition, the current behavior doesn\u0027t always provide the best performance"},{"line_number":36,"context_line":"solution. If some compute nodes don\u0027t report NUMA node info for PCI devices,"},{"line_number":37,"context_line":"then these PCI devices will be regarded as \"belonging to all NUMA node\". This"},{"line_number":38,"context_line":"can lead to a situation whereby PCI devices are not on the NUMA node which the"},{"line_number":39,"context_line":"CPU\\RAM is on. In this way, the real purpose of \"I/O (PCIe) based NUMA"},{"line_number":40,"context_line":"Scheduling\" spec is not reached. The user should be able to ensure this does"}],"source_content_type":"text/x-rst","patch_set":14,"id":"3a71b18c_813bb2ea","line":37,"range":{"start_line":37,"start_character":43,"end_line":37,"end_character":71},"updated":"2016-12-09 16:17:29.000000000","message":"actully they are treated as beloning to no numa node. \n\nthere are some pci devices that are connected to multiple numanodes concurently so there is a disticiiton between a deivice with no numa affinity and ad device that has multiple numa affintiy as implied by all numa nodes.\n\ni belive the netronome smart nics are one such example.","commit_id":"46061d3c524473c9d383bd7e2b90dba6cda280ca"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"f7cfab35ab7328788b4270a77af53cc33df90951","unresolved":false,"context_lines":[{"line_number":34,"context_line":""},{"line_number":35,"context_line":"In addition, the current behavior doesn\u0027t always provide the best performance"},{"line_number":36,"context_line":"solution. If some compute nodes don\u0027t report NUMA node info for PCI devices,"},{"line_number":37,"context_line":"then these PCI devices will be regarded as \"belonging to all NUMA node\". This"},{"line_number":38,"context_line":"can lead to a situation whereby PCI devices are not on the NUMA node which the"},{"line_number":39,"context_line":"CPU\\RAM is on. In this way, the real purpose of \"I/O (PCIe) based NUMA"},{"line_number":40,"context_line":"Scheduling\" spec is not reached. The user should be able to ensure this does"}],"source_content_type":"text/x-rst","patch_set":14,"id":"1a430d35_6c8605bb","line":37,"range":{"start_line":37,"start_character":43,"end_line":37,"end_character":71},"in_reply_to":"3a71b18c_813bb2ea","updated":"2017-02-03 13:30:14.000000000","message":"\"they are treated as beloning to no numa node.\" Yes, from hardware point of view it\u0027s true. But in this case Nova think that such devices \"belong to all NUMA node\" and use them for VMs from any NUMA node.\n\nhttps://github.com/openstack/nova/blob/master/nova/pci/stats.py#L215-L219","commit_id":"46061d3c524473c9d383bd7e2b90dba6cda280ca"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"702c6e48fe7cbd3a0cc367c345d6f2f564ef0842","unresolved":false,"context_lines":[{"line_number":40,"context_line":"Scheduling\" spec is not reached. The user should be able to ensure this does"},{"line_number":41,"context_line":"not happen and guarantee the best performance."},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"Of course this ability should be configurable and the current scheduling"},{"line_number":44,"context_line":"behaviour must be default."},{"line_number":45,"context_line":""},{"line_number":46,"context_line":"Use Cases"},{"line_number":47,"context_line":"---------"}],"source_content_type":"text/x-rst","patch_set":14,"id":"3a71b18c_ac9c255e","line":44,"range":{"start_line":43,"start_character":0,"end_line":44,"end_character":26},"updated":"2016-12-09 16:17:29.000000000","message":"+1\n\nthe current behavior is to require numa affinity when device numa info is available and to allow devices with no numa info to allways be used.","commit_id":"46061d3c524473c9d383bd7e2b90dba6cda280ca"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"702c6e48fe7cbd3a0cc367c345d6f2f564ef0842","unresolved":false,"context_lines":[{"line_number":54,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":55,"context_line":""},{"line_number":56,"context_line":"This spec is needed to decide which PCI device will be used by new VM. To this"},{"line_number":57,"context_line":"end, we will add a new flavor extra spec ``hw:pci_numa_policy`` and image"},{"line_number":58,"context_line":"metadata ``hw_pci_numa_policy``. They will have one of three values."},{"line_number":59,"context_line":""},{"line_number":60,"context_line":"**preferred**"}],"source_content_type":"text/x-rst","patch_set":14,"id":"3a71b18c_6785c657","line":57,"range":{"start_line":57,"start_character":43,"end_line":57,"end_character":61},"updated":"2016-12-09 16:17:29.000000000","message":"hw:pci_numa_affinity_policy","commit_id":"46061d3c524473c9d383bd7e2b90dba6cda280ca"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"f7cfab35ab7328788b4270a77af53cc33df90951","unresolved":false,"context_lines":[{"line_number":54,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":55,"context_line":""},{"line_number":56,"context_line":"This spec is needed to decide which PCI device will be used by new VM. To this"},{"line_number":57,"context_line":"end, we will add a new flavor extra spec ``hw:pci_numa_policy`` and image"},{"line_number":58,"context_line":"metadata ``hw_pci_numa_policy``. They will have one of three values."},{"line_number":59,"context_line":""},{"line_number":60,"context_line":"**preferred**"}],"source_content_type":"text/x-rst","patch_set":14,"id":"1a430d35_574b922b","line":57,"range":{"start_line":57,"start_character":43,"end_line":57,"end_character":61},"in_reply_to":"3a71b18c_6785c657","updated":"2017-02-03 13:30:14.000000000","message":"Done","commit_id":"46061d3c524473c9d383bd7e2b90dba6cda280ca"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"702c6e48fe7cbd3a0cc367c345d6f2f564ef0842","unresolved":false,"context_lines":[{"line_number":55,"context_line":""},{"line_number":56,"context_line":"This spec is needed to decide which PCI device will be used by new VM. To this"},{"line_number":57,"context_line":"end, we will add a new flavor extra spec ``hw:pci_numa_policy`` and image"},{"line_number":58,"context_line":"metadata ``hw_pci_numa_policy``. They will have one of three values."},{"line_number":59,"context_line":""},{"line_number":60,"context_line":"**preferred**"},{"line_number":61,"context_line":""}],"source_content_type":"text/x-rst","patch_set":14,"id":"3a71b18c_07917299","line":58,"range":{"start_line":58,"start_character":9,"end_line":58,"end_character":31},"updated":"2016-12-09 16:17:29.000000000","message":"hw_pci_numa_affinity_policy","commit_id":"46061d3c524473c9d383bd7e2b90dba6cda280ca"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"f7cfab35ab7328788b4270a77af53cc33df90951","unresolved":false,"context_lines":[{"line_number":55,"context_line":""},{"line_number":56,"context_line":"This spec is needed to decide which PCI device will be used by new VM. To this"},{"line_number":57,"context_line":"end, we will add a new flavor extra spec ``hw:pci_numa_policy`` and image"},{"line_number":58,"context_line":"metadata ``hw_pci_numa_policy``. They will have one of three values."},{"line_number":59,"context_line":""},{"line_number":60,"context_line":"**preferred**"},{"line_number":61,"context_line":""}],"source_content_type":"text/x-rst","patch_set":14,"id":"1a430d35_77488e2e","line":58,"range":{"start_line":58,"start_character":9,"end_line":58,"end_character":31},"in_reply_to":"3a71b18c_07917299","updated":"2017-02-03 13:30:14.000000000","message":"Done","commit_id":"46061d3c524473c9d383bd7e2b90dba6cda280ca"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"702c6e48fe7cbd3a0cc367c345d6f2f564ef0842","unresolved":false,"context_lines":[{"line_number":62,"context_line":"  This is the default value and describes the current nova behavior. Usually"},{"line_number":63,"context_line":"  compute hosts provide information about association of PCI devices with NUMA"},{"line_number":64,"context_line":"  nodes. However, some compute hosts does not provide such information and PCI"},{"line_number":65,"context_line":"  devices on said hosts \"belong to all NUMA nodes\". The ``preferred`` value"},{"line_number":66,"context_line":"  will mean that nova will boot VMs with PCI devices if:"},{"line_number":67,"context_line":""},{"line_number":68,"context_line":"  * the PCI device is associated with at least one NUMA nodes on which the"}],"source_content_type":"text/x-rst","patch_set":14,"id":"3a71b18c_67d30660","line":65,"range":{"start_line":65,"start_character":35,"end_line":65,"end_character":38},"updated":"2016-12-09 16:17:29.000000000","message":"no","commit_id":"46061d3c524473c9d383bd7e2b90dba6cda280ca"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"702c6e48fe7cbd3a0cc367c345d6f2f564ef0842","unresolved":false,"context_lines":[{"line_number":57,"context_line":"end, we will add a new flavor extra spec ``hw:pci_numa_policy`` and image"},{"line_number":58,"context_line":"metadata ``hw_pci_numa_policy``. They will have one of three values."},{"line_number":59,"context_line":""},{"line_number":60,"context_line":"**preferred**"},{"line_number":61,"context_line":""},{"line_number":62,"context_line":"  This is the default value and describes the current nova behavior. Usually"},{"line_number":63,"context_line":"  compute hosts provide information about association of PCI devices with NUMA"},{"line_number":64,"context_line":"  nodes. However, some compute hosts does not provide such information and PCI"},{"line_number":65,"context_line":"  devices on said hosts \"belong to all NUMA nodes\". The ``preferred`` value"},{"line_number":66,"context_line":"  will mean that nova will boot VMs with PCI devices if:"},{"line_number":67,"context_line":""},{"line_number":68,"context_line":"  * the PCI device is associated with at least one NUMA nodes on which the"},{"line_number":69,"context_line":"    instance will be booted"},{"line_number":70,"context_line":"  * there is no information about PCI-NUMA association"},{"line_number":71,"context_line":""},{"line_number":72,"context_line":"**required**"},{"line_number":73,"context_line":""}],"source_content_type":"text/x-rst","patch_set":14,"id":"3a71b18c_ccf8817f","line":70,"range":{"start_line":60,"start_character":0,"end_line":70,"end_character":54},"updated":"2016-12-09 16:17:29.000000000","message":"this defffintion of preferred is what i whould have defined as requried.\n\ni would have defiend preferred as first trying to affinties to the same numa node and then falling back to a defice from a different numa node if that was not possible.","commit_id":"46061d3c524473c9d383bd7e2b90dba6cda280ca"},{"author":{"_account_id":7730,"name":"Sahid Orentino Ferdjaoui","email":"sahid.ferdjaoui@industrialdiscipline.com","username":"sahid"},"change_message_id":"a10220ea605df9df793cdfe3605e690810ff98a0","unresolved":false,"context_lines":[{"line_number":76,"context_line":"  if NUMA node info for some pci devices wouldn\u0027t be provided these PCI devices"},{"line_number":77,"context_line":"  wouldn\u0027t be used. This is a maximum performance behavior."},{"line_number":78,"context_line":""},{"line_number":79,"context_line":"**shared**"},{"line_number":80,"context_line":""},{"line_number":81,"context_line":"  This value will mean that scheduler will choose a compute host ignoring"},{"line_number":82,"context_line":"  association of NUMA nodes and PCI devices. It means that, if host complies"}],"source_content_type":"text/x-rst","patch_set":14,"id":"ba5da102_49f7668c","line":79,"updated":"2016-11-04 13:26:38.000000000","message":"I don\u0027t think shared make a lot of sense, we should just merge it with preferred. It adds complexity without gain.\n\n  **preferred**\n\n  - no guest numa topology: current behavior\n  - guest numa topology: try to do best effort to fit devices, if not able so fallback to: no guest numa topology\n\n  **required**\n\n  - no guest numa topology: ko, raise exception\n  - guest numa topology: fit devices or ko\n\n\nWith: \"fit devices\" all PCI devices requested should be related to the host NUMA nodes which refer guest NUMA nodes.","commit_id":"46061d3c524473c9d383bd7e2b90dba6cda280ca"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"702c6e48fe7cbd3a0cc367c345d6f2f564ef0842","unresolved":false,"context_lines":[{"line_number":76,"context_line":"  if NUMA node info for some pci devices wouldn\u0027t be provided these PCI devices"},{"line_number":77,"context_line":"  wouldn\u0027t be used. This is a maximum performance behavior."},{"line_number":78,"context_line":""},{"line_number":79,"context_line":"**shared**"},{"line_number":80,"context_line":""},{"line_number":81,"context_line":"  This value will mean that scheduler will choose a compute host ignoring"},{"line_number":82,"context_line":"  association of NUMA nodes and PCI devices. It means that, if host complies"}],"source_content_type":"text/x-rst","patch_set":14,"id":"3a71b18c_21c49e5d","line":79,"range":{"start_line":79,"start_character":0,"end_line":79,"end_character":10},"updated":"2016-12-09 16:17:29.000000000","message":"on this point i dont thing the value \"shared\" makes grapmatical sense. the value is for the pci numa affinity policy. in that context reqired and perferred both intuitvly indicate the behavior that is disired without requiredin the enduser to read the spec, that is not the case with shared. i would suggest changing this value to any.","commit_id":"46061d3c524473c9d383bd7e2b90dba6cda280ca"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"f7cfab35ab7328788b4270a77af53cc33df90951","unresolved":false,"context_lines":[{"line_number":76,"context_line":"  if NUMA node info for some pci devices wouldn\u0027t be provided these PCI devices"},{"line_number":77,"context_line":"  wouldn\u0027t be used. This is a maximum performance behavior."},{"line_number":78,"context_line":""},{"line_number":79,"context_line":"**shared**"},{"line_number":80,"context_line":""},{"line_number":81,"context_line":"  This value will mean that scheduler will choose a compute host ignoring"},{"line_number":82,"context_line":"  association of NUMA nodes and PCI devices. It means that, if host complies"}],"source_content_type":"text/x-rst","patch_set":14,"id":"1a430d35_26bb39c6","line":79,"range":{"start_line":79,"start_character":0,"end_line":79,"end_character":10},"in_reply_to":"3a71b18c_21c49e5d","updated":"2017-02-03 13:30:14.000000000","message":"This spec wasn\u0027t updated for a long time. So let discuss policy names in new revision","commit_id":"46061d3c524473c9d383bd7e2b90dba6cda280ca"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"ef69d9fe3e816fcefb43bc8e9d9ac57212cd51b4","unresolved":false,"context_lines":[{"line_number":76,"context_line":"  if NUMA node info for some pci devices wouldn\u0027t be provided these PCI devices"},{"line_number":77,"context_line":"  wouldn\u0027t be used. This is a maximum performance behavior."},{"line_number":78,"context_line":""},{"line_number":79,"context_line":"**shared**"},{"line_number":80,"context_line":""},{"line_number":81,"context_line":"  This value will mean that scheduler will choose a compute host ignoring"},{"line_number":82,"context_line":"  association of NUMA nodes and PCI devices. It means that, if host complies"}],"source_content_type":"text/x-rst","patch_set":14,"id":"ba5da102_5f246dc5","line":79,"in_reply_to":"ba5da102_1faf557e","updated":"2016-11-04 16:40:16.000000000","message":"Stephen, +1. It will change current behavior.","commit_id":"46061d3c524473c9d383bd7e2b90dba6cda280ca"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"f81d694fd07aa3d5a2ba797c172d659d7fbe0136","unresolved":false,"context_lines":[{"line_number":76,"context_line":"  if NUMA node info for some pci devices wouldn\u0027t be provided these PCI devices"},{"line_number":77,"context_line":"  wouldn\u0027t be used. This is a maximum performance behavior."},{"line_number":78,"context_line":""},{"line_number":79,"context_line":"**shared**"},{"line_number":80,"context_line":""},{"line_number":81,"context_line":"  This value will mean that scheduler will choose a compute host ignoring"},{"line_number":82,"context_line":"  association of NUMA nodes and PCI devices. It means that, if host complies"}],"source_content_type":"text/x-rst","patch_set":14,"id":"ba5da102_1faf557e","line":79,"in_reply_to":"ba5da102_49f7668c","updated":"2016-11-04 16:13:31.000000000","message":"I don\u0027t think we can do this, though. This will change behavior for existing users, who will suddenly not be getting the guarantee of PCI-NUMA affinity they previously had","commit_id":"46061d3c524473c9d383bd7e2b90dba6cda280ca"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"022b19f7056aaacbf684a660ac05c71c6817b7a1","unresolved":false,"context_lines":[{"line_number":90,"context_line":"  is only when this fails that nova compute will fall back to scheduling on a"},{"line_number":91,"context_line":"  NUMA nodes that is not associated with the PCI device."},{"line_number":92,"context_line":""},{"line_number":93,"context_line":"If both image and flavor properties are not set (equals \u0027None\u0027) the"},{"line_number":94,"context_line":"``preferred`` policy will be used. If one of image *or* flavor property is not"},{"line_number":95,"context_line":"set (equals \u0027None\u0027) but the other property is set then the value of the set"},{"line_number":96,"context_line":"property will be used. In a case of conflicts between flavor and image"},{"line_number":97,"context_line":"properties (both properties are set and they are not equal) an exception will"},{"line_number":98,"context_line":"be raised."},{"line_number":99,"context_line":""},{"line_number":100,"context_line":"Alternatives"},{"line_number":101,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":14,"id":"ba5da102_38719e78","line":98,"range":{"start_line":93,"start_character":0,"end_line":98,"end_character":10},"updated":"2016-11-04 11:48:29.000000000","message":"nit: I get what you\u0027re saying, but I still don\u0027t think it\u0027s possible to have the image metadata not set. IIUC, it will default to \u0027preferred\u0027 as the default option?","commit_id":"46061d3c524473c9d383bd7e2b90dba6cda280ca"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"f7cfab35ab7328788b4270a77af53cc33df90951","unresolved":false,"context_lines":[{"line_number":90,"context_line":"  is only when this fails that nova compute will fall back to scheduling on a"},{"line_number":91,"context_line":"  NUMA nodes that is not associated with the PCI device."},{"line_number":92,"context_line":""},{"line_number":93,"context_line":"If both image and flavor properties are not set (equals \u0027None\u0027) the"},{"line_number":94,"context_line":"``preferred`` policy will be used. If one of image *or* flavor property is not"},{"line_number":95,"context_line":"set (equals \u0027None\u0027) but the other property is set then the value of the set"},{"line_number":96,"context_line":"property will be used. In a case of conflicts between flavor and image"},{"line_number":97,"context_line":"properties (both properties are set and they are not equal) an exception will"},{"line_number":98,"context_line":"be raised."},{"line_number":99,"context_line":""},{"line_number":100,"context_line":"Alternatives"},{"line_number":101,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":14,"id":"1a430d35_72233c06","line":98,"range":{"start_line":93,"start_character":0,"end_line":98,"end_character":10},"in_reply_to":"3a71b18c_6750660b","updated":"2017-02-03 13:30:14.000000000","message":"yes, this exception will be handled by client and the end user will see it","commit_id":"46061d3c524473c9d383bd7e2b90dba6cda280ca"},{"author":{"_account_id":8768,"name":"Chris Friesen","email":"chris.friesen@windriver.com","username":"cbf123"},"change_message_id":"f6ded700e142f727ad0bf9dcab669071c738808c","unresolved":false,"context_lines":[{"line_number":90,"context_line":"  is only when this fails that nova compute will fall back to scheduling on a"},{"line_number":91,"context_line":"  NUMA nodes that is not associated with the PCI device."},{"line_number":92,"context_line":""},{"line_number":93,"context_line":"If both image and flavor properties are not set (equals \u0027None\u0027) the"},{"line_number":94,"context_line":"``preferred`` policy will be used. If one of image *or* flavor property is not"},{"line_number":95,"context_line":"set (equals \u0027None\u0027) but the other property is set then the value of the set"},{"line_number":96,"context_line":"property will be used. In a case of conflicts between flavor and image"},{"line_number":97,"context_line":"properties (both properties are set and they are not equal) an exception will"},{"line_number":98,"context_line":"be raised."},{"line_number":99,"context_line":""},{"line_number":100,"context_line":"Alternatives"},{"line_number":101,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":14,"id":"7a77a97e_7f1dc3bc","line":98,"range":{"start_line":93,"start_character":0,"end_line":98,"end_character":10},"in_reply_to":"7a77a97e_19686fba","updated":"2016-11-14 19:29:18.000000000","message":"I don\u0027t think this is a big deal.  The only way that the image could increase QPI traffic is if the provider has not stated a preference.  Even then, we would still use PCI devices on the same NUMA node as the instance if they\u0027re available.  The \"shared\" option simply means \"I care more about getting the instance up at all than I do about getting the best performance.\"","commit_id":"46061d3c524473c9d383bd7e2b90dba6cda280ca"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"702c6e48fe7cbd3a0cc367c345d6f2f564ef0842","unresolved":false,"context_lines":[{"line_number":90,"context_line":"  is only when this fails that nova compute will fall back to scheduling on a"},{"line_number":91,"context_line":"  NUMA nodes that is not associated with the PCI device."},{"line_number":92,"context_line":""},{"line_number":93,"context_line":"If both image and flavor properties are not set (equals \u0027None\u0027) the"},{"line_number":94,"context_line":"``preferred`` policy will be used. If one of image *or* flavor property is not"},{"line_number":95,"context_line":"set (equals \u0027None\u0027) but the other property is set then the value of the set"},{"line_number":96,"context_line":"property will be used. In a case of conflicts between flavor and image"},{"line_number":97,"context_line":"properties (both properties are set and they are not equal) an exception will"},{"line_number":98,"context_line":"be raised."},{"line_number":99,"context_line":""},{"line_number":100,"context_line":"Alternatives"},{"line_number":101,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":14,"id":"3a71b18c_6750660b","line":98,"range":{"start_line":93,"start_character":0,"end_line":98,"end_character":10},"in_reply_to":"7a77a97e_7f1dc3bc","updated":"2016-12-09 16:17:29.000000000","message":"so on this topic in general what is nova policy on image vs flavor. in kilo/juno the policy seemed to be if there is a missmatch then the flavor value is uses as the operator always had the last say. things seam to be moving to if\nthere is a mismatch then an exception is raised.\n\ncan someone point me to where this is defined.\nif this is not consitent for all cases not jsut this feature\nit will be an usablity nightmare.\n\npersonally i always viewed the flavor as expressing the hard constraintes and the image properties as expressing soft constraintes. that said  i see the merits of failing fast and raising the exception but only if that excetion is propergated all the way back via the api to the client. if it is mearly caught and loged then i dont think it is the right approch.\n\nwe discussed this breilfy in the code review but i wanted to raise it here also.","commit_id":"46061d3c524473c9d383bd7e2b90dba6cda280ca"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"ef69d9fe3e816fcefb43bc8e9d9ac57212cd51b4","unresolved":false,"context_lines":[{"line_number":90,"context_line":"  is only when this fails that nova compute will fall back to scheduling on a"},{"line_number":91,"context_line":"  NUMA nodes that is not associated with the PCI device."},{"line_number":92,"context_line":""},{"line_number":93,"context_line":"If both image and flavor properties are not set (equals \u0027None\u0027) the"},{"line_number":94,"context_line":"``preferred`` policy will be used. If one of image *or* flavor property is not"},{"line_number":95,"context_line":"set (equals \u0027None\u0027) but the other property is set then the value of the set"},{"line_number":96,"context_line":"property will be used. In a case of conflicts between flavor and image"},{"line_number":97,"context_line":"properties (both properties are set and they are not equal) an exception will"},{"line_number":98,"context_line":"be raised."},{"line_number":99,"context_line":""},{"line_number":100,"context_line":"Alternatives"},{"line_number":101,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":14,"id":"ba5da102_d1acf18e","line":98,"range":{"start_line":93,"start_character":0,"end_line":98,"end_character":10},"in_reply_to":"ba5da102_38719e78","updated":"2016-11-04 16:40:16.000000000","message":"Correct me if I\u0027m wrong but as I understand image metadata fields has no default values and if some option is not set, \u0027None\u0027 value will be returned.\n\nhttps://github.com/openstack/nova/blob/master/nova/objects/image_meta.py#L557-L579","commit_id":"46061d3c524473c9d383bd7e2b90dba6cda280ca"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"41eba4994bb5f13f2b5130aca05c8916896b0888","unresolved":false,"context_lines":[{"line_number":90,"context_line":"  is only when this fails that nova compute will fall back to scheduling on a"},{"line_number":91,"context_line":"  NUMA nodes that is not associated with the PCI device."},{"line_number":92,"context_line":""},{"line_number":93,"context_line":"If both image and flavor properties are not set (equals \u0027None\u0027) the"},{"line_number":94,"context_line":"``preferred`` policy will be used. If one of image *or* flavor property is not"},{"line_number":95,"context_line":"set (equals \u0027None\u0027) but the other property is set then the value of the set"},{"line_number":96,"context_line":"property will be used. In a case of conflicts between flavor and image"},{"line_number":97,"context_line":"properties (both properties are set and they are not equal) an exception will"},{"line_number":98,"context_line":"be raised."},{"line_number":99,"context_line":""},{"line_number":100,"context_line":"Alternatives"},{"line_number":101,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":14,"id":"ba5da102_f102d50a","line":98,"range":{"start_line":93,"start_character":0,"end_line":98,"end_character":10},"in_reply_to":"ba5da102_d1acf18e","updated":"2016-11-04 16:53:28.000000000","message":"Right you are. My mistake.","commit_id":"46061d3c524473c9d383bd7e2b90dba6cda280ca"},{"author":{"_account_id":3217,"name":"Ian Wells","username":"ijw-ubuntu"},"change_message_id":"2513a2da96c6c95d747318e4bd0cbbe0fb42b2c3","unresolved":false,"context_lines":[{"line_number":90,"context_line":"  is only when this fails that nova compute will fall back to scheduling on a"},{"line_number":91,"context_line":"  NUMA nodes that is not associated with the PCI device."},{"line_number":92,"context_line":""},{"line_number":93,"context_line":"If both image and flavor properties are not set (equals \u0027None\u0027) the"},{"line_number":94,"context_line":"``preferred`` policy will be used. If one of image *or* flavor property is not"},{"line_number":95,"context_line":"set (equals \u0027None\u0027) but the other property is set then the value of the set"},{"line_number":96,"context_line":"property will be used. In a case of conflicts between flavor and image"},{"line_number":97,"context_line":"properties (both properties are set and they are not equal) an exception will"},{"line_number":98,"context_line":"be raised."},{"line_number":99,"context_line":""},{"line_number":100,"context_line":"Alternatives"},{"line_number":101,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":14,"id":"7a77a97e_19686fba","line":98,"range":{"start_line":93,"start_character":0,"end_line":98,"end_character":10},"in_reply_to":"ba5da102_f102d50a","updated":"2016-11-14 18:22:21.000000000","message":"There may be an issue of fairness with the image option.  \n\nUnbound PCI affinity increases traffic over the QPI bus, which impacts all users of the system.  Thus, giving a user control over this allows them to make their workloads more noisy to others and generally detrimental to host operation.  That may be acceptable but someone with a fairly administrative responsibility should decide this.\n\nFlavors are typically set up with an understanding of the hardware and authority to match - frequently by an authorised adminstrator, and, when not, with an understanding of what control is being given up.\n\nImages are more frequently permitted for upload by unauthorised users, so the user is selecting the hardware usage in a way that may affect other tenants if they have control of this setting.","commit_id":"46061d3c524473c9d383bd7e2b90dba6cda280ca"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"702c6e48fe7cbd3a0cc367c345d6f2f564ef0842","unresolved":false,"context_lines":[{"line_number":110,"context_line":"Data model impact"},{"line_number":111,"context_line":"-----------------"},{"line_number":112,"context_line":""},{"line_number":113,"context_line":"New field ``pci_numa_policy`` will be added to the InstanceNUMACell object."},{"line_number":114,"context_line":"Fortunately this object is stored as JSON in DB. So we need no DB migrations"},{"line_number":115,"context_line":"to add new field to this object."},{"line_number":116,"context_line":""}],"source_content_type":"text/x-rst","patch_set":14,"id":"3a71b18c_076332dd","line":113,"range":{"start_line":113,"start_character":12,"end_line":113,"end_character":27},"updated":"2016-12-09 16:17:29.000000000","message":"nit: \"pci_numa_affinity_policy\" would be more descriptive.","commit_id":"46061d3c524473c9d383bd7e2b90dba6cda280ca"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"f7cfab35ab7328788b4270a77af53cc33df90951","unresolved":false,"context_lines":[{"line_number":110,"context_line":"Data model impact"},{"line_number":111,"context_line":"-----------------"},{"line_number":112,"context_line":""},{"line_number":113,"context_line":"New field ``pci_numa_policy`` will be added to the InstanceNUMACell object."},{"line_number":114,"context_line":"Fortunately this object is stored as JSON in DB. So we need no DB migrations"},{"line_number":115,"context_line":"to add new field to this object."},{"line_number":116,"context_line":""}],"source_content_type":"text/x-rst","patch_set":14,"id":"1a430d35_b21da4bf","line":113,"range":{"start_line":113,"start_character":12,"end_line":113,"end_character":27},"in_reply_to":"3a71b18c_076332dd","updated":"2017-02-03 13:30:14.000000000","message":"Done","commit_id":"46061d3c524473c9d383bd7e2b90dba6cda280ca"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"702c6e48fe7cbd3a0cc367c345d6f2f564ef0842","unresolved":false,"context_lines":[{"line_number":139,"context_line":""},{"line_number":140,"context_line":"If described config option will have ``shared`` value the performance of VMs"},{"line_number":141,"context_line":"on NUMA nodes without associated PCI devices will be worse than performance of"},{"line_number":142,"context_line":"VMs on NUMA nodes with associated PCI devices. But with current scheduler"},{"line_number":143,"context_line":"behavior NUMA nodes without associated PCI devices wouldn\u0027t be used at all."},{"line_number":144,"context_line":""},{"line_number":145,"context_line":"If option will have ``preferred`` value nova behaviour will be the same as"},{"line_number":146,"context_line":"current behaviour."}],"source_content_type":"text/x-rst","patch_set":14,"id":"3a71b18c_4737eaaf","line":143,"range":{"start_line":142,"start_character":47,"end_line":143,"end_character":75},"updated":"2016-12-09 16:17:29.000000000","message":"they will if there is no numa node reported for the device requested.","commit_id":"46061d3c524473c9d383bd7e2b90dba6cda280ca"}],"specs/pike/approved/share-pci-between-numa-nodes.rst":[{"author":{"_account_id":8768,"name":"Chris Friesen","email":"chris.friesen@windriver.com","username":"cbf123"},"change_message_id":"8159de7bfd6836eff54a4434c258f969784b376e","unresolved":false,"context_lines":[{"line_number":62,"context_line":""},{"line_number":63,"context_line":"  This is the default value and describes the current nova behavior. Usually"},{"line_number":64,"context_line":"  we have information about association of PCI devices with NUMA nodes."},{"line_number":65,"context_line":"  However, some PCI devices does not provide such information and such devices"},{"line_number":66,"context_line":"  are regarded by nova as \"belong to all NUMA nodes\". The ``preferred`` value"},{"line_number":67,"context_line":"  will mean that nova will boot VMs with PCI devices if:"},{"line_number":68,"context_line":""}],"source_content_type":"text/x-rst","patch_set":15,"id":"1a430d35_58dd7041","line":65,"updated":"2017-02-03 19:56:44.000000000","message":"nit: should be \"some PCI devices do not\"","commit_id":"3d5305ac624fceae76ec7b394795a4e5a297731d"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"37fd3cb0b11f008af0112c22f3e6144cc0949611","unresolved":false,"context_lines":[{"line_number":62,"context_line":""},{"line_number":63,"context_line":"  This is the default value and describes the current nova behavior. Usually"},{"line_number":64,"context_line":"  we have information about association of PCI devices with NUMA nodes."},{"line_number":65,"context_line":"  However, some PCI devices does not provide such information and such devices"},{"line_number":66,"context_line":"  are regarded by nova as \"belong to all NUMA nodes\". The ``preferred`` value"},{"line_number":67,"context_line":"  will mean that nova will boot VMs with PCI devices if:"},{"line_number":68,"context_line":""}],"source_content_type":"text/x-rst","patch_set":15,"id":"1a430d35_f68fe920","line":65,"in_reply_to":"1a430d35_58dd7041","updated":"2017-02-06 07:57:53.000000000","message":"Done","commit_id":"3d5305ac624fceae76ec7b394795a4e5a297731d"},{"author":{"_account_id":8768,"name":"Chris Friesen","email":"chris.friesen@windriver.com","username":"cbf123"},"change_message_id":"8159de7bfd6836eff54a4434c258f969784b376e","unresolved":false,"context_lines":[{"line_number":64,"context_line":"  we have information about association of PCI devices with NUMA nodes."},{"line_number":65,"context_line":"  However, some PCI devices does not provide such information and such devices"},{"line_number":66,"context_line":"  are regarded by nova as \"belong to all NUMA nodes\". The ``preferred`` value"},{"line_number":67,"context_line":"  will mean that nova will boot VMs with PCI devices if:"},{"line_number":68,"context_line":""},{"line_number":69,"context_line":"  * the PCI device is associated with at least one NUMA nodes on which the"},{"line_number":70,"context_line":"    instance will be booted"}],"source_content_type":"text/x-rst","patch_set":15,"id":"1a430d35_5882d001","line":67,"updated":"2017-02-03 19:56:44.000000000","message":"Do both the points below need to be true, or is it sufficient for one of them to be true?  (ie AND or OR)","commit_id":"3d5305ac624fceae76ec7b394795a4e5a297731d"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"37fd3cb0b11f008af0112c22f3e6144cc0949611","unresolved":false,"context_lines":[{"line_number":64,"context_line":"  we have information about association of PCI devices with NUMA nodes."},{"line_number":65,"context_line":"  However, some PCI devices does not provide such information and such devices"},{"line_number":66,"context_line":"  are regarded by nova as \"belong to all NUMA nodes\". The ``preferred`` value"},{"line_number":67,"context_line":"  will mean that nova will boot VMs with PCI devices if:"},{"line_number":68,"context_line":""},{"line_number":69,"context_line":"  * the PCI device is associated with at least one NUMA nodes on which the"},{"line_number":70,"context_line":"    instance will be booted"}],"source_content_type":"text/x-rst","patch_set":15,"id":"1a430d35_b6ae4183","line":67,"in_reply_to":"1a430d35_5882d001","updated":"2017-02-06 07:57:53.000000000","message":"OR","commit_id":"3d5305ac624fceae76ec7b394795a4e5a297731d"},{"author":{"_account_id":8768,"name":"Chris Friesen","email":"chris.friesen@windriver.com","username":"cbf123"},"change_message_id":"8159de7bfd6836eff54a4434c258f969784b376e","unresolved":false,"context_lines":[{"line_number":66,"context_line":"  are regarded by nova as \"belong to all NUMA nodes\". The ``preferred`` value"},{"line_number":67,"context_line":"  will mean that nova will boot VMs with PCI devices if:"},{"line_number":68,"context_line":""},{"line_number":69,"context_line":"  * the PCI device is associated with at least one NUMA nodes on which the"},{"line_number":70,"context_line":"    instance will be booted"},{"line_number":71,"context_line":"  * there is no information about PCI-NUMA association"},{"line_number":72,"context_line":""}],"source_content_type":"text/x-rst","patch_set":15,"id":"1a430d35_18ba08e2","line":69,"updated":"2017-02-03 19:56:44.000000000","message":"nit: should be \"one NUMA node\"","commit_id":"3d5305ac624fceae76ec7b394795a4e5a297731d"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"37fd3cb0b11f008af0112c22f3e6144cc0949611","unresolved":false,"context_lines":[{"line_number":66,"context_line":"  are regarded by nova as \"belong to all NUMA nodes\". The ``preferred`` value"},{"line_number":67,"context_line":"  will mean that nova will boot VMs with PCI devices if:"},{"line_number":68,"context_line":""},{"line_number":69,"context_line":"  * the PCI device is associated with at least one NUMA nodes on which the"},{"line_number":70,"context_line":"    instance will be booted"},{"line_number":71,"context_line":"  * there is no information about PCI-NUMA association"},{"line_number":72,"context_line":""}],"source_content_type":"text/x-rst","patch_set":15,"id":"1a430d35_b6dc8113","line":69,"in_reply_to":"1a430d35_18ba08e2","updated":"2017-02-06 07:57:53.000000000","message":"Done","commit_id":"3d5305ac624fceae76ec7b394795a4e5a297731d"},{"author":{"_account_id":8768,"name":"Chris Friesen","email":"chris.friesen@windriver.com","username":"cbf123"},"change_message_id":"8159de7bfd6836eff54a4434c258f969784b376e","unresolved":false,"context_lines":[{"line_number":73,"context_line":"**required**"},{"line_number":74,"context_line":""},{"line_number":75,"context_line":"  This value will mean that nova will boot VMs with PCI devices *only* if at"},{"line_number":76,"context_line":"  least one of the NUMA nodes associated with these PCI devices.  It means than"},{"line_number":77,"context_line":"  if NUMA node info for some pci devices wouldn\u0027t be provided these PCI devices"},{"line_number":78,"context_line":"  wouldn\u0027t be used. This is a maximum performance behavior."},{"line_number":79,"context_line":""}],"source_content_type":"text/x-rst","patch_set":15,"id":"1a430d35_f8aa9c6f","line":76,"updated":"2017-02-03 19:56:44.000000000","message":"This isn\u0027t a complete sentence.  Should that be \"one of the NUMA nodes is associated\"?  And are we talking about host or guest NUMA nodes here?","commit_id":"3d5305ac624fceae76ec7b394795a4e5a297731d"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"37fd3cb0b11f008af0112c22f3e6144cc0949611","unresolved":false,"context_lines":[{"line_number":73,"context_line":"**required**"},{"line_number":74,"context_line":""},{"line_number":75,"context_line":"  This value will mean that nova will boot VMs with PCI devices *only* if at"},{"line_number":76,"context_line":"  least one of the NUMA nodes associated with these PCI devices.  It means than"},{"line_number":77,"context_line":"  if NUMA node info for some pci devices wouldn\u0027t be provided these PCI devices"},{"line_number":78,"context_line":"  wouldn\u0027t be used. This is a maximum performance behavior."},{"line_number":79,"context_line":""}],"source_content_type":"text/x-rst","patch_set":15,"id":"1a430d35_b6f3a19d","line":76,"in_reply_to":"1a430d35_f8aa9c6f","updated":"2017-02-06 07:57:53.000000000","message":"We are talking about host NUMA nodes","commit_id":"3d5305ac624fceae76ec7b394795a4e5a297731d"},{"author":{"_account_id":8768,"name":"Chris Friesen","email":"chris.friesen@windriver.com","username":"cbf123"},"change_message_id":"f852778903d62d2916ffe7d3105884c404a6d977","unresolved":false,"context_lines":[{"line_number":64,"context_line":"  we have information about association of PCI devices with NUMA nodes."},{"line_number":65,"context_line":"  However, some PCI devices do not provide such information and such devices"},{"line_number":66,"context_line":"  are regarded by nova as \"belong to all NUMA nodes\". The ``preferred`` value"},{"line_number":67,"context_line":"  will mean that nova will boot VMs with PCI devices if:"},{"line_number":68,"context_line":""},{"line_number":69,"context_line":"  * the PCI device is associated with at least one NUMA nodes on which the"},{"line_number":70,"context_line":"    instance will be booted"}],"source_content_type":"text/x-rst","patch_set":16,"id":"1a430d35_70d4ca1f","line":67,"updated":"2017-02-06 15:14:42.000000000","message":"maybe \"...PCI devices if either:\" given that you said it\u0027s a logical OR?  That way there is no ambiguity.","commit_id":"2b399661fee365c5c6ad2c19b6435c174982bc16"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"552785730681153c7d4f0c64f4ff850dea253c15","unresolved":false,"context_lines":[{"line_number":64,"context_line":"  we have information about association of PCI devices with NUMA nodes."},{"line_number":65,"context_line":"  However, some PCI devices do not provide such information and such devices"},{"line_number":66,"context_line":"  are regarded by nova as \"belong to all NUMA nodes\". The ``preferred`` value"},{"line_number":67,"context_line":"  will mean that nova will boot VMs with PCI devices if:"},{"line_number":68,"context_line":""},{"line_number":69,"context_line":"  * the PCI device is associated with at least one NUMA nodes on which the"},{"line_number":70,"context_line":"    instance will be booted"}],"source_content_type":"text/x-rst","patch_set":16,"id":"1a430d35_8a5ba3f5","line":67,"in_reply_to":"1a430d35_70d4ca1f","updated":"2017-02-07 07:59:45.000000000","message":"sure","commit_id":"2b399661fee365c5c6ad2c19b6435c174982bc16"},{"author":{"_account_id":8768,"name":"Chris Friesen","email":"chris.friesen@windriver.com","username":"cbf123"},"change_message_id":"f852778903d62d2916ffe7d3105884c404a6d977","unresolved":false,"context_lines":[{"line_number":74,"context_line":""},{"line_number":75,"context_line":"  This value will mean that nova will boot VMs with PCI devices *only* if at"},{"line_number":76,"context_line":"  least one of the NUMA node is associated with these PCI devices.  It means"},{"line_number":77,"context_line":"  than if NUMA node info for some pci devices wouldn\u0027t be provided these PCI"},{"line_number":78,"context_line":"  devices wouldn\u0027t be used. This is a maximum performance behavior."},{"line_number":79,"context_line":""},{"line_number":80,"context_line":"**shared**"}],"source_content_type":"text/x-rst","patch_set":16,"id":"1a430d35_70904a44","line":77,"updated":"2017-02-06 15:14:42.000000000","message":"nit: /s/than/that","commit_id":"2b399661fee365c5c6ad2c19b6435c174982bc16"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"552785730681153c7d4f0c64f4ff850dea253c15","unresolved":false,"context_lines":[{"line_number":74,"context_line":""},{"line_number":75,"context_line":"  This value will mean that nova will boot VMs with PCI devices *only* if at"},{"line_number":76,"context_line":"  least one of the NUMA node is associated with these PCI devices.  It means"},{"line_number":77,"context_line":"  than if NUMA node info for some pci devices wouldn\u0027t be provided these PCI"},{"line_number":78,"context_line":"  devices wouldn\u0027t be used. This is a maximum performance behavior."},{"line_number":79,"context_line":""},{"line_number":80,"context_line":"**shared**"}],"source_content_type":"text/x-rst","patch_set":16,"id":"1a430d35_2ad54f1b","line":77,"in_reply_to":"1a430d35_70904a44","updated":"2017-02-07 07:59:45.000000000","message":"Done","commit_id":"2b399661fee365c5c6ad2c19b6435c174982bc16"},{"author":{"_account_id":8768,"name":"Chris Friesen","email":"chris.friesen@windriver.com","username":"cbf123"},"change_message_id":"f852778903d62d2916ffe7d3105884c404a6d977","unresolved":false,"context_lines":[{"line_number":77,"context_line":"  than if NUMA node info for some pci devices wouldn\u0027t be provided these PCI"},{"line_number":78,"context_line":"  devices wouldn\u0027t be used. This is a maximum performance behavior."},{"line_number":79,"context_line":""},{"line_number":80,"context_line":"**shared**"},{"line_number":81,"context_line":""},{"line_number":82,"context_line":"  This value will mean that scheduler will choose a compute host ignoring"},{"line_number":83,"context_line":"  association of NUMA nodes and PCI devices. It means that, if host complies"}],"source_content_type":"text/x-rst","patch_set":16,"id":"1a430d35_bb21b3a1","line":80,"updated":"2017-02-06 15:14:42.000000000","message":"I agree with Sean that \"shared\" is not the best name for this and \"any\" would be better.","commit_id":"2b399661fee365c5c6ad2c19b6435c174982bc16"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"552785730681153c7d4f0c64f4ff850dea253c15","unresolved":false,"context_lines":[{"line_number":77,"context_line":"  than if NUMA node info for some pci devices wouldn\u0027t be provided these PCI"},{"line_number":78,"context_line":"  devices wouldn\u0027t be used. This is a maximum performance behavior."},{"line_number":79,"context_line":""},{"line_number":80,"context_line":"**shared**"},{"line_number":81,"context_line":""},{"line_number":82,"context_line":"  This value will mean that scheduler will choose a compute host ignoring"},{"line_number":83,"context_line":"  association of NUMA nodes and PCI devices. It means that, if host complies"}],"source_content_type":"text/x-rst","patch_set":16,"id":"1a430d35_4ada9b4d","line":80,"in_reply_to":"1a430d35_bb21b3a1","updated":"2017-02-07 07:59:45.000000000","message":"Done","commit_id":"2b399661fee365c5c6ad2c19b6435c174982bc16"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"3db14bf9aae19fc52fcd3405465294cdeab423da","unresolved":false,"context_lines":[{"line_number":32,"context_line":"are placed on second NUMA node. The user should be able to boot VMs on"},{"line_number":33,"context_line":"different NUMA nodes, even if it makes performance worse."},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"In addition, the current behavior doesn\u0027t always provide the best performance"},{"line_number":36,"context_line":"solution. If there is no information about NUMA nodes and PCI devices affinity"},{"line_number":37,"context_line":"for some set of devices, then they will be regarded by nova as \"belonging to"},{"line_number":38,"context_line":"all NUMA nodes\". This can lead to a situation whereby PCI devices are not on"},{"line_number":39,"context_line":"the NUMA node which the CPU\\RAM is on. In this way, the real purpose of"}],"source_content_type":"text/x-rst","patch_set":17,"id":"da36d5c6_c60433c8","line":36,"range":{"start_line":35,"start_character":0,"end_line":36,"end_character":9},"updated":"2017-02-18 14:42:37.000000000","message":"yes this was done intentionally.this behavior should be kept as the default going forward.","commit_id":"36ebfef3227e74d3a0f5ce3622f013f866ff9b76"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"3db14bf9aae19fc52fcd3405465294cdeab423da","unresolved":false,"context_lines":[{"line_number":34,"context_line":""},{"line_number":35,"context_line":"In addition, the current behavior doesn\u0027t always provide the best performance"},{"line_number":36,"context_line":"solution. If there is no information about NUMA nodes and PCI devices affinity"},{"line_number":37,"context_line":"for some set of devices, then they will be regarded by nova as \"belonging to"},{"line_number":38,"context_line":"all NUMA nodes\". This can lead to a situation whereby PCI devices are not on"},{"line_number":39,"context_line":"the NUMA node which the CPU\\RAM is on. In this way, the real purpose of"},{"line_number":40,"context_line":"\"I/O (PCIe) based NUMA Scheduling\" spec is not reached. The user should be able"},{"line_number":41,"context_line":"to ensure this does not happen and guarantee the best performance."}],"source_content_type":"text/x-rst","patch_set":17,"id":"da36d5c6_a652dfdb","line":38,"range":{"start_line":37,"start_character":63,"end_line":38,"end_character":16},"updated":"2017-02-18 14:42:37.000000000","message":"as i said in in a previous comment this is incorrect, they are regarded as not haveing any numa affinity, not that they are attached to all numa nodes.\n\nthe difference is sutble but there are devices that attach to multiple numa node that ligitimatly can be attached to all numa nodes in which case the memory acess time is the same for all numa nodes.\n\nthat is not what we mean if the numa node feild in None.\nit litaly means that the defvice has declared no numa affinty which is a less strict guarantee in that it may be addressable from different numa nodes with the same overhead or it may not.","commit_id":"36ebfef3227e74d3a0f5ce3622f013f866ff9b76"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"d0bfdb01180aeb3673929333735aefa590a9b021","unresolved":false,"context_lines":[{"line_number":34,"context_line":""},{"line_number":35,"context_line":"In addition, the current behavior doesn\u0027t always provide the best performance"},{"line_number":36,"context_line":"solution. If there is no information about NUMA nodes and PCI devices affinity"},{"line_number":37,"context_line":"for some set of devices, then they will be regarded by nova as \"belonging to"},{"line_number":38,"context_line":"all NUMA nodes\". This can lead to a situation whereby PCI devices are not on"},{"line_number":39,"context_line":"the NUMA node which the CPU\\RAM is on. In this way, the real purpose of"},{"line_number":40,"context_line":"\"I/O (PCIe) based NUMA Scheduling\" spec is not reached. The user should be able"},{"line_number":41,"context_line":"to ensure this does not happen and guarantee the best performance."}],"source_content_type":"text/x-rst","patch_set":17,"id":"da36d5c6_9446ce21","line":38,"range":{"start_line":37,"start_character":63,"end_line":38,"end_character":16},"in_reply_to":"da36d5c6_a652dfdb","updated":"2017-02-20 14:17:08.000000000","message":"ok. will fix","commit_id":"36ebfef3227e74d3a0f5ce3622f013f866ff9b76"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"3db14bf9aae19fc52fcd3405465294cdeab423da","unresolved":false,"context_lines":[{"line_number":36,"context_line":"solution. If there is no information about NUMA nodes and PCI devices affinity"},{"line_number":37,"context_line":"for some set of devices, then they will be regarded by nova as \"belonging to"},{"line_number":38,"context_line":"all NUMA nodes\". This can lead to a situation whereby PCI devices are not on"},{"line_number":39,"context_line":"the NUMA node which the CPU\\RAM is on. In this way, the real purpose of"},{"line_number":40,"context_line":"\"I/O (PCIe) based NUMA Scheduling\" spec is not reached. The user should be able"},{"line_number":41,"context_line":"to ensure this does not happen and guarantee the best performance."},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"Of course this ability should be configurable and the current scheduling"}],"source_content_type":"text/x-rst","patch_set":17,"id":"da36d5c6_86031bb8","line":40,"range":{"start_line":39,"start_character":38,"end_line":40,"end_character":55},"updated":"2017-02-18 14:42:37.000000000","message":"no, i was one of the 4 people who designed this feature originally, we defined 4 policies originally and while you are correct that until all 4 are implemented we have not reached the level of granularity we desired, the current behavior is precisely the behavior that we wanted by default which is why we implement that first. we defined the strict policy(\"where if no numa info was available we would fail to boot) for completeness but we did not want that policy to be used as the default behavior.\n\nwe wanted the flexibility to relax or tighten the placement constraints but we have archived the desired level of performance that motivated us to create this feature in the first place.","commit_id":"36ebfef3227e74d3a0f5ce3622f013f866ff9b76"},{"author":{"_account_id":6772,"name":"Stephen Gordon","email":"sgordon@redhat.com","username":"sgordon"},"change_message_id":"0e7e19f52f116afa28e54f2a7fbfc60749a2bc8c","unresolved":false,"context_lines":[{"line_number":58,"context_line":"image metadata ``hw_pci_numa_affinity_policy``. They will have one of three"},{"line_number":59,"context_line":"values."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"**preferred**"},{"line_number":62,"context_line":""},{"line_number":63,"context_line":"  This is the default value and describes the current nova behavior. Usually"},{"line_number":64,"context_line":"  we have information about association of PCI devices with NUMA nodes."}],"source_content_type":"text/x-rst","patch_set":17,"id":"1a430d35_94fac641","line":61,"range":{"start_line":61,"start_character":2,"end_line":61,"end_character":11},"updated":"2017-02-07 20:57:09.000000000","message":"I\u0027ve been struggling with the language here, and I think it\u0027s because we\u0027ve lost the use case that was declared out of scope on https://review.openstack.org/#/c/364468/ (soft affinity/locality for want of a better word).\n\n* Preferred to me implies \"we\u0027ll try get you local to the device, but if we can\u0027t we\u0027ll still schedule your instance\".\n\n* Required to me implies \"we\u0027ll try get you local to the device, but if we can\u0027t we\u0027ll fail scheduling your instance\".\n\n* Any to me is basically as described.\n\nThe scenario that is lost is the one whereby there *is* PCI-NUMA association info available and I would *prefer* that my instance is scheduled local to a device *but* I am still happy for it to schedule if that is not possible (no space on those NUMA nodes).\n\nIt appears the spec as written doesn\u0027t address it and via the use of \"preferred\" to actually mean \"mandatory/required if the info is available\" makes it difficult to add in future. Ideally I\u0027d like to see \"preferred\" encompass this case, but I guess this is problematic if you want to make it the default.","commit_id":"36ebfef3227e74d3a0f5ce3622f013f866ff9b76"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"a1712bb99989a3a927b466a8d1ec6494dcd7bbd6","unresolved":false,"context_lines":[{"line_number":58,"context_line":"image metadata ``hw_pci_numa_affinity_policy``. They will have one of three"},{"line_number":59,"context_line":"values."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"**preferred**"},{"line_number":62,"context_line":""},{"line_number":63,"context_line":"  This is the default value and describes the current nova behavior. Usually"},{"line_number":64,"context_line":"  we have information about association of PCI devices with NUMA nodes."}],"source_content_type":"text/x-rst","patch_set":17,"id":"fa31d9ce_ba5e5d43","line":61,"range":{"start_line":61,"start_character":2,"end_line":61,"end_character":11},"in_reply_to":"1a430d35_1af7818d","updated":"2017-02-10 14:03:08.000000000","message":"Hi guys!\nAbout current three names. I agree that these names are not excellent. And of course we can change them. \n\nAbout idea with two options. In general it\u0027s ok. But:\n1. It\u0027s not obvious for end user that second option (with \"yes/no\") values can be used only for \"required\" policy. In fact we can use \"yes/no\" options for \"preferred\" too.\n2. Maybe it\u0027s silly but I think that one option is better than two.\n\nI think that the better way is to use \"soft/firm/hard\" values. These names wouldn\u0027t give any confusing information about behaviour and used will have to go to docs and read whole description of these policies. Maybe with some table like:\n\n      no PCI-NUMA info | PCI in NUMA node| PCI out of NUMA node\n--------------------------------------------------------------\nhard |       False     |     True        |      False\n--------------------------------------------------------------\nfirm |       True      |     True        |      False\n--------------------------------------------------------------\nsoft |       True      |     True        |      True","commit_id":"36ebfef3227e74d3a0f5ce3622f013f866ff9b76"},{"author":{"_account_id":6772,"name":"Stephen Gordon","email":"sgordon@redhat.com","username":"sgordon"},"change_message_id":"50b6914a05470eedd205f35aa0f76baec1a3375c","unresolved":false,"context_lines":[{"line_number":58,"context_line":"image metadata ``hw_pci_numa_affinity_policy``. They will have one of three"},{"line_number":59,"context_line":"values."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"**preferred**"},{"line_number":62,"context_line":""},{"line_number":63,"context_line":"  This is the default value and describes the current nova behavior. Usually"},{"line_number":64,"context_line":"  we have information about association of PCI devices with NUMA nodes."}],"source_content_type":"text/x-rst","patch_set":17,"id":"1a430d35_dc333e60","line":61,"range":{"start_line":61,"start_character":2,"end_line":61,"end_character":11},"in_reply_to":"1a430d35_424dcd98","updated":"2017-02-09 13:57:18.000000000","message":"Yes, that approach makes sense to me. The only question is for the first one, would preferred be the right default or would it need to be required to match the current default behavior?","commit_id":"36ebfef3227e74d3a0f5ce3622f013f866ff9b76"},{"author":{"_account_id":6772,"name":"Stephen Gordon","email":"sgordon@redhat.com","username":"sgordon"},"change_message_id":"7109a3e51d0cd8beb52bdab8d0d760da8f8a1da9","unresolved":false,"context_lines":[{"line_number":58,"context_line":"image metadata ``hw_pci_numa_affinity_policy``. They will have one of three"},{"line_number":59,"context_line":"values."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"**preferred**"},{"line_number":62,"context_line":""},{"line_number":63,"context_line":"  This is the default value and describes the current nova behavior. Usually"},{"line_number":64,"context_line":"  we have information about association of PCI devices with NUMA nodes."}],"source_content_type":"text/x-rst","patch_set":17,"id":"1a430d35_719b7918","line":61,"range":{"start_line":61,"start_character":2,"end_line":61,"end_character":11},"in_reply_to":"1a430d35_4b535c6b","updated":"2017-02-08 18:45:34.000000000","message":"Indeed, my issue basically boils down to the logic described by \"any\" is actually what would seem expected/implied by the word \"preferred\" while the logic that is described by \"preferred\" is somewhere in between preferred and mandatory (and arguably closer to the latter end of the scale).\n\nI think soft/firm/hard could work, I feel like soft and hard came up in earlier revisions but the suggestion was to use preferred/required based on other policy related settings. For me mentally that would map like this though:\n\ncurrent \"preferred\" -\u003e ?\ncurrent \"required\" -\u003e required\ncurrent \"any\" -\u003e preferred\n\nThe problem I have in naming it is you can\u0027t really call this first scenario \"optional\" or \"preferred\" because the presence or not of the PCI-NUMA association information has such a drastically different impact on the outcome. In effect we\u0027re saying it\u0027s \"preferred\" that PCI-NUMA info be available, but if it is then the placement based on locality is actually mandatory and certainly not just preferred/optional as the naming would suggest. To a user of a cloud where all the nodes expose PCI-NUMA info, nothing about the behavior would seem to line up with the implications of the word \"preferred\".\n\nSeems like we have painted ourselves into a corner a little bit here because of the need to encompass the existing behavior w.r.t. to availability or not of the PCI-NUMA information rather than move it into a separate option (e.g. \"any\" already encompasses it, but we are forced to repeat it here to provide an exact match for what happens today).","commit_id":"36ebfef3227e74d3a0f5ce3622f013f866ff9b76"},{"author":{"_account_id":8768,"name":"Chris Friesen","email":"chris.friesen@windriver.com","username":"cbf123"},"change_message_id":"f5668aaae09ae216eb2a7ec841f04bb86974a999","unresolved":false,"context_lines":[{"line_number":58,"context_line":"image metadata ``hw_pci_numa_affinity_policy``. They will have one of three"},{"line_number":59,"context_line":"values."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"**preferred**"},{"line_number":62,"context_line":""},{"line_number":63,"context_line":"  This is the default value and describes the current nova behavior. Usually"},{"line_number":64,"context_line":"  we have information about association of PCI devices with NUMA nodes."}],"source_content_type":"text/x-rst","patch_set":17,"id":"1a430d35_424dcd98","line":61,"range":{"start_line":61,"start_character":2,"end_line":61,"end_character":11},"in_reply_to":"1a430d35_719b7918","updated":"2017-02-08 20:07:36.000000000","message":"I agree with everything you\u0027ve said here.\n\nMaybe we should break it into two extra spec keys?  One of them would be \"pci numa affinity\" and would allow values of preferred/required using the logical meaning of those words, with preferred being the default.\n\nThe other would represent \"is it acceptable to use PCI devices that don\u0027t report NUMA info\".  (This would only have any impact if specifying \"required\" for the previous option.)\n\nFor this latter option, \"yes\" would represent the current behaviour and would be the default, and \"no\" would represent the stricter behaviour where ultimate performance is desired.\n\n(The key/values of this second option are up for discussion, I\u0027m just describing the intent.)","commit_id":"36ebfef3227e74d3a0f5ce3622f013f866ff9b76"},{"author":{"_account_id":8768,"name":"Chris Friesen","email":"chris.friesen@windriver.com","username":"cbf123"},"change_message_id":"472e48775dceb2677210580d4a7f49109dd1b6f9","unresolved":false,"context_lines":[{"line_number":58,"context_line":"image metadata ``hw_pci_numa_affinity_policy``. They will have one of three"},{"line_number":59,"context_line":"values."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"**preferred**"},{"line_number":62,"context_line":""},{"line_number":63,"context_line":"  This is the default value and describes the current nova behavior. Usually"},{"line_number":64,"context_line":"  we have information about association of PCI devices with NUMA nodes."}],"source_content_type":"text/x-rst","patch_set":17,"id":"1a430d35_4b535c6b","line":61,"range":{"start_line":61,"start_character":2,"end_line":61,"end_character":11},"in_reply_to":"1a430d35_94fac641","updated":"2017-02-07 22:09:42.000000000","message":"I think that \"any\" as described below is effectively what you are describing, given that we will always first try to schedule with affinity in mind.  (As described down at lines 89-92.)\n\nAs I understand it the distinction between preferred and required here is about how to handle PCI devices that have no NUMA affinity reported.\n\nI do understand where you\u0027re coming from, that \"preferred\" really isn\u0027t the best fit here because it\u0027s actually \"more preferred\" than \"any\".  One option would be to switch to soft/firm/hard, or something that better reflects that we\u0027re talking about a continuum of increasing strictness.","commit_id":"36ebfef3227e74d3a0f5ce3622f013f866ff9b76"},{"author":{"_account_id":8768,"name":"Chris Friesen","email":"chris.friesen@windriver.com","username":"cbf123"},"change_message_id":"5bf1a320716c595ead79f8a73f69af11d003ffc4","unresolved":false,"context_lines":[{"line_number":58,"context_line":"image metadata ``hw_pci_numa_affinity_policy``. They will have one of three"},{"line_number":59,"context_line":"values."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"**preferred**"},{"line_number":62,"context_line":""},{"line_number":63,"context_line":"  This is the default value and describes the current nova behavior. Usually"},{"line_number":64,"context_line":"  we have information about association of PCI devices with NUMA nodes."}],"source_content_type":"text/x-rst","patch_set":17,"id":"1a430d35_1af7818d","line":61,"range":{"start_line":61,"start_character":2,"end_line":61,"end_character":11},"in_reply_to":"1a430d35_dc333e60","updated":"2017-02-09 16:02:04.000000000","message":"Oops...I meant to make the current behaviour the default but got it backwards.","commit_id":"36ebfef3227e74d3a0f5ce3622f013f866ff9b76"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"3db14bf9aae19fc52fcd3405465294cdeab423da","unresolved":false,"context_lines":[{"line_number":58,"context_line":"image metadata ``hw_pci_numa_affinity_policy``. They will have one of three"},{"line_number":59,"context_line":"values."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"**preferred**"},{"line_number":62,"context_line":""},{"line_number":63,"context_line":"  This is the default value and describes the current nova behavior. Usually"},{"line_number":64,"context_line":"  we have information about association of PCI devices with NUMA nodes."}],"source_content_type":"text/x-rst","patch_set":17,"id":"da36d5c6_661d470f","line":61,"range":{"start_line":61,"start_character":2,"end_line":61,"end_character":11},"in_reply_to":"da36d5c6_06f3ab0b","updated":"2017-02-18 14:42:37.000000000","message":"one other thing that occurs to me is that when placing vms on a host that provides pcidevices we shoul proablably prefer to fill numa nodes that dont have pci devices availale first if the vm maks no request for a pci device to ensure the highest likely hood that when a new vm is spawned that request a pci device, there is one available in a local numa node.","commit_id":"36ebfef3227e74d3a0f5ce3622f013f866ff9b76"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"b79cc6cd3cd625f95e2591294c6903ec2de59c25","unresolved":false,"context_lines":[{"line_number":58,"context_line":"image metadata ``hw_pci_numa_affinity_policy``. They will have one of three"},{"line_number":59,"context_line":"values."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"**preferred**"},{"line_number":62,"context_line":""},{"line_number":63,"context_line":"  This is the default value and describes the current nova behavior. Usually"},{"line_number":64,"context_line":"  we have information about association of PCI devices with NUMA nodes."}],"source_content_type":"text/x-rst","patch_set":17,"id":"da36d5c6_da094d33","line":61,"range":{"start_line":61,"start_character":2,"end_line":61,"end_character":11},"in_reply_to":"da36d5c6_14e57e36","updated":"2017-02-20 15:26:30.000000000","message":"\u003e Sean: Thank you for \"dipping in history of SR-IOV in Nova\". Now current behavior looks clearer. As I see option \u0027any\u0027 was added just to have a backward compatibility with kilo. I think now it is useless and we should use only 3 others (strict, require, prefer).\n\nI concur - the \u0027any\u0027 policy seems pretty useless, and Kilo is ancient enough that we don\u0027t really need to reintroduce a compatibility shim.","commit_id":"36ebfef3227e74d3a0f5ce3622f013f866ff9b76"},{"author":{"_account_id":6772,"name":"Stephen Gordon","email":"sgordon@redhat.com","username":"sgordon"},"change_message_id":"30a45556111c9e1aa1f90caf3e30c20e976a5291","unresolved":false,"context_lines":[{"line_number":58,"context_line":"image metadata ``hw_pci_numa_affinity_policy``. They will have one of three"},{"line_number":59,"context_line":"values."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"**preferred**"},{"line_number":62,"context_line":""},{"line_number":63,"context_line":"  This is the default value and describes the current nova behavior. Usually"},{"line_number":64,"context_line":"  we have information about association of PCI devices with NUMA nodes."}],"source_content_type":"text/x-rst","patch_set":17,"id":"da36d5c6_fc08e82a","line":61,"range":{"start_line":61,"start_character":2,"end_line":61,"end_character":11},"in_reply_to":"da36d5c6_661d470f","updated":"2017-02-18 19:11:17.000000000","message":"The piece around preferring filling nodes without attached devices first where the guest does not need one is covered by another spec: https://review.openstack.org/#/c/364468/","commit_id":"36ebfef3227e74d3a0f5ce3622f013f866ff9b76"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"d0bfdb01180aeb3673929333735aefa590a9b021","unresolved":false,"context_lines":[{"line_number":58,"context_line":"image metadata ``hw_pci_numa_affinity_policy``. They will have one of three"},{"line_number":59,"context_line":"values."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"**preferred**"},{"line_number":62,"context_line":""},{"line_number":63,"context_line":"  This is the default value and describes the current nova behavior. Usually"},{"line_number":64,"context_line":"  we have information about association of PCI devices with NUMA nodes."}],"source_content_type":"text/x-rst","patch_set":17,"id":"da36d5c6_14e57e36","line":61,"range":{"start_line":61,"start_character":2,"end_line":61,"end_character":11},"in_reply_to":"da36d5c6_fc08e82a","updated":"2017-02-20 14:17:08.000000000","message":"Sean: Thank you for \"dipping in history of SR-IOV in Nova\". Now current behavior looks clearer. As I see option \u0027any\u0027 was added just to have a backward compatibility with kilo. I think now it is useless and we should use only 3 others (strict, require, prefer). \n\nStephen: yes, that spec will fix the problem","commit_id":"36ebfef3227e74d3a0f5ce3622f013f866ff9b76"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"fc4ac8b90c1b9e2e37d2233ee8eeaec6e4f44dfe","unresolved":false,"context_lines":[{"line_number":58,"context_line":"image metadata ``hw_pci_numa_affinity_policy``. They will have one of three"},{"line_number":59,"context_line":"values."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"**preferred**"},{"line_number":62,"context_line":""},{"line_number":63,"context_line":"  This is the default value and describes the current nova behavior. Usually"},{"line_number":64,"context_line":"  we have information about association of PCI devices with NUMA nodes."}],"source_content_type":"text/x-rst","patch_set":17,"id":"da36d5c6_06f3ab0b","line":61,"range":{"start_line":61,"start_character":2,"end_line":61,"end_character":11},"in_reply_to":"fa31d9ce_5fc8a822","updated":"2017-02-18 14:17:02.000000000","message":"when we desinded this feature back in kilo we defined 4 policies that we wanted to support and only implmented 1. the 4 were strict,require,prefer and any.\n\nstrict ment that we would only boot the vm if there was numa info for the device and that it was associated with one of the numa nodes used by the guest. \n\nrequired was the same as strict except if no numa info was associated with the device it was allowed to boot.\n\nprefer was best effort, it would prefer to maintain numa affinity if there was a device free on one of the numa nodes of the gust but would allow cross numa devices if there were no local devices availale.\n\nany skipped all numa afinity checking and gave you the pre kilo behavior where the first device retruned form the db was chosen e.g. random selection.\n\nas i said we only implmented 1 policy which was my definition of require above since we did not have time to implment the others before kilo closed.\n\nwe should maintain the current behavior when we intoduce new policies which would be my definition of required above.\n\ne.g nova will only boot the vm if it can provide a pci device on one of the numa node of the guest or the pci device has no numa affinity.","commit_id":"36ebfef3227e74d3a0f5ce3622f013f866ff9b76"},{"author":{"_account_id":6772,"name":"Stephen Gordon","email":"sgordon@redhat.com","username":"sgordon"},"change_message_id":"a647a52002e7adebf0a2d25ded2b0db9e4f5c766","unresolved":false,"context_lines":[{"line_number":58,"context_line":"image metadata ``hw_pci_numa_affinity_policy``. They will have one of three"},{"line_number":59,"context_line":"values."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"**preferred**"},{"line_number":62,"context_line":""},{"line_number":63,"context_line":"  This is the default value and describes the current nova behavior. Usually"},{"line_number":64,"context_line":"  we have information about association of PCI devices with NUMA nodes."}],"source_content_type":"text/x-rst","patch_set":17,"id":"fa31d9ce_c95ea675","line":61,"range":{"start_line":61,"start_character":2,"end_line":61,"end_character":11},"in_reply_to":"fa31d9ce_ba5e5d43","updated":"2017-02-13 13:41:59.000000000","message":"Going back I think it was Ludovic who commented on patchset 3 that soft/hard might be too confusing, I think both what you propose and what Chris propose\u0027s encompass my concerns about covering all of the scenarios though. I assume here you would be making hard the default to match current behavior (again I think we really backed ourselves into a corner here)?","commit_id":"36ebfef3227e74d3a0f5ce3622f013f866ff9b76"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"884f6f54e014bd59544348304931a55d30fc9412","unresolved":false,"context_lines":[{"line_number":58,"context_line":"image metadata ``hw_pci_numa_affinity_policy``. They will have one of three"},{"line_number":59,"context_line":"values."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"**preferred**"},{"line_number":62,"context_line":""},{"line_number":63,"context_line":"  This is the default value and describes the current nova behavior. Usually"},{"line_number":64,"context_line":"  we have information about association of PCI devices with NUMA nodes."}],"source_content_type":"text/x-rst","patch_set":17,"id":"fa31d9ce_5fc8a822","line":61,"range":{"start_line":61,"start_character":2,"end_line":61,"end_character":11},"in_reply_to":"fa31d9ce_c95ea675","updated":"2017-02-13 14:36:59.000000000","message":"\u003e Going back I think it was Ludovic who commented on patchset 3 that soft/hard might be too confusing.\n\nI agree that soft/hard are confusing. But all other proposals are confusing too. So I suggest to choose hard/firm/soft because these values give *no* information about 3 different behavior. Let me explain. If we have \u0027preffered/shared/any/required\u0027 values, different users may understand these values differently. For example one of them may think that \u0027preffered\u0027 value describes \u0027soft\u0027 behavior from the table above. I mean such values may cause wrong expectations. In these case maybe better to choose uninformative names (hard/firm/soft) to force the user to go read docs and to see good description of behaviors. Maybe no information is better than wrong expectations?\n\n\u003e I assume here you would be making hard the default to match current behavior?\n\nNo, my suggestion is to make \u0027firm\u0027 the default behavior because current behavior allows to boot VM if PCI in NUMA node and if there is no information about affinity","commit_id":"36ebfef3227e74d3a0f5ce3622f013f866ff9b76"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"6354ad9a636ca5c4d277d505d89a5f60915c6f10","unresolved":false,"context_lines":[{"line_number":49,"context_line":""},{"line_number":50,"context_line":"A user wants to boot a guest with PCI devices and numa co-locality of those"},{"line_number":51,"context_line":"devices with the vcpus is not important for him (even if performance will be"},{"line_number":52,"context_line":"less)."},{"line_number":53,"context_line":""},{"line_number":54,"context_line":"Proposed change"},{"line_number":55,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":19,"id":"9ff52f3f_60b69edc","line":52,"updated":"2017-04-13 13:00:37.000000000","message":"You know, I would love it if the three cases below were given the context of the hardware resources and app requirements folks have that push them down into each cases. Like \"sometimes there is a mix of hosts that can\u0027t report PCI devices and can report affinity\", etc, etc.","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"3c4bddc0adad56578a228e9cdf97af37b5182252","unresolved":false,"context_lines":[{"line_number":55,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":56,"context_line":""},{"line_number":57,"context_line":"This spec is needed to decide which PCI device will be used by new VM. To this"},{"line_number":58,"context_line":"end, we will add a new flavor extra spec ``hw:pci_numa_affinity_policy`` and"},{"line_number":59,"context_line":"image metadata ``hw_pci_numa_affinity_policy``. They will have one of three"},{"line_number":60,"context_line":"values."},{"line_number":61,"context_line":""}],"source_content_type":"text/x-rst","patch_set":19,"id":"dfeb2761_46eea0ed","line":58,"updated":"2017-04-04 14:40:29.000000000","message":"The only problem with putting this in flavor extra specs is now you\u0027ll need to have multiple flavors describing exactly the same resources but differing only in their tolerance for a particular scheduling constraint/policy. This will lead to an explosion of flavors, unfortunately.","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"3c4bddc0adad56578a228e9cdf97af37b5182252","unresolved":false,"context_lines":[{"line_number":56,"context_line":""},{"line_number":57,"context_line":"This spec is needed to decide which PCI device will be used by new VM. To this"},{"line_number":58,"context_line":"end, we will add a new flavor extra spec ``hw:pci_numa_affinity_policy`` and"},{"line_number":59,"context_line":"image metadata ``hw_pci_numa_affinity_policy``. They will have one of three"},{"line_number":60,"context_line":"values."},{"line_number":61,"context_line":""},{"line_number":62,"context_line":"**required**"}],"source_content_type":"text/x-rst","patch_set":19,"id":"dfeb2761_26eb9cdc","line":59,"updated":"2017-04-04 14:40:29.000000000","message":"I actually think that the image metadata is the *only* place to put this scheduler policy/hint. Certain images contain workloads that must have strict PCI to NUMA affinity requirements. So, only those images should be decorated with this metadata tag. Flavors shouldn\u0027t be part of the equation here, IMHO.","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"fcf29043d5a70f9c6650fe51ff341134ebbd0486","unresolved":false,"context_lines":[{"line_number":56,"context_line":""},{"line_number":57,"context_line":"This spec is needed to decide which PCI device will be used by new VM. To this"},{"line_number":58,"context_line":"end, we will add a new flavor extra spec ``hw:pci_numa_affinity_policy`` and"},{"line_number":59,"context_line":"image metadata ``hw_pci_numa_affinity_policy``. They will have one of three"},{"line_number":60,"context_line":"values."},{"line_number":61,"context_line":""},{"line_number":62,"context_line":"**required**"}],"source_content_type":"text/x-rst","patch_set":19,"id":"dfeb2761_2c44acb6","line":59,"range":{"start_line":59,"start_character":0,"end_line":59,"end_character":46},"updated":"2017-04-04 08:36:09.000000000","message":"What the use case for needing image metadata.\n\nI guess the admin might want to setup the flavors so they relax the requirements for certain flavors.","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"2c1a85d1ec744fd3f58f03b356c5200c846ab17c","unresolved":false,"context_lines":[{"line_number":56,"context_line":""},{"line_number":57,"context_line":"This spec is needed to decide which PCI device will be used by new VM. To this"},{"line_number":58,"context_line":"end, we will add a new flavor extra spec ``hw:pci_numa_affinity_policy`` and"},{"line_number":59,"context_line":"image metadata ``hw_pci_numa_affinity_policy``. They will have one of three"},{"line_number":60,"context_line":"values."},{"line_number":61,"context_line":""},{"line_number":62,"context_line":"**required**"}],"source_content_type":"text/x-rst","patch_set":19,"id":"9ff52f3f_f4bb940b","line":59,"in_reply_to":"9ff52f3f_8de1e0d3","updated":"2017-04-13 11:26:27.000000000","message":"Sean\u0027s right that the best solution would be for it to be in the boot request, independent of the flavor or image, but we don\u0027t live in that world yet. :(\n\nIt seems that there\u0027s going to be an explosion in either direction, and it is easy to imagine cases where the feature is desirable, independent of the flavor or the image, so, *shrug*.","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"6354ad9a636ca5c4d277d505d89a5f60915c6f10","unresolved":false,"context_lines":[{"line_number":56,"context_line":""},{"line_number":57,"context_line":"This spec is needed to decide which PCI device will be used by new VM. To this"},{"line_number":58,"context_line":"end, we will add a new flavor extra spec ``hw:pci_numa_affinity_policy`` and"},{"line_number":59,"context_line":"image metadata ``hw_pci_numa_affinity_policy``. They will have one of three"},{"line_number":60,"context_line":"values."},{"line_number":61,"context_line":""},{"line_number":62,"context_line":"**required**"}],"source_content_type":"text/x-rst","patch_set":19,"id":"9ff52f3f_005472c9","line":59,"in_reply_to":"9ff52f3f_f4bb940b","updated":"2017-04-13 13:00:37.000000000","message":"@jay my thinking was it should be in the flavor because of the capacity planning implications. But I get some workloads are pickier than others and need/want the better performance.","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"e329bad0ef530a55a5f72e77233b18904b2e3806","unresolved":false,"context_lines":[{"line_number":56,"context_line":""},{"line_number":57,"context_line":"This spec is needed to decide which PCI device will be used by new VM. To this"},{"line_number":58,"context_line":"end, we will add a new flavor extra spec ``hw:pci_numa_affinity_policy`` and"},{"line_number":59,"context_line":"image metadata ``hw_pci_numa_affinity_policy``. They will have one of three"},{"line_number":60,"context_line":"values."},{"line_number":61,"context_line":""},{"line_number":62,"context_line":"**required**"}],"source_content_type":"text/x-rst","patch_set":19,"id":"bff0334d_aec0c77c","line":59,"in_reply_to":"bff0334d_438eb42e","updated":"2017-04-07 14:39:03.000000000","message":"about explotion problem:\nIf it cares us then we already has problems with \u0027cpu_policy\u0027 and \u0027cpu_thread_policy\u0027 because they are in image and flavor metadata too.","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"e44d341a8e818b4126f3003b484b408e9afcdcab","unresolved":false,"context_lines":[{"line_number":56,"context_line":""},{"line_number":57,"context_line":"This spec is needed to decide which PCI device will be used by new VM. To this"},{"line_number":58,"context_line":"end, we will add a new flavor extra spec ``hw:pci_numa_affinity_policy`` and"},{"line_number":59,"context_line":"image metadata ``hw_pci_numa_affinity_policy``. They will have one of three"},{"line_number":60,"context_line":"values."},{"line_number":61,"context_line":""},{"line_number":62,"context_line":"**required**"}],"source_content_type":"text/x-rst","patch_set":19,"id":"9ff52f3f_8de1e0d3","line":59,"in_reply_to":"dfeb2761_26eb9cdc","updated":"2017-04-13 10:39:54.000000000","message":"\u003e I actually think that the image metadata is the *only* place to put this scheduler policy/hint. Certain images contain workloads that must have strict PCI to NUMA affinity requirements. So, only those images should be decorated with this metadata tag. Flavors shouldn\u0027t be part of the equation here, IMHO.\n\nI respectfully disagree :) The traditional approach we\u0027ve taken to this is to use both flavor extra specs and image metadata properties. The former are expected to be more user-centric, while the latter are operator-centric and take priority. This is how things work for cpu_policy, cpu_thread_policy, serial_port_count and others. It\u0027s also how I would have liked things to work for emulator_thread_policy, but that\u0027s an argument for a different day.\n\nIn any case, if we drop the image option we risk a flavor explosion, while dropping the flavor option risks image explosion. Providing both should provide maximum flexibility in avoiding this.","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"76806fafeabb0eda9889d0d3db452393df38ba06","unresolved":false,"context_lines":[{"line_number":56,"context_line":""},{"line_number":57,"context_line":"This spec is needed to decide which PCI device will be used by new VM. To this"},{"line_number":58,"context_line":"end, we will add a new flavor extra spec ``hw:pci_numa_affinity_policy`` and"},{"line_number":59,"context_line":"image metadata ``hw_pci_numa_affinity_policy``. They will have one of three"},{"line_number":60,"context_line":"values."},{"line_number":61,"context_line":""},{"line_number":62,"context_line":"**required**"}],"source_content_type":"text/x-rst","patch_set":19,"id":"dfeb2761_8e26fa14","line":59,"in_reply_to":"dfeb2761_26eb9cdc","updated":"2017-04-04 16:36:48.000000000","message":"I\u0027d prefer to see this just be image metadata as well, because of the flavor explosion problem, but perhaps that\u0027s inevitable?","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"ee3e636397bb6bd71a9ee0f7c995ae32189277cd","unresolved":false,"context_lines":[{"line_number":56,"context_line":""},{"line_number":57,"context_line":"This spec is needed to decide which PCI device will be used by new VM. To this"},{"line_number":58,"context_line":"end, we will add a new flavor extra spec ``hw:pci_numa_affinity_policy`` and"},{"line_number":59,"context_line":"image metadata ``hw_pci_numa_affinity_policy``. They will have one of three"},{"line_number":60,"context_line":"values."},{"line_number":61,"context_line":""},{"line_number":62,"context_line":"**required**"}],"source_content_type":"text/x-rst","patch_set":19,"id":"dfeb2761_cfb2d2e6","line":59,"range":{"start_line":59,"start_character":0,"end_line":59,"end_character":46},"in_reply_to":"dfeb2761_2c44acb6","updated":"2017-04-04 10:02:31.000000000","message":"I suppose that is for completeness. Its very tempting not to add it though.","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"ddcdd64398c636acda733626d8863251649e2f8c","unresolved":false,"context_lines":[{"line_number":56,"context_line":""},{"line_number":57,"context_line":"This spec is needed to decide which PCI device will be used by new VM. To this"},{"line_number":58,"context_line":"end, we will add a new flavor extra spec ``hw:pci_numa_affinity_policy`` and"},{"line_number":59,"context_line":"image metadata ``hw_pci_numa_affinity_policy``. They will have one of three"},{"line_number":60,"context_line":"values."},{"line_number":61,"context_line":""},{"line_number":62,"context_line":"**required**"}],"source_content_type":"text/x-rst","patch_set":19,"id":"dfeb2761_e2312da4","line":59,"range":{"start_line":59,"start_character":0,"end_line":59,"end_character":46},"in_reply_to":"dfeb2761_2c44acb6","updated":"2017-04-04 09:26:49.000000000","message":"In general it was added just for completeness. But for example it can be used in following case: we have some image with some real-time (or near real-time) stuff inside (scripts or something else). And we want to guarantee the best performance of this VM. It can be done by image property.","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"e9a3d04b7f9f1abb12d0fe827fee53cb7f1f0fe9","unresolved":false,"context_lines":[{"line_number":56,"context_line":""},{"line_number":57,"context_line":"This spec is needed to decide which PCI device will be used by new VM. To this"},{"line_number":58,"context_line":"end, we will add a new flavor extra spec ``hw:pci_numa_affinity_policy`` and"},{"line_number":59,"context_line":"image metadata ``hw_pci_numa_affinity_policy``. They will have one of three"},{"line_number":60,"context_line":"values."},{"line_number":61,"context_line":""},{"line_number":62,"context_line":"**required**"}],"source_content_type":"text/x-rst","patch_set":19,"id":"bff0334d_438eb42e","line":59,"in_reply_to":"dfeb2761_8e26fa14","updated":"2017-04-07 13:23:34.000000000","message":"if you do it in image metadata you now have an image explotion problem which is worse as you actully need to store multiple copies of the whole image.\n\ni think supporting both is the best option personally\nbut what i would really like to see in the future is to be able to pass these types of requrest at the nova boot commandline\ne.g. nova boot --require{\"hw_pci_numa_affinity_policy\":\"strict\"}\n\ni think that is planned as part of the placement api change corrct?","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"76806fafeabb0eda9889d0d3db452393df38ba06","unresolved":false,"context_lines":[{"line_number":59,"context_line":"image metadata ``hw_pci_numa_affinity_policy``. They will have one of three"},{"line_number":60,"context_line":"values."},{"line_number":61,"context_line":""},{"line_number":62,"context_line":"**required**"},{"line_number":63,"context_line":""},{"line_number":64,"context_line":"  This is the default value and it describes the current nova behavior. Usually"},{"line_number":65,"context_line":"  we have information about association of PCI devices with NUMA nodes."}],"source_content_type":"text/x-rst","patch_set":19,"id":"dfeb2761_ce11a206","line":62,"updated":"2017-04-04 16:36:48.000000000","message":"Unless I\u0027m misunderstanding something the word \"required\" here is incorrect. \"required\" and \"strict\", in English, mean sort of the same thing. The description here for \"required\" reads as \"do it if we can find a good host OR if we can find a host that isn\u0027t setting PCI-NUMA stuff at all\"\n\nThat\u0027s not what \"required\" means.\n\nWhat\u0027s missing from the description that\u0027s causing me to be misled?\n\nWould \"default\" or \"standard\" or \"broken like the old way\" (j/k) work?","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"2c1a85d1ec744fd3f58f03b356c5200c846ab17c","unresolved":false,"context_lines":[{"line_number":59,"context_line":"image metadata ``hw_pci_numa_affinity_policy``. They will have one of three"},{"line_number":60,"context_line":"values."},{"line_number":61,"context_line":""},{"line_number":62,"context_line":"**required**"},{"line_number":63,"context_line":""},{"line_number":64,"context_line":"  This is the default value and it describes the current nova behavior. Usually"},{"line_number":65,"context_line":"  we have information about association of PCI devices with NUMA nodes."}],"source_content_type":"text/x-rst","patch_set":19,"id":"9ff52f3f_f4fef4ac","line":62,"in_reply_to":"9ff52f3f_ad05c47c","updated":"2017-04-13 11:26:27.000000000","message":"Thanks Sean, I know the difference between required and strict. My question is more around your description in your third paragraph: is that assumption an assumption you can always make and thus \"required\" gets your the desired behavior?\n\nMy initial read of the text below was that the assumption was correctly \"usually\" but not \"always\". If it is \"usually\" then \"required\" is not good.\n\nBut I\u0027m not suitably clear on all the issues here for me to say yay or nay based on just this section.","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"6354ad9a636ca5c4d277d505d89a5f60915c6f10","unresolved":false,"context_lines":[{"line_number":59,"context_line":"image metadata ``hw_pci_numa_affinity_policy``. They will have one of three"},{"line_number":60,"context_line":"values."},{"line_number":61,"context_line":""},{"line_number":62,"context_line":"**required**"},{"line_number":63,"context_line":""},{"line_number":64,"context_line":"  This is the default value and it describes the current nova behavior. Usually"},{"line_number":65,"context_line":"  we have information about association of PCI devices with NUMA nodes."}],"source_content_type":"text/x-rst","patch_set":19,"id":"9ff52f3f_40f83ac3","line":62,"in_reply_to":"9ff52f3f_f4fef4ac","updated":"2017-04-13 13:00:37.000000000","message":"You know totally miss-read this bit. Appreciate Chris calling this out.\n\nThis does sound more like \"prefer\" sort of behaviour, all be it different to the \"prefer\" you have below, that one is more like \"best effort\".","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"e44d341a8e818b4126f3003b484b408e9afcdcab","unresolved":false,"context_lines":[{"line_number":59,"context_line":"image metadata ``hw_pci_numa_affinity_policy``. They will have one of three"},{"line_number":60,"context_line":"values."},{"line_number":61,"context_line":""},{"line_number":62,"context_line":"**required**"},{"line_number":63,"context_line":""},{"line_number":64,"context_line":"  This is the default value and it describes the current nova behavior. Usually"},{"line_number":65,"context_line":"  we have information about association of PCI devices with NUMA nodes."}],"source_content_type":"text/x-rst","patch_set":19,"id":"9ff52f3f_ad05c47c","line":62,"in_reply_to":"bff0334d_430914db","updated":"2017-04-13 10:39:54.000000000","message":"Yeah, this is a very tricky one to name. We need a single word to describe \"required in all cases except this very specific one (no NUMA topology)\" and another to describe \"required in all cases full stop\". I think this is as good as we\u0027re going to get here","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"e9a3d04b7f9f1abb12d0fe827fee53cb7f1f0fe9","unresolved":false,"context_lines":[{"line_number":59,"context_line":"image metadata ``hw_pci_numa_affinity_policy``. They will have one of three"},{"line_number":60,"context_line":"values."},{"line_number":61,"context_line":""},{"line_number":62,"context_line":"**required**"},{"line_number":63,"context_line":""},{"line_number":64,"context_line":"  This is the default value and it describes the current nova behavior. Usually"},{"line_number":65,"context_line":"  we have information about association of PCI devices with NUMA nodes."}],"source_content_type":"text/x-rst","patch_set":19,"id":"bff0334d_430914db","line":62,"in_reply_to":"dfeb2761_ce11a206","updated":"2017-04-07 13:23:34.000000000","message":"strict and required are not really interchangeable in english generally. you can say a company has a strict dress code or a required dress code but haveing a required dress code does not imply that dress code is strict in defining what you ware(required dress of smart casual vs must ware a shirt a shirt and tie).\n\ndefault would work as this is the default option but that tells you nothing about what it does.\n\nif you make the assumtion that a device with no numa infomation is accessable with equal latency form all numa nodes then required makes sense as it is effectivly attached to all numa nodes. i prefer to think of device with no numa infomation as haveing no numa affinity which is suttlely different then equal affinity to all numa nodes but the effect in this case is the same.","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"3c4bddc0adad56578a228e9cdf97af37b5182252","unresolved":false,"context_lines":[{"line_number":75,"context_line":"  This value will mean that nova will boot VMs with PCI devices *only* if at"},{"line_number":76,"context_line":"  least one of the NUMA node is associated with these PCI devices. It means"},{"line_number":77,"context_line":"  that if NUMA node info for some pci devices wouldn\u0027t be provided these PCI"},{"line_number":78,"context_line":"  devices wouldn\u0027t be used. This is a maximum performance behavior."},{"line_number":79,"context_line":""},{"line_number":80,"context_line":"**preferred**"},{"line_number":81,"context_line":""}],"source_content_type":"text/x-rst","patch_set":19,"id":"dfeb2761_06be58d4","line":78,"updated":"2017-04-04 14:40:29.000000000","message":"It\u0027s not that the \"PCI devices wouldn\u0027t be used\". It\u0027s that the scheduler will fail to find a valid host if no compute hosts are found with PCI devices having NUMA affinity information.","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"e329bad0ef530a55a5f72e77233b18904b2e3806","unresolved":false,"context_lines":[{"line_number":75,"context_line":"  This value will mean that nova will boot VMs with PCI devices *only* if at"},{"line_number":76,"context_line":"  least one of the NUMA node is associated with these PCI devices. It means"},{"line_number":77,"context_line":"  that if NUMA node info for some pci devices wouldn\u0027t be provided these PCI"},{"line_number":78,"context_line":"  devices wouldn\u0027t be used. This is a maximum performance behavior."},{"line_number":79,"context_line":""},{"line_number":80,"context_line":"**preferred**"},{"line_number":81,"context_line":""}],"source_content_type":"text/x-rst","patch_set":19,"id":"bff0334d_8e9aa36b","line":78,"in_reply_to":"bff0334d_03266cb8","updated":"2017-04-07 14:39:03.000000000","message":"yes, you are right, but you are looking from scheduler point of view. I talking about nova-compute point of view. When scheduler chose some host, this host may has a PCI devices without NUMA info. These devices \"wouldn\u0027t be used\" because this host has PCI devices with direct numa affinity.","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"e44d341a8e818b4126f3003b484b408e9afcdcab","unresolved":false,"context_lines":[{"line_number":75,"context_line":"  This value will mean that nova will boot VMs with PCI devices *only* if at"},{"line_number":76,"context_line":"  least one of the NUMA node is associated with these PCI devices. It means"},{"line_number":77,"context_line":"  that if NUMA node info for some pci devices wouldn\u0027t be provided these PCI"},{"line_number":78,"context_line":"  devices wouldn\u0027t be used. This is a maximum performance behavior."},{"line_number":79,"context_line":""},{"line_number":80,"context_line":"**preferred**"},{"line_number":81,"context_line":""}],"source_content_type":"text/x-rst","patch_set":19,"id":"9ff52f3f_edf2ac9a","line":78,"in_reply_to":"bff0334d_8e9aa36b","updated":"2017-04-13 10:39:54.000000000","message":"I think this is good enough for now and can be cleaned up later if it confuses people","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"e9a3d04b7f9f1abb12d0fe827fee53cb7f1f0fe9","unresolved":false,"context_lines":[{"line_number":75,"context_line":"  This value will mean that nova will boot VMs with PCI devices *only* if at"},{"line_number":76,"context_line":"  least one of the NUMA node is associated with these PCI devices. It means"},{"line_number":77,"context_line":"  that if NUMA node info for some pci devices wouldn\u0027t be provided these PCI"},{"line_number":78,"context_line":"  devices wouldn\u0027t be used. This is a maximum performance behavior."},{"line_number":79,"context_line":""},{"line_number":80,"context_line":"**preferred**"},{"line_number":81,"context_line":""}],"source_content_type":"text/x-rst","patch_set":19,"id":"bff0334d_03266cb8","line":78,"in_reply_to":"dfeb2761_06be58d4","updated":"2017-04-07 13:23:34.000000000","message":"yes this is strictly true because the filter will eliminate the host as a candiate and either filter the next host or if all host were eliminated result in the oh so useful no valid host found message.","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"3c4bddc0adad56578a228e9cdf97af37b5182252","unresolved":false,"context_lines":[{"line_number":80,"context_line":"**preferred**"},{"line_number":81,"context_line":""},{"line_number":82,"context_line":"  This value will mean that scheduler will choose a compute host ignoring"},{"line_number":83,"context_line":"  association of NUMA nodes and PCI devices. It means that, if host complies"},{"line_number":84,"context_line":"  with the NUMA topology requirements and it has requested PCI device (maybe in"},{"line_number":85,"context_line":"  other NUMA node), it will be passed by ``NUMATopologyFilter``. The scheduler"},{"line_number":86,"context_line":"  will choose one of these \u0027good\u0027 hosts and ask nova compute to boot a VM on"}],"source_content_type":"text/x-rst","patch_set":19,"id":"dfeb2761_e6c25459","line":83,"updated":"2017-04-04 14:40:29.000000000","message":"The first sentence doesn\u0027t make sense. In this situation, the scheduler does not *ignore* the association of NUMA nodes and PCI devices. Instead, the scheduler *prefers* to place the workload on hosts having PCI devices with NUMA affinity but won\u0027t filter out hosts that have PCI devices that are not affined to the NUMA node selected by the NUMATopologyFilter.","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"2c1a85d1ec744fd3f58f03b356c5200c846ab17c","unresolved":false,"context_lines":[{"line_number":80,"context_line":"**preferred**"},{"line_number":81,"context_line":""},{"line_number":82,"context_line":"  This value will mean that scheduler will choose a compute host ignoring"},{"line_number":83,"context_line":"  association of NUMA nodes and PCI devices. It means that, if host complies"},{"line_number":84,"context_line":"  with the NUMA topology requirements and it has requested PCI device (maybe in"},{"line_number":85,"context_line":"  other NUMA node), it will be passed by ``NUMATopologyFilter``. The scheduler"},{"line_number":86,"context_line":"  will choose one of these \u0027good\u0027 hosts and ask nova compute to boot a VM on"}],"source_content_type":"text/x-rst","patch_set":19,"id":"9ff52f3f_749d4458","line":83,"in_reply_to":"9ff52f3f_ed474cbc","updated":"2017-04-13 11:26:27.000000000","message":"I agree with Stephen that a bit more description could help here, to avoid future confusion. There\u0027s good information in Sergey\u0027s comment that ought to be included.","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"e44d341a8e818b4126f3003b484b408e9afcdcab","unresolved":false,"context_lines":[{"line_number":80,"context_line":"**preferred**"},{"line_number":81,"context_line":""},{"line_number":82,"context_line":"  This value will mean that scheduler will choose a compute host ignoring"},{"line_number":83,"context_line":"  association of NUMA nodes and PCI devices. It means that, if host complies"},{"line_number":84,"context_line":"  with the NUMA topology requirements and it has requested PCI device (maybe in"},{"line_number":85,"context_line":"  other NUMA node), it will be passed by ``NUMATopologyFilter``. The scheduler"},{"line_number":86,"context_line":"  will choose one of these \u0027good\u0027 hosts and ask nova compute to boot a VM on"}],"source_content_type":"text/x-rst","patch_set":19,"id":"9ff52f3f_ed474cbc","line":83,"in_reply_to":"bff0334d_5ba8d875","updated":"2017-04-13 10:39:54.000000000","message":"I think that\u0027s ^ an adequate description, though it does suggest some rewording/clarification is necessary. My work around a PCI weigher should allow this to work","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"e329bad0ef530a55a5f72e77233b18904b2e3806","unresolved":false,"context_lines":[{"line_number":80,"context_line":"**preferred**"},{"line_number":81,"context_line":""},{"line_number":82,"context_line":"  This value will mean that scheduler will choose a compute host ignoring"},{"line_number":83,"context_line":"  association of NUMA nodes and PCI devices. It means that, if host complies"},{"line_number":84,"context_line":"  with the NUMA topology requirements and it has requested PCI device (maybe in"},{"line_number":85,"context_line":"  other NUMA node), it will be passed by ``NUMATopologyFilter``. The scheduler"},{"line_number":86,"context_line":"  will choose one of these \u0027good\u0027 hosts and ask nova compute to boot a VM on"}],"source_content_type":"text/x-rst","patch_set":19,"id":"bff0334d_5ba8d875","line":83,"in_reply_to":"bff0334d_e3cca86b","updated":"2017-04-07 14:39:03.000000000","message":"No. This is exactly what I meant. Scheduler will *ignore* numa affinity. But nova compute would *not* ignore numa affinity and at some selected host it will prefer some PCI to another. Let me explain:\n\nThe only way to *prefer* one host to another is to use weight tool. With out a weight,  sheduler can only pick a random node from all \"good\" (which have PCI devices) nodes. Moreover this is a problem of all three policies because for example with \"requested\" policy scheduler can pick a host with *only* PCI devices without NUMA info instead of host with PCI devices with direct affinity.\nThat is why prefering of one *host* to another is a goal of weight tool. It was moved to this spec: https://specs.openstack.org/openstack/nova-specs/specs/pike/approved/reserve-numa-with-pci.html\npatch: https://review.openstack.org/#/c/379524/15 \n\nWhy preferring of one *host* to another is not so necessary for option \"preferred\":\n90% of used wouldn\u0027t use all this options at all because we have an optimal default behavior \"preferred\". Some of them can use \"strict\" policy because it\u0027s a high performance behavior. But the only reason to use a \"preferred\" policy is when user can\u0027t boot a VM because of \"no valid host found\". \"preferred\" means \"I totally don\u0027t care about performance, just boot a VM somewhere\". \n\nThere is no reason to boot *all* VMs with \"preferred\" policy. This policy is needed when you tried to use all other policies and now you want to boot VM somewhere.\n\nSummary: \n1. the goal about preferring one host to another is not the goal of this spec. It\u0027s the goal of Weight spec. \n2. The goal of this spec is to give a user an ability to choose between maximum performance behavior and maximum chance of successfully boot:\n\"strict\" policy is guarantee that we will use only PCI with direct affinity.\n\"preferred\" policy is guarantee that if you have a PCI device somewhere then VM will be booted.\n\nPreferring of best PCI device is just a bonus from nova compute. If we want to have such bonus from scheduler - we need to use Weights. And this is a task of PCIWeight from the patch above.","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"e9a3d04b7f9f1abb12d0fe827fee53cb7f1f0fe9","unresolved":false,"context_lines":[{"line_number":80,"context_line":"**preferred**"},{"line_number":81,"context_line":""},{"line_number":82,"context_line":"  This value will mean that scheduler will choose a compute host ignoring"},{"line_number":83,"context_line":"  association of NUMA nodes and PCI devices. It means that, if host complies"},{"line_number":84,"context_line":"  with the NUMA topology requirements and it has requested PCI device (maybe in"},{"line_number":85,"context_line":"  other NUMA node), it will be passed by ``NUMATopologyFilter``. The scheduler"},{"line_number":86,"context_line":"  will choose one of these \u0027good\u0027 hosts and ask nova compute to boot a VM on"}],"source_content_type":"text/x-rst","patch_set":19,"id":"bff0334d_e3cca86b","line":83,"in_reply_to":"dfeb2761_e6c25459","updated":"2017-04-07 13:23:34.000000000","message":"i think that sentenc was taken form the previous any definition which disabled numa affinity checking entirly so it can proably just be deleted.","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"303039c9693265fd9c1bfe3f2106ef09e3fed9dd","unresolved":false,"context_lines":[{"line_number":89,"context_line":"  To optimise the performance of instance, nova compute will first attempt to"},{"line_number":90,"context_line":"  schedule the instance on a NUMA node associated with requested PCI device. It"},{"line_number":91,"context_line":"  is only when this fails that nova compute will fall back to scheduling on a"},{"line_number":92,"context_line":"  NUMA nodes that is not associated with the PCI device."},{"line_number":93,"context_line":""},{"line_number":94,"context_line":"If both image and flavor properties are not set (equals \u0027None\u0027) the"},{"line_number":95,"context_line":"``required`` policy will be used. If one of image *or* flavor property is not"}],"source_content_type":"text/x-rst","patch_set":19,"id":"dfeb2761_6cc75447","line":92,"updated":"2017-04-04 08:34:12.000000000","message":"So you mean there is a weigher so we prefer the ones where you get affinity?","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"ddcdd64398c636acda733626d8863251649e2f8c","unresolved":false,"context_lines":[{"line_number":89,"context_line":"  To optimise the performance of instance, nova compute will first attempt to"},{"line_number":90,"context_line":"  schedule the instance on a NUMA node associated with requested PCI device. It"},{"line_number":91,"context_line":"  is only when this fails that nova compute will fall back to scheduling on a"},{"line_number":92,"context_line":"  NUMA nodes that is not associated with the PCI device."},{"line_number":93,"context_line":""},{"line_number":94,"context_line":"If both image and flavor properties are not set (equals \u0027None\u0027) the"},{"line_number":95,"context_line":"``required`` policy will be used. If one of image *or* flavor property is not"}],"source_content_type":"text/x-rst","patch_set":19,"id":"dfeb2761_e2bfedcd","line":92,"in_reply_to":"dfeb2761_6cc75447","updated":"2017-04-04 09:26:49.000000000","message":"No. It will be done in pci/stats.py https://review.openstack.org/#/c/390520/8/nova/pci/stats.py","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"ee3e636397bb6bd71a9ee0f7c995ae32189277cd","unresolved":false,"context_lines":[{"line_number":89,"context_line":"  To optimise the performance of instance, nova compute will first attempt to"},{"line_number":90,"context_line":"  schedule the instance on a NUMA node associated with requested PCI device. It"},{"line_number":91,"context_line":"  is only when this fails that nova compute will fall back to scheduling on a"},{"line_number":92,"context_line":"  NUMA nodes that is not associated with the PCI device."},{"line_number":93,"context_line":""},{"line_number":94,"context_line":"If both image and flavor properties are not set (equals \u0027None\u0027) the"},{"line_number":95,"context_line":"``required`` policy will be used. If one of image *or* flavor property is not"}],"source_content_type":"text/x-rst","patch_set":19,"id":"dfeb2761_05f1bb6b","line":92,"in_reply_to":"dfeb2761_6cc75447","updated":"2017-04-04 10:02:31.000000000","message":"Right, so this is all in virt/hardware.py. Fair enough.\n\nIts tempting to ask to add a note in here that it doesn\u0027t affect the scheduler. But thats not a big deal.","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"3c4bddc0adad56578a228e9cdf97af37b5182252","unresolved":false,"context_lines":[{"line_number":106,"context_line":"if possible. Unfortunately it wouldn\u0027t fully solve the problem. For example"},{"line_number":107,"context_line":"compute host has two NUMA nodes. First node has PCI devices, second - hasn\u0027t."},{"line_number":108,"context_line":"Also first node is full by VMs with PCI devices. In this case user wouldn\u0027t be"},{"line_number":109,"context_line":"able to boot any VMs with PCI devices even second NUMA node is free."},{"line_number":110,"context_line":""},{"line_number":111,"context_line":"Data model impact"},{"line_number":112,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":19,"id":"dfeb2761_c6b3d009","line":109,"updated":"2017-04-04 14:40:29.000000000","message":"@johnthetubaguy: yeah, and nested resource providers are behind traits and claims in scheduler in priority...","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"303039c9693265fd9c1bfe3f2106ef09e3fed9dd","unresolved":false,"context_lines":[{"line_number":106,"context_line":"if possible. Unfortunately it wouldn\u0027t fully solve the problem. For example"},{"line_number":107,"context_line":"compute host has two NUMA nodes. First node has PCI devices, second - hasn\u0027t."},{"line_number":108,"context_line":"Also first node is full by VMs with PCI devices. In this case user wouldn\u0027t be"},{"line_number":109,"context_line":"able to boot any VMs with PCI devices even second NUMA node is free."},{"line_number":110,"context_line":""},{"line_number":111,"context_line":"Data model impact"},{"line_number":112,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":19,"id":"dfeb2761_ac9c3c5d","line":109,"updated":"2017-04-04 08:34:12.000000000","message":"Would it not be better to use the nested resource providers structure on the node to report what relationship is here.\n\ni.e. on hosts where you only have PCI devices on one of the NUMA nodes, you just report the PCI devices as attach to the compute node, not a NUMA node?\n\nI am just curious why we want to make the user request \"bad\" scheduling the in flavour for something that is more of an operator choice.","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"ee3e636397bb6bd71a9ee0f7c995ae32189277cd","unresolved":false,"context_lines":[{"line_number":106,"context_line":"if possible. Unfortunately it wouldn\u0027t fully solve the problem. For example"},{"line_number":107,"context_line":"compute host has two NUMA nodes. First node has PCI devices, second - hasn\u0027t."},{"line_number":108,"context_line":"Also first node is full by VMs with PCI devices. In this case user wouldn\u0027t be"},{"line_number":109,"context_line":"able to boot any VMs with PCI devices even second NUMA node is free."},{"line_number":110,"context_line":""},{"line_number":111,"context_line":"Data model impact"},{"line_number":112,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":19,"id":"dfeb2761_25f95f4c","line":109,"in_reply_to":"dfeb2761_ac9c3c5d","updated":"2017-04-04 10:02:31.000000000","message":"Actually thats not really a choice here if you mix various types of hosts, and the people requesting servers want to pick the ones with the better performance.","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"ddcdd64398c636acda733626d8863251649e2f8c","unresolved":false,"context_lines":[{"line_number":106,"context_line":"if possible. Unfortunately it wouldn\u0027t fully solve the problem. For example"},{"line_number":107,"context_line":"compute host has two NUMA nodes. First node has PCI devices, second - hasn\u0027t."},{"line_number":108,"context_line":"Also first node is full by VMs with PCI devices. In this case user wouldn\u0027t be"},{"line_number":109,"context_line":"able to boot any VMs with PCI devices even second NUMA node is free."},{"line_number":110,"context_line":""},{"line_number":111,"context_line":"Data model impact"},{"line_number":112,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":19,"id":"dfeb2761_e28dad31","line":109,"in_reply_to":"dfeb2761_ac9c3c5d","updated":"2017-04-04 09:26:49.000000000","message":"User will be able to request \"bad\" scheduling to boot VM *somewhere* if the default policy failed. I have an example in \"problem description\" section","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"3c4bddc0adad56578a228e9cdf97af37b5182252","unresolved":false,"context_lines":[{"line_number":111,"context_line":"Data model impact"},{"line_number":112,"context_line":"-----------------"},{"line_number":113,"context_line":""},{"line_number":114,"context_line":"New field ``pci_numa_affinity_policy`` will be added to the InstanceNUMACell"},{"line_number":115,"context_line":"object. Fortunately this object is stored as JSON in DB. So we need no DB"},{"line_number":116,"context_line":"migrations to add new field to this object."},{"line_number":117,"context_line":""}],"source_content_type":"text/x-rst","patch_set":19,"id":"dfeb2761_a6b8cce4","line":114,"updated":"2017-04-04 14:40:29.000000000","message":"Why is there a need for a new field on the InstanceNUMACell? AFAICT, the only thing needed is a new InstanceMetadata key for pci_numa_affinity_policy and that should be stored in the instance\u0027s extra/metadata for use in migrations.","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"e44d341a8e818b4126f3003b484b408e9afcdcab","unresolved":false,"context_lines":[{"line_number":111,"context_line":"Data model impact"},{"line_number":112,"context_line":"-----------------"},{"line_number":113,"context_line":""},{"line_number":114,"context_line":"New field ``pci_numa_affinity_policy`` will be added to the InstanceNUMACell"},{"line_number":115,"context_line":"object. Fortunately this object is stored as JSON in DB. So we need no DB"},{"line_number":116,"context_line":"migrations to add new field to this object."},{"line_number":117,"context_line":""}],"source_content_type":"text/x-rst","patch_set":19,"id":"9ff52f3f_0d85d0ad","line":114,"in_reply_to":"dfeb2761_a6b8cce4","updated":"2017-04-13 10:39:54.000000000","message":"Would it be possible to extract that information during scheduling (mainly \u0027numa_fit_instance_to_host\u0027 and its calls)? In any case, can we come back and evaluate this during implementation and rework the spec as necessary?","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"e44d341a8e818b4126f3003b484b408e9afcdcab","unresolved":false,"context_lines":[{"line_number":112,"context_line":"-----------------"},{"line_number":113,"context_line":""},{"line_number":114,"context_line":"New field ``pci_numa_affinity_policy`` will be added to the InstanceNUMACell"},{"line_number":115,"context_line":"object. Fortunately this object is stored as JSON in DB. So we need no DB"},{"line_number":116,"context_line":"migrations to add new field to this object."},{"line_number":117,"context_line":""},{"line_number":118,"context_line":"REST API impact"}],"source_content_type":"text/x-rst","patch_set":19,"id":"9ff52f3f_6d0d7c8f","line":115,"range":{"start_line":115,"start_character":8,"end_line":115,"end_character":19},"updated":"2017-04-13 10:39:54.000000000","message":"Fortunately or unfortunately :)","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"303039c9693265fd9c1bfe3f2106ef09e3fed9dd","unresolved":false,"context_lines":[{"line_number":178,"context_line":""},{"line_number":179,"context_line":"* Add new spec to the flavor"},{"line_number":180,"context_line":"* Add new field to the InstanceNUMACell object"},{"line_number":181,"context_line":"* Change the process of NUMA node choosing, considering new policy"},{"line_number":182,"context_line":"* Update user docs"},{"line_number":183,"context_line":""},{"line_number":184,"context_line":"Dependencies"}],"source_content_type":"text/x-rst","patch_set":19,"id":"dfeb2761_ece004cf","line":181,"updated":"2017-04-04 08:34:12.000000000","message":"Seem a touch sparse on the details of the scheduler changes. I am curious about those.","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"ee3e636397bb6bd71a9ee0f7c995ae32189277cd","unresolved":false,"context_lines":[{"line_number":178,"context_line":""},{"line_number":179,"context_line":"* Add new spec to the flavor"},{"line_number":180,"context_line":"* Add new field to the InstanceNUMACell object"},{"line_number":181,"context_line":"* Change the process of NUMA node choosing, considering new policy"},{"line_number":182,"context_line":"* Update user docs"},{"line_number":183,"context_line":""},{"line_number":184,"context_line":"Dependencies"}],"source_content_type":"text/x-rst","patch_set":19,"id":"dfeb2761_25a71f52","line":181,"in_reply_to":"dfeb2761_ece004cf","updated":"2017-04-04 10:02:31.000000000","message":"Ah, so thats because there aren\u0027t any. Fair enough.","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"ddcdd64398c636acda733626d8863251649e2f8c","unresolved":false,"context_lines":[{"line_number":178,"context_line":""},{"line_number":179,"context_line":"* Add new spec to the flavor"},{"line_number":180,"context_line":"* Add new field to the InstanceNUMACell object"},{"line_number":181,"context_line":"* Change the process of NUMA node choosing, considering new policy"},{"line_number":182,"context_line":"* Update user docs"},{"line_number":183,"context_line":""},{"line_number":184,"context_line":"Dependencies"}],"source_content_type":"text/x-rst","patch_set":19,"id":"dfeb2761_82cca16a","line":181,"in_reply_to":"dfeb2761_ece004cf","updated":"2017-04-04 09:26:49.000000000","message":"no changes in NUMATopologyFilter. just in pci/stats.py. NUMATopologyFilter uses it to figure out if some host is \u0027good\u0027 or \u0027bad\u0027 for new VM","commit_id":"b69ec4fa9313e4bfa2ee6050fe519365e1ef4552"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"1b080732d385283c1f57579f09d4292d24cb7c2e","unresolved":false,"context_lines":[{"line_number":89,"context_line":"  any consideration for the NUMA affinity of PCI devices. However,"},{"line_number":90,"context_line":"  `nova-compute` will attempt a best effort selection of PCI devices based on"},{"line_number":91,"context_line":"  NUMA affinity. It is only if this fails that `nova-compute` will fall back to"},{"line_number":92,"context_line":"  scheduling on a NUMA nodes that is not associated with the PCI device."},{"line_number":93,"context_line":""},{"line_number":94,"context_line":"  Note that even though the ``NUMATopologyFilter`` will not the consider  NUMA"},{"line_number":95,"context_line":"  affinity, the weigher proposed in the *Reserve NUMA Nodes with PCI Devices"}],"source_content_type":"text/x-rst","patch_set":21,"id":"9ff52f3f_5a94a28b","line":92,"updated":"2017-04-13 14:12:56.000000000","message":"\"a NUMA node\", if there\u0027s another","commit_id":"b0c532291b5c34e220b76f3857d81a5bde6156a8"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"4bde1cd00d5cc23f88f07ecea50331e2e5bae48d","unresolved":false,"context_lines":[{"line_number":89,"context_line":"  any consideration for the NUMA affinity of PCI devices. However,"},{"line_number":90,"context_line":"  `nova-compute` will attempt a best effort selection of PCI devices based on"},{"line_number":91,"context_line":"  NUMA affinity. It is only if this fails that `nova-compute` will fall back to"},{"line_number":92,"context_line":"  scheduling on a NUMA nodes that is not associated with the PCI device."},{"line_number":93,"context_line":""},{"line_number":94,"context_line":"  Note that even though the ``NUMATopologyFilter`` will not the consider  NUMA"},{"line_number":95,"context_line":"  affinity, the weigher proposed in the *Reserve NUMA Nodes with PCI Devices"}],"source_content_type":"text/x-rst","patch_set":21,"id":"9ff52f3f_3a21defc","line":92,"in_reply_to":"9ff52f3f_5a94a28b","updated":"2017-04-13 14:20:54.000000000","message":"yep. thanks!","commit_id":"b0c532291b5c34e220b76f3857d81a5bde6156a8"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"b4763ca156238851e4d0d7ea42c739d5016d7db6","unresolved":false,"context_lines":[{"line_number":47,"context_line":"Use Cases"},{"line_number":48,"context_line":"---------"},{"line_number":49,"context_line":""},{"line_number":50,"context_line":"A user wants to boot a guest with PCI devices and numa co-locality of those"},{"line_number":51,"context_line":"devices with the vcpus is not important for him (even if performance will be"},{"line_number":52,"context_line":"less)."},{"line_number":53,"context_line":""},{"line_number":54,"context_line":"- As an operator who cares about obtaining maximum performance from my PCI"},{"line_number":55,"context_line":"  devices, I want to ensure my PCI devices are always NUMA affinitized, even"}],"source_content_type":"text/x-rst","patch_set":22,"id":"9ff52f3f_1dbcd403","line":52,"range":{"start_line":50,"start_character":0,"end_line":52,"end_character":6},"updated":"2017-04-13 14:32:34.000000000","message":"nit: This could probably go (I missed it during the reword)","commit_id":"729592e18e9bc96c79e8673d1da035a921f3441f"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"b1fbb43b636ede1e679a9086572f8639d62aa2fa","unresolved":false,"context_lines":[{"line_number":47,"context_line":"Use Cases"},{"line_number":48,"context_line":"---------"},{"line_number":49,"context_line":""},{"line_number":50,"context_line":"A user wants to boot a guest with PCI devices and numa co-locality of those"},{"line_number":51,"context_line":"devices with the vcpus is not important for him (even if performance will be"},{"line_number":52,"context_line":"less)."},{"line_number":53,"context_line":""},{"line_number":54,"context_line":"- As an operator who cares about obtaining maximum performance from my PCI"},{"line_number":55,"context_line":"  devices, I want to ensure my PCI devices are always NUMA affinitized, even"}],"source_content_type":"text/x-rst","patch_set":22,"id":"9ff52f3f_7de64019","line":52,"range":{"start_line":50,"start_character":0,"end_line":52,"end_character":6},"in_reply_to":"9ff52f3f_1dbcd403","updated":"2017-04-13 14:39:17.000000000","message":"yep","commit_id":"729592e18e9bc96c79e8673d1da035a921f3441f"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"b4763ca156238851e4d0d7ea42c739d5016d7db6","unresolved":false,"context_lines":[{"line_number":66,"context_line":""},{"line_number":67,"context_line":"  Alternatively, as an operator with an existing deployment using PCI devices,"},{"line_number":68,"context_line":"  I don\u0027t want nova to pull the rug from under my feet and suddenly refuse to"},{"line_number":69,"context_line":"  schedule to instances with no NUMA information when it used to."},{"line_number":70,"context_line":""},{"line_number":71,"context_line":"Proposed change"},{"line_number":72,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":22,"id":"9ff52f3f_7d4ac027","line":69,"range":{"start_line":69,"start_character":14,"end_line":69,"end_character":23},"updated":"2017-04-13 14:32:34.000000000","message":"hosts","commit_id":"729592e18e9bc96c79e8673d1da035a921f3441f"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"b1fbb43b636ede1e679a9086572f8639d62aa2fa","unresolved":false,"context_lines":[{"line_number":66,"context_line":""},{"line_number":67,"context_line":"  Alternatively, as an operator with an existing deployment using PCI devices,"},{"line_number":68,"context_line":"  I don\u0027t want nova to pull the rug from under my feet and suddenly refuse to"},{"line_number":69,"context_line":"  schedule to instances with no NUMA information when it used to."},{"line_number":70,"context_line":""},{"line_number":71,"context_line":"Proposed change"},{"line_number":72,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":22,"id":"9ff52f3f_5de37c06","line":69,"range":{"start_line":69,"start_character":14,"end_line":69,"end_character":23},"in_reply_to":"9ff52f3f_7d4ac027","updated":"2017-04-13 14:39:17.000000000","message":"Done","commit_id":"729592e18e9bc96c79e8673d1da035a921f3441f"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"2cf51df951d63713ec1f1695cac7aca8e5b1da2c","unresolved":false,"context_lines":[{"line_number":22,"context_line":"Problem description"},{"line_number":23,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"Booting VMs with PCI devices only if at least one of the NUMA nodes is"},{"line_number":26,"context_line":"associated with the PCI devices is good for performance. This ensures there is"},{"line_number":27,"context_line":"no cross NUMA node memory traffic."},{"line_number":28,"context_line":""},{"line_number":29,"context_line":"However, if a user has an environment with two NUMA nodes and only one PCI"}],"source_content_type":"text/x-rst","patch_set":23,"id":"7ffa3b31_3a875812","line":26,"range":{"start_line":25,"start_character":0,"end_line":26,"end_character":55},"updated":"2017-04-18 15:33:52.000000000","message":"This is confusing, especially when combined with some of the other verbiage below. Constantly referring to \"booting VMs\" as a boolean is the confusing part I think.\n\nI believe you mean:\n\n Instances with PCI devices achieve maximum performance when they\n have memory on the same NUMA node as those devices.\n\nSomething like that?","commit_id":"d10266334e62b4d4dfc50502d06981755c0991db"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"1bcdcdb6b28caea8f5e7639bcb895cbaf54ee8cc","unresolved":false,"context_lines":[{"line_number":22,"context_line":"Problem description"},{"line_number":23,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"Booting VMs with PCI devices only if at least one of the NUMA nodes is"},{"line_number":26,"context_line":"associated with the PCI devices is good for performance. This ensures there is"},{"line_number":27,"context_line":"no cross NUMA node memory traffic."},{"line_number":28,"context_line":""},{"line_number":29,"context_line":"However, if a user has an environment with two NUMA nodes and only one PCI"}],"source_content_type":"text/x-rst","patch_set":23,"id":"5ff73747_274c1a9d","line":26,"range":{"start_line":25,"start_character":0,"end_line":26,"end_character":55},"in_reply_to":"7ffa3b31_3a875812","updated":"2017-04-24 15:30:52.000000000","message":"Yes, something like that","commit_id":"d10266334e62b4d4dfc50502d06981755c0991db"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"2cf51df951d63713ec1f1695cac7aca8e5b1da2c","unresolved":false,"context_lines":[{"line_number":26,"context_line":"associated with the PCI devices is good for performance. This ensures there is"},{"line_number":27,"context_line":"no cross NUMA node memory traffic."},{"line_number":28,"context_line":""},{"line_number":29,"context_line":"However, if a user has an environment with two NUMA nodes and only one PCI"},{"line_number":30,"context_line":"device (for example SR-IOV card associated with first NUMA node) he would be"},{"line_number":31,"context_line":"able to boot VMs with *one* NUMA node and SR-IOV ports only on the first NUMA"},{"line_number":32,"context_line":"node. In this case, the user cannot use half of the CPUs and RAM because these"},{"line_number":33,"context_line":"resources are placed on second NUMA node. The user should be able to boot VMs"},{"line_number":34,"context_line":"on different NUMA nodes, even if it makes performance worse."},{"line_number":35,"context_line":""}],"source_content_type":"text/x-rst","patch_set":23,"id":"7ffa3b31_1aa37ca0","line":32,"range":{"start_line":29,"start_character":0,"end_line":32,"end_character":5},"updated":"2017-04-18 15:33:52.000000000","message":"This sentence doesn\u0027t make sense to me at all, and I\u0027m not sure what to suggest.","commit_id":"d10266334e62b4d4dfc50502d06981755c0991db"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"1bcdcdb6b28caea8f5e7639bcb895cbaf54ee8cc","unresolved":false,"context_lines":[{"line_number":26,"context_line":"associated with the PCI devices is good for performance. This ensures there is"},{"line_number":27,"context_line":"no cross NUMA node memory traffic."},{"line_number":28,"context_line":""},{"line_number":29,"context_line":"However, if a user has an environment with two NUMA nodes and only one PCI"},{"line_number":30,"context_line":"device (for example SR-IOV card associated with first NUMA node) he would be"},{"line_number":31,"context_line":"able to boot VMs with *one* NUMA node and SR-IOV ports only on the first NUMA"},{"line_number":32,"context_line":"node. In this case, the user cannot use half of the CPUs and RAM because these"},{"line_number":33,"context_line":"resources are placed on second NUMA node. The user should be able to boot VMs"},{"line_number":34,"context_line":"on different NUMA nodes, even if it makes performance worse."},{"line_number":35,"context_line":""}],"source_content_type":"text/x-rst","patch_set":23,"id":"5ff73747_c1456619","line":32,"range":{"start_line":29,"start_character":0,"end_line":32,"end_character":5},"in_reply_to":"7ffa3b31_1aa37ca0","updated":"2017-04-24 15:30:52.000000000","message":"In this example you have 2 NUMA nodes (node-1 and node-2). Also you have one PCI device which is associated with one of these nodes (for example with node-1).\n\nAs node-2 has no directly associated PCI devices Nova wouldn\u0027t allow us to boot a VM on this node. However technically it is possible. New policies will remove such limitations from nova.","commit_id":"d10266334e62b4d4dfc50502d06981755c0991db"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"2cf51df951d63713ec1f1695cac7aca8e5b1da2c","unresolved":false,"context_lines":[{"line_number":31,"context_line":"able to boot VMs with *one* NUMA node and SR-IOV ports only on the first NUMA"},{"line_number":32,"context_line":"node. In this case, the user cannot use half of the CPUs and RAM because these"},{"line_number":33,"context_line":"resources are placed on second NUMA node. The user should be able to boot VMs"},{"line_number":34,"context_line":"on different NUMA nodes, even if it makes performance worse."},{"line_number":35,"context_line":""},{"line_number":36,"context_line":"In addition, the current behavior doesn\u0027t always provide the best performance"},{"line_number":37,"context_line":"solution because a VM can use a PCI device if there is no information about"}],"source_content_type":"text/x-rst","patch_set":23,"id":"7ffa3b31_bac0082c","line":34,"updated":"2017-04-18 15:33:52.000000000","message":"The user *should*? I\u0027m not sure why (according to the previous sentence) the user cannot use half the ram if split across numa nodes, which might be related to why I don\u0027t understand this sentence either.","commit_id":"d10266334e62b4d4dfc50502d06981755c0991db"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"1bcdcdb6b28caea8f5e7639bcb895cbaf54ee8cc","unresolved":false,"context_lines":[{"line_number":31,"context_line":"able to boot VMs with *one* NUMA node and SR-IOV ports only on the first NUMA"},{"line_number":32,"context_line":"node. In this case, the user cannot use half of the CPUs and RAM because these"},{"line_number":33,"context_line":"resources are placed on second NUMA node. The user should be able to boot VMs"},{"line_number":34,"context_line":"on different NUMA nodes, even if it makes performance worse."},{"line_number":35,"context_line":""},{"line_number":36,"context_line":"In addition, the current behavior doesn\u0027t always provide the best performance"},{"line_number":37,"context_line":"solution because a VM can use a PCI device if there is no information about"}],"source_content_type":"text/x-rst","patch_set":23,"id":"5ff73747_41ff16aa","line":34,"in_reply_to":"7ffa3b31_bac0082c","updated":"2017-04-24 15:30:52.000000000","message":"user cannot use half of ram and cpu because PciPassthroughFilter and nova compute would not allow him to do it.","commit_id":"d10266334e62b4d4dfc50502d06981755c0991db"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"2cf51df951d63713ec1f1695cac7aca8e5b1da2c","unresolved":false,"context_lines":[{"line_number":39,"context_line":"whereby PCI device is not on the NUMA node which the CPU and RAM is on. The"},{"line_number":40,"context_line":"scheduling mechanism should be more flexible. The user should be able to choose"},{"line_number":41,"context_line":"between maximum performance behavior and maximum chance of successfully VM"},{"line_number":42,"context_line":"launch."},{"line_number":43,"context_line":""},{"line_number":44,"context_line":"Of course this ability should be configurable and the current scheduling"},{"line_number":45,"context_line":"behaviour must remain as the default."}],"source_content_type":"text/x-rst","patch_set":23,"id":"7ffa3b31_baeec8b1","line":42,"updated":"2017-04-18 15:33:52.000000000","message":"I don\u0027t really understand this either. If we don\u0027t have numa information for the device, how can we do better? Perhaps you mean the user should be able to avoid such devices, saying \"if you don\u0027t know the numa affinity for pci device X on compute node Y, then I want a different one\" ?","commit_id":"d10266334e62b4d4dfc50502d06981755c0991db"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"1bcdcdb6b28caea8f5e7639bcb895cbaf54ee8cc","unresolved":false,"context_lines":[{"line_number":39,"context_line":"whereby PCI device is not on the NUMA node which the CPU and RAM is on. The"},{"line_number":40,"context_line":"scheduling mechanism should be more flexible. The user should be able to choose"},{"line_number":41,"context_line":"between maximum performance behavior and maximum chance of successfully VM"},{"line_number":42,"context_line":"launch."},{"line_number":43,"context_line":""},{"line_number":44,"context_line":"Of course this ability should be configurable and the current scheduling"},{"line_number":45,"context_line":"behaviour must remain as the default."}],"source_content_type":"text/x-rst","patch_set":23,"id":"5ff73747_21dd2a14","line":42,"in_reply_to":"7ffa3b31_baeec8b1","updated":"2017-04-24 15:30:52.000000000","message":"yes I meant that \u0027required\u0027 policy requires (haha) that pci devices for some VM *must* have information about PCI NUMA assiciation and *must* be associated with NUMA nodes on which VM will be booted","commit_id":"d10266334e62b4d4dfc50502d06981755c0991db"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"2cf51df951d63713ec1f1695cac7aca8e5b1da2c","unresolved":false,"context_lines":[{"line_number":42,"context_line":"launch."},{"line_number":43,"context_line":""},{"line_number":44,"context_line":"Of course this ability should be configurable and the current scheduling"},{"line_number":45,"context_line":"behaviour must remain as the default."},{"line_number":46,"context_line":""},{"line_number":47,"context_line":"Use Cases"},{"line_number":48,"context_line":"---------"}],"source_content_type":"text/x-rst","patch_set":23,"id":"7ffa3b31_ba4368af","line":45,"range":{"start_line":45,"start_character":10,"end_line":45,"end_character":37},"updated":"2017-04-18 15:33:52.000000000","message":"Why? It sounds like the current behavior is broken based on the above, so why should we maintain it? It seems to me like the user either cares about affinity or doesn\u0027t. Meaning, either hard fail if the constraints can\u0027t be honored, or try-but-don\u0027t-fail.","commit_id":"d10266334e62b4d4dfc50502d06981755c0991db"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"1bcdcdb6b28caea8f5e7639bcb895cbaf54ee8cc","unresolved":false,"context_lines":[{"line_number":42,"context_line":"launch."},{"line_number":43,"context_line":""},{"line_number":44,"context_line":"Of course this ability should be configurable and the current scheduling"},{"line_number":45,"context_line":"behaviour must remain as the default."},{"line_number":46,"context_line":""},{"line_number":47,"context_line":"Use Cases"},{"line_number":48,"context_line":"---------"}],"source_content_type":"text/x-rst","patch_set":23,"id":"5ff73747_61fd7239","line":45,"range":{"start_line":45,"start_character":10,"end_line":45,"end_character":37},"in_reply_to":"7ffa3b31_ba4368af","updated":"2017-04-24 15:30:52.000000000","message":"In needed for backwards compatibility. It\u0027s not good to change current policy. We can deprecate it ant remove in future. Here is a Sean mooney\u0027s comment form patch set 17 about the history of SR-IOV in Nova:\n\nsean mooney\nFeb 18 6:17 PM\nwhen we desinded this feature back in kilo we defined 4 policies that we wanted to support and only implmented 1. the 4 were strict,require,prefer and any.\nstrict ment that we would only boot the vm if there was numa info for the device and that it was associated with one of the numa nodes used by the guest. \nrequired was the same as strict except if no numa info was associated with the device it was allowed to boot.\nprefer was best effort, it would prefer to maintain numa affinity if there was a device free on one of the numa nodes of the gust but would allow cross numa devices if there were no local devices availale.\nany skipped all numa afinity checking and gave you the pre kilo behavior where the first device retruned form the db was chosen e.g. random selection.\nas i said we only implmented 1 policy which was my definition of require above since we did not have time to implment the others before kilo closed.\nwe should maintain the current behavior when we intoduce new policies which would be my definition of required above.\ne.g nova will only boot the vm if it can provide a pci device on one of the numa node of the guest or the pci device has no numa affinity.","commit_id":"d10266334e62b4d4dfc50502d06981755c0991db"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"0675fc7be702e5397893e5e36b616c1469da2bad","unresolved":false,"context_lines":[{"line_number":47,"context_line":"Use Cases"},{"line_number":48,"context_line":"---------"},{"line_number":49,"context_line":""},{"line_number":50,"context_line":"- As an operator who cares about obtaining maximum performance from my PCI"},{"line_number":51,"context_line":"  devices, I want to ensure my PCI devices are always NUMA affinitized, even"},{"line_number":52,"context_line":"  if this results in lower resource usage."},{"line_number":53,"context_line":""}],"source_content_type":"text/x-rst","patch_set":23,"id":"9ff52f3f_d0d65350","line":50,"range":{"start_line":50,"start_character":8,"end_line":50,"end_character":16},"updated":"2017-04-13 14:54:28.000000000","message":"As an operator or tenant who ...\n\noperator in the case of flavour\ntenant in the case of image metadata.\nsame below.","commit_id":"d10266334e62b4d4dfc50502d06981755c0991db"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"1bcdcdb6b28caea8f5e7639bcb895cbaf54ee8cc","unresolved":false,"context_lines":[{"line_number":47,"context_line":"Use Cases"},{"line_number":48,"context_line":"---------"},{"line_number":49,"context_line":""},{"line_number":50,"context_line":"- As an operator who cares about obtaining maximum performance from my PCI"},{"line_number":51,"context_line":"  devices, I want to ensure my PCI devices are always NUMA affinitized, even"},{"line_number":52,"context_line":"  if this results in lower resource usage."},{"line_number":53,"context_line":""}],"source_content_type":"text/x-rst","patch_set":23,"id":"5ff73747_ca0d7b38","line":50,"range":{"start_line":50,"start_character":8,"end_line":50,"end_character":16},"in_reply_to":"9ff52f3f_d0d65350","updated":"2017-04-24 15:30:52.000000000","message":"yes, you are right. But this is just use cases. Maybe it will be better to use here \u0027user\u0027. Because I don\u0027t think that tenant may \"want\" something. And the user who uploads images may.","commit_id":"d10266334e62b4d4dfc50502d06981755c0991db"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"2cf51df951d63713ec1f1695cac7aca8e5b1da2c","unresolved":false,"context_lines":[{"line_number":77,"context_line":"  This value will mean that nova will boot VMs with PCI devices *only* if at"},{"line_number":78,"context_line":"  least one of the NUMA node is associated with these PCI devices. It means"},{"line_number":79,"context_line":"  that if NUMA node info for some PCI devices wouldn\u0027t be provided these PCI"},{"line_number":80,"context_line":"  devices wouldn\u0027t be used. This is provides maximum performance."},{"line_number":81,"context_line":""},{"line_number":82,"context_line":"**preferred**"},{"line_number":83,"context_line":""}],"source_content_type":"text/x-rst","patch_set":23,"id":"7ffa3b31_ff1fa26d","line":80,"updated":"2017-04-18 15:33:52.000000000","message":"So if this is set, and a PCI device is requested that has no numa information, we fail to boot, is that right?\n\nAnd if I ask for two devices, one of which provides numa info and the other does not, then I fail?","commit_id":"d10266334e62b4d4dfc50502d06981755c0991db"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"1bcdcdb6b28caea8f5e7639bcb895cbaf54ee8cc","unresolved":false,"context_lines":[{"line_number":77,"context_line":"  This value will mean that nova will boot VMs with PCI devices *only* if at"},{"line_number":78,"context_line":"  least one of the NUMA node is associated with these PCI devices. It means"},{"line_number":79,"context_line":"  that if NUMA node info for some PCI devices wouldn\u0027t be provided these PCI"},{"line_number":80,"context_line":"  devices wouldn\u0027t be used. This is provides maximum performance."},{"line_number":81,"context_line":""},{"line_number":82,"context_line":"**preferred**"},{"line_number":83,"context_line":""}],"source_content_type":"text/x-rst","patch_set":23,"id":"5ff73747_87388ed3","line":80,"in_reply_to":"7ffa3b31_ff1fa26d","updated":"2017-04-24 15:30:52.000000000","message":"Yes, you will fail in both cases","commit_id":"d10266334e62b4d4dfc50502d06981755c0991db"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"2cf51df951d63713ec1f1695cac7aca8e5b1da2c","unresolved":false,"context_lines":[{"line_number":102,"context_line":"  * The PCI device is associated with at least one NUMA nodes on which the"},{"line_number":103,"context_line":"    instance will be booted"},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"  * There is no information about PCI-NUMA affinity available"},{"line_number":106,"context_line":""},{"line_number":107,"context_line":"If both image and flavor properties are not set (equals ``None``) the"},{"line_number":108,"context_line":"``legacy`` policy will be used. If one of image *or* flavor property is not set"}],"source_content_type":"text/x-rst","patch_set":23,"id":"7ffa3b31_1a981cbe","line":105,"updated":"2017-04-18 15:33:52.000000000","message":"I don\u0027t really see the point in this legacy mode. It seems very similar to preferred and probably not something would ever _ask_ for. Is this a feature_mode\u003dbroken flag?","commit_id":"d10266334e62b4d4dfc50502d06981755c0991db"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"1bcdcdb6b28caea8f5e7639bcb895cbaf54ee8cc","unresolved":false,"context_lines":[{"line_number":102,"context_line":"  * The PCI device is associated with at least one NUMA nodes on which the"},{"line_number":103,"context_line":"    instance will be booted"},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"  * There is no information about PCI-NUMA affinity available"},{"line_number":106,"context_line":""},{"line_number":107,"context_line":"If both image and flavor properties are not set (equals ``None``) the"},{"line_number":108,"context_line":"``legacy`` policy will be used. If one of image *or* flavor property is not set"}],"source_content_type":"text/x-rst","patch_set":23,"id":"5ff73747_a7dcaa74","line":105,"in_reply_to":"7ffa3b31_1a981cbe","updated":"2017-04-24 15:30:52.000000000","message":"The reason is described in comment above (line 45)","commit_id":"d10266334e62b4d4dfc50502d06981755c0991db"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"2cf51df951d63713ec1f1695cac7aca8e5b1da2c","unresolved":false,"context_lines":[{"line_number":109,"context_line":"(equals ``None``) but the other property is set then the value of the set"},{"line_number":110,"context_line":"property will be used. In a case of conflicts between flavor and image"},{"line_number":111,"context_line":"properties (both properties are set and they are not equal) an exception will"},{"line_number":112,"context_line":"be raised."},{"line_number":113,"context_line":""},{"line_number":114,"context_line":"Alternatives"},{"line_number":115,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":23,"id":"7ffa3b31_9a378cb1","line":112,"updated":"2017-04-18 15:33:52.000000000","message":"So, in a perfect world, we wouldn\u0027t do this until we have traits. Or more specifically, we would have traits already done. Eventually, this will have to be a boolean, as traits are just tag-like.\n\nI don\u0027t really see the point of \"legacy\" above. Why not just define \"required\" as the new mode, and if not specified, assume your \"preferred\" behavior above? That gets us in line for traits, and simplifies the potential behaviors users and operators will see from this.","commit_id":"d10266334e62b4d4dfc50502d06981755c0991db"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"1bcdcdb6b28caea8f5e7639bcb895cbaf54ee8cc","unresolved":false,"context_lines":[{"line_number":109,"context_line":"(equals ``None``) but the other property is set then the value of the set"},{"line_number":110,"context_line":"property will be used. In a case of conflicts between flavor and image"},{"line_number":111,"context_line":"properties (both properties are set and they are not equal) an exception will"},{"line_number":112,"context_line":"be raised."},{"line_number":113,"context_line":""},{"line_number":114,"context_line":"Alternatives"},{"line_number":115,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":23,"id":"5ff73747_676d8297","line":112,"in_reply_to":"7ffa3b31_9a378cb1","updated":"2017-04-24 15:30:52.000000000","message":"Yes, I think we should make \u0027preferred\u0027 default behavior. But I think first of all we need to deprecate legacy to inform users about it.","commit_id":"d10266334e62b4d4dfc50502d06981755c0991db"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"2cf51df951d63713ec1f1695cac7aca8e5b1da2c","unresolved":false,"context_lines":[{"line_number":114,"context_line":"Alternatives"},{"line_number":115,"context_line":"------------"},{"line_number":116,"context_line":""},{"line_number":117,"context_line":"Now we have no implemented alternatives. But we can change placement behavior"},{"line_number":118,"context_line":"to do *not* boot VMs which need no PCI devices on NUMA nodes with PCI devices"},{"line_number":119,"context_line":"if possible. Unfortunately it wouldn\u0027t fully solve the problem. For example"},{"line_number":120,"context_line":"compute host has two NUMA nodes. First node has PCI devices, second - hasn\u0027t."},{"line_number":121,"context_line":"Also first node is full by VMs with PCI devices. In this case user wouldn\u0027t be"},{"line_number":122,"context_line":"able to boot any VMs with PCI devices even second NUMA node is free."}],"source_content_type":"text/x-rst","patch_set":23,"id":"7ffa3b31_1a749c67","line":119,"range":{"start_line":117,"start_character":0,"end_line":119,"end_character":12},"updated":"2017-04-18 15:33:52.000000000","message":"This is super confusing and I don\u0027t think I quite understand what it means.","commit_id":"d10266334e62b4d4dfc50502d06981755c0991db"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"1bcdcdb6b28caea8f5e7639bcb895cbaf54ee8cc","unresolved":false,"context_lines":[{"line_number":114,"context_line":"Alternatives"},{"line_number":115,"context_line":"------------"},{"line_number":116,"context_line":""},{"line_number":117,"context_line":"Now we have no implemented alternatives. But we can change placement behavior"},{"line_number":118,"context_line":"to do *not* boot VMs which need no PCI devices on NUMA nodes with PCI devices"},{"line_number":119,"context_line":"if possible. Unfortunately it wouldn\u0027t fully solve the problem. For example"},{"line_number":120,"context_line":"compute host has two NUMA nodes. First node has PCI devices, second - hasn\u0027t."},{"line_number":121,"context_line":"Also first node is full by VMs with PCI devices. In this case user wouldn\u0027t be"},{"line_number":122,"context_line":"able to boot any VMs with PCI devices even second NUMA node is free."}],"source_content_type":"text/x-rst","patch_set":23,"id":"5ff73747_272bfaeb","line":119,"range":{"start_line":117,"start_character":0,"end_line":119,"end_character":12},"in_reply_to":"7ffa3b31_1a749c67","updated":"2017-04-24 15:30:52.000000000","message":"it described here https://specs.openstack.org/openstack/nova-specs/specs/pike/approved/reserve-numa-with-pci.html\n\nIf a VM do not need a PCI device we:\n* will choose some host which have no PCI devices (if possible)\nor \n* will choose some NUMA node without PCI devices on already selected host.\n\nIt needed to do not full hosts/NUMA nodes which have PCI devices by VMs which do *not* need these devices.","commit_id":"d10266334e62b4d4dfc50502d06981755c0991db"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"2cf51df951d63713ec1f1695cac7aca8e5b1da2c","unresolved":false,"context_lines":[{"line_number":126,"context_line":""},{"line_number":127,"context_line":"New field ``pci_numa_affinity_policy`` will be added to the InstanceNUMACell"},{"line_number":128,"context_line":"object. Fortunately this object is stored as JSON in DB. So we need no DB"},{"line_number":129,"context_line":"migrations to add new field to this object."},{"line_number":130,"context_line":""},{"line_number":131,"context_line":"REST API impact"},{"line_number":132,"context_line":"---------------"}],"source_content_type":"text/x-rst","patch_set":23,"id":"7ffa3b31_3a2bd830","line":129,"updated":"2017-04-18 15:33:52.000000000","message":"Won\u0027t we have the flavor and image metadata stored with the instance such that we could re-calculate the flag if we need to? TBH, I really don\u0027t like how the numa stuff seems to replicate a lot of data in its own data structures from other places. If there\u0027s a good reason to, then okay, but can you call that out here if so?","commit_id":"d10266334e62b4d4dfc50502d06981755c0991db"},{"author":{"_account_id":9569,"name":"Sergey Nikitin","email":"nikitinserv@gmail.com","username":"snikitin"},"change_message_id":"1bcdcdb6b28caea8f5e7639bcb895cbaf54ee8cc","unresolved":false,"context_lines":[{"line_number":126,"context_line":""},{"line_number":127,"context_line":"New field ``pci_numa_affinity_policy`` will be added to the InstanceNUMACell"},{"line_number":128,"context_line":"object. Fortunately this object is stored as JSON in DB. So we need no DB"},{"line_number":129,"context_line":"migrations to add new field to this object."},{"line_number":130,"context_line":""},{"line_number":131,"context_line":"REST API impact"},{"line_number":132,"context_line":"---------------"}],"source_content_type":"text/x-rst","patch_set":23,"id":"5ff73747_0768fef2","line":129,"in_reply_to":"7ffa3b31_3a2bd830","updated":"2017-04-24 15:30:52.000000000","message":"Honestly I made it because all NUMA stuff is already in InstanceNUMACell or in InstanceNUMATopology. (cpu_policy, cpu_thread_policy, emulator_threads_policy). So I decided to do the same. \n\nAlso to filter out PCI devices Nova use InstanceNUMAcell objects\nhttps://github.com/openstack/nova/blob/master/nova/pci/stats.py#L214-L223\n\nSo it is easy to change it if we will place policy into cells.\n\nIf we want work with flavor of image we should remove all policies from InatsnceNUMACell","commit_id":"d10266334e62b4d4dfc50502d06981755c0991db"}],"specs/queens/approved/share-pci-between-numa-nodes.rst":[{"author":{"_account_id":25625,"name":"Tetsuro Nakamura","email":"tetsuro.nakamura.bc@hco.ntt.co.jp","username":"tetsuro0907"},"change_message_id":"c704e5c1d21a9b8e96d92e81138e0adcb3c010a7","unresolved":false,"context_lines":[{"line_number":77,"context_line":"  This value will mean that nova will boot VMs with PCI devices *only* if at"},{"line_number":78,"context_line":"  least one of the NUMA node is associated with these PCI devices. It means"},{"line_number":79,"context_line":"  that if NUMA node info for some PCI devices wouldn\u0027t be provided these PCI"},{"line_number":80,"context_line":"  devices wouldn\u0027t be used. This is provides maximum performance."},{"line_number":81,"context_line":""},{"line_number":82,"context_line":"**preferred**"},{"line_number":83,"context_line":""}],"source_content_type":"text/x-rst","patch_set":25,"id":"7f287b81_2ca261fe","line":80,"range":{"start_line":80,"start_character":28,"end_line":80,"end_character":44},"updated":"2017-08-28 05:03:27.000000000","message":"This is provides -\u003e This provides ?","commit_id":"ee0cd509f2009d467863d3237c9cde5f7243839c"},{"author":{"_account_id":25625,"name":"Tetsuro Nakamura","email":"tetsuro.nakamura.bc@hco.ntt.co.jp","username":"tetsuro0907"},"change_message_id":"c704e5c1d21a9b8e96d92e81138e0adcb3c010a7","unresolved":false,"context_lines":[{"line_number":87,"context_line":"  NUMA affinity. It is only if this fails that `nova-compute` will fall back to"},{"line_number":88,"context_line":"  scheduling on a NUMA node that is not associated with the PCI device."},{"line_number":89,"context_line":""},{"line_number":90,"context_line":"  Note that even though the ``NUMATopologyFilter`` will not the consider  NUMA"},{"line_number":91,"context_line":"  affinity, the weigher proposed in the *Reserve NUMA Nodes with PCI Devices"},{"line_number":92,"context_line":"  Attached* spec [2]_ can be used to maximize the chance that a chosen host"},{"line_number":93,"context_line":"  will have NUMA-affinitized PCI devices."}],"source_content_type":"text/x-rst","patch_set":25,"id":"7f287b81_acad31ea","line":90,"range":{"start_line":90,"start_character":50,"end_line":90,"end_character":72},"updated":"2017-08-28 05:03:27.000000000","message":"will not the consider -\u003e will not consider ?","commit_id":"ee0cd509f2009d467863d3237c9cde5f7243839c"},{"author":{"_account_id":25625,"name":"Tetsuro Nakamura","email":"tetsuro.nakamura.bc@hco.ntt.co.jp","username":"tetsuro0907"},"change_message_id":"c704e5c1d21a9b8e96d92e81138e0adcb3c010a7","unresolved":false,"context_lines":[{"line_number":152,"context_line":"------------------"},{"line_number":153,"context_line":""},{"line_number":154,"context_line":"If the ``required`` policy is selected, the performance of instances with PCI"},{"line_number":155,"context_line":"devices will be be more consistent in deployments with non-NUMA aware compute"},{"line_number":156,"context_line":"hosts present. This is because nova would no longer use these hosts. However,"},{"line_number":157,"context_line":"this will also result in a smaller number of hosts available on which to"},{"line_number":158,"context_line":"schedule instances. If all hosts correctly provide NUMA information,"}],"source_content_type":"text/x-rst","patch_set":25,"id":"7f287b81_4c86cd68","line":155,"range":{"start_line":155,"start_character":8,"end_line":155,"end_character":18},"updated":"2017-08-28 05:03:27.000000000","message":"will be be -\u003e will be ?","commit_id":"ee0cd509f2009d467863d3237c9cde5f7243839c"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"885307124ea588345ce1d9841c3096aec84a5183","unresolved":false,"context_lines":[{"line_number":38,"context_line":"affinity of NUMA nodes with this PCI device. This can lead to a situation"},{"line_number":39,"context_line":"whereby PCI device is not on the NUMA node which the CPU and RAM is on. The"},{"line_number":40,"context_line":"scheduling mechanism should be more flexible. The user should be able to choose"},{"line_number":41,"context_line":"between maximum performance behavior and maximum chance of successfully VM"},{"line_number":42,"context_line":"launch."},{"line_number":43,"context_line":""},{"line_number":44,"context_line":"Of course this ability should be configurable and the current scheduling"},{"line_number":45,"context_line":"behaviour must remain as the default."}],"source_content_type":"text/x-rst","patch_set":26,"id":"7f515b1d_0e37f3e5","line":42,"range":{"start_line":41,"start_character":59,"end_line":42,"end_character":7},"updated":"2017-09-25 12:33:14.000000000","message":"successfully launching the VM","commit_id":"d0ada1ffe07b5d46783a0be2746a1e06fa7946dd"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"ff3129b5839757780584195f92cac38e36640730","unresolved":false,"context_lines":[{"line_number":38,"context_line":"affinity of NUMA nodes with this PCI device. This can lead to a situation"},{"line_number":39,"context_line":"whereby PCI device is not on the NUMA node which the CPU and RAM is on. The"},{"line_number":40,"context_line":"scheduling mechanism should be more flexible. The user should be able to choose"},{"line_number":41,"context_line":"between maximum performance behavior and maximum chance of successfully VM"},{"line_number":42,"context_line":"launch."},{"line_number":43,"context_line":""},{"line_number":44,"context_line":"Of course this ability should be configurable and the current scheduling"},{"line_number":45,"context_line":"behaviour must remain as the default."}],"source_content_type":"text/x-rst","patch_set":26,"id":"7f515b1d_380743cd","line":42,"range":{"start_line":41,"start_character":59,"end_line":42,"end_character":7},"in_reply_to":"7f515b1d_0e37f3e5","updated":"2017-09-27 14:45:30.000000000","message":"Done","commit_id":"d0ada1ffe07b5d46783a0be2746a1e06fa7946dd"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"885307124ea588345ce1d9841c3096aec84a5183","unresolved":false,"context_lines":[{"line_number":75,"context_line":"**required**"},{"line_number":76,"context_line":""},{"line_number":77,"context_line":"  This value will mean that nova will boot VMs with PCI devices *only* if at"},{"line_number":78,"context_line":"  least one of the NUMA node is associated with these PCI devices. It means"},{"line_number":79,"context_line":"  that if NUMA node info for some PCI devices wouldn\u0027t be provided these PCI"},{"line_number":80,"context_line":"  devices wouldn\u0027t be used. This provides maximum performance."},{"line_number":81,"context_line":""}],"source_content_type":"text/x-rst","patch_set":26,"id":"7f515b1d_ceb31b4d","line":78,"range":{"start_line":78,"start_character":24,"end_line":78,"end_character":28},"updated":"2017-09-25 12:33:14.000000000","message":"nodes","commit_id":"d0ada1ffe07b5d46783a0be2746a1e06fa7946dd"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"ff3129b5839757780584195f92cac38e36640730","unresolved":false,"context_lines":[{"line_number":75,"context_line":"**required**"},{"line_number":76,"context_line":""},{"line_number":77,"context_line":"  This value will mean that nova will boot VMs with PCI devices *only* if at"},{"line_number":78,"context_line":"  least one of the NUMA node is associated with these PCI devices. It means"},{"line_number":79,"context_line":"  that if NUMA node info for some PCI devices wouldn\u0027t be provided these PCI"},{"line_number":80,"context_line":"  devices wouldn\u0027t be used. This provides maximum performance."},{"line_number":81,"context_line":""}],"source_content_type":"text/x-rst","patch_set":26,"id":"7f515b1d_d8114f06","line":78,"range":{"start_line":78,"start_character":24,"end_line":78,"end_character":28},"in_reply_to":"7f515b1d_ceb31b4d","updated":"2017-09-27 14:45:30.000000000","message":"Done","commit_id":"d0ada1ffe07b5d46783a0be2746a1e06fa7946dd"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"885307124ea588345ce1d9841c3096aec84a5183","unresolved":false,"context_lines":[{"line_number":76,"context_line":""},{"line_number":77,"context_line":"  This value will mean that nova will boot VMs with PCI devices *only* if at"},{"line_number":78,"context_line":"  least one of the NUMA node is associated with these PCI devices. It means"},{"line_number":79,"context_line":"  that if NUMA node info for some PCI devices wouldn\u0027t be provided these PCI"},{"line_number":80,"context_line":"  devices wouldn\u0027t be used. This provides maximum performance."},{"line_number":81,"context_line":""},{"line_number":82,"context_line":"**preferred**"}],"source_content_type":"text/x-rst","patch_set":26,"id":"7f515b1d_aee9ff1a","line":79,"range":{"start_line":79,"start_character":46,"end_line":79,"end_character":73},"updated":"2017-09-25 12:33:14.000000000","message":"could not be determined, those","commit_id":"d0ada1ffe07b5d46783a0be2746a1e06fa7946dd"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"ff3129b5839757780584195f92cac38e36640730","unresolved":false,"context_lines":[{"line_number":76,"context_line":""},{"line_number":77,"context_line":"  This value will mean that nova will boot VMs with PCI devices *only* if at"},{"line_number":78,"context_line":"  least one of the NUMA node is associated with these PCI devices. It means"},{"line_number":79,"context_line":"  that if NUMA node info for some PCI devices wouldn\u0027t be provided these PCI"},{"line_number":80,"context_line":"  devices wouldn\u0027t be used. This provides maximum performance."},{"line_number":81,"context_line":""},{"line_number":82,"context_line":"**preferred**"}],"source_content_type":"text/x-rst","patch_set":26,"id":"7f515b1d_58fd5faf","line":79,"range":{"start_line":79,"start_character":46,"end_line":79,"end_character":73},"in_reply_to":"7f515b1d_aee9ff1a","updated":"2017-09-27 14:45:30.000000000","message":"Done","commit_id":"d0ada1ffe07b5d46783a0be2746a1e06fa7946dd"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"885307124ea588345ce1d9841c3096aec84a5183","unresolved":false,"context_lines":[{"line_number":77,"context_line":"  This value will mean that nova will boot VMs with PCI devices *only* if at"},{"line_number":78,"context_line":"  least one of the NUMA node is associated with these PCI devices. It means"},{"line_number":79,"context_line":"  that if NUMA node info for some PCI devices wouldn\u0027t be provided these PCI"},{"line_number":80,"context_line":"  devices wouldn\u0027t be used. This provides maximum performance."},{"line_number":81,"context_line":""},{"line_number":82,"context_line":"**preferred**"},{"line_number":83,"context_line":""}],"source_content_type":"text/x-rst","patch_set":26,"id":"7f515b1d_6ef3070c","line":80,"range":{"start_line":80,"start_character":22,"end_line":80,"end_character":26},"updated":"2017-09-25 12:33:14.000000000","message":"consumable by the instance","commit_id":"d0ada1ffe07b5d46783a0be2746a1e06fa7946dd"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"ff3129b5839757780584195f92cac38e36640730","unresolved":false,"context_lines":[{"line_number":77,"context_line":"  This value will mean that nova will boot VMs with PCI devices *only* if at"},{"line_number":78,"context_line":"  least one of the NUMA node is associated with these PCI devices. It means"},{"line_number":79,"context_line":"  that if NUMA node info for some PCI devices wouldn\u0027t be provided these PCI"},{"line_number":80,"context_line":"  devices wouldn\u0027t be used. This provides maximum performance."},{"line_number":81,"context_line":""},{"line_number":82,"context_line":"**preferred**"},{"line_number":83,"context_line":""}],"source_content_type":"text/x-rst","patch_set":26,"id":"7f515b1d_9820774a","line":80,"range":{"start_line":80,"start_character":22,"end_line":80,"end_character":26},"in_reply_to":"7f515b1d_6ef3070c","updated":"2017-09-27 14:45:30.000000000","message":"Done","commit_id":"d0ada1ffe07b5d46783a0be2746a1e06fa7946dd"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"885307124ea588345ce1d9841c3096aec84a5183","unresolved":false,"context_lines":[{"line_number":81,"context_line":""},{"line_number":82,"context_line":"**preferred**"},{"line_number":83,"context_line":""},{"line_number":84,"context_line":"  This value will mean that `nova-scheduler` will choose a compute host without"},{"line_number":85,"context_line":"  any consideration for the NUMA affinity of PCI devices. However,"},{"line_number":86,"context_line":"  `nova-compute` will attempt a best effort selection of PCI devices based on"},{"line_number":87,"context_line":"  NUMA affinity. It is only if this fails that `nova-compute` will fall back to"},{"line_number":88,"context_line":"  scheduling on a NUMA node that is not associated with the PCI device."}],"source_content_type":"text/x-rst","patch_set":26,"id":"7f515b1d_4ecf2bb0","line":85,"range":{"start_line":84,"start_character":50,"end_line":85,"end_character":21},"updated":"2017-09-25 12:33:14.000000000","message":"not exclude hosts where PCI devices have no NUMA affinity.","commit_id":"d0ada1ffe07b5d46783a0be2746a1e06fa7946dd"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"ff3129b5839757780584195f92cac38e36640730","unresolved":false,"context_lines":[{"line_number":81,"context_line":""},{"line_number":82,"context_line":"**preferred**"},{"line_number":83,"context_line":""},{"line_number":84,"context_line":"  This value will mean that `nova-scheduler` will choose a compute host without"},{"line_number":85,"context_line":"  any consideration for the NUMA affinity of PCI devices. However,"},{"line_number":86,"context_line":"  `nova-compute` will attempt a best effort selection of PCI devices based on"},{"line_number":87,"context_line":"  NUMA affinity. It is only if this fails that `nova-compute` will fall back to"},{"line_number":88,"context_line":"  scheduling on a NUMA node that is not associated with the PCI device."}],"source_content_type":"text/x-rst","patch_set":26,"id":"7f515b1d_d8df0f38","line":85,"range":{"start_line":84,"start_character":50,"end_line":85,"end_character":21},"in_reply_to":"7f515b1d_4ecf2bb0","updated":"2017-09-27 14:45:30.000000000","message":"Not quite - we will pick hosts where PCI devices have NUMA affinity but ignore said affinity if necessary.","commit_id":"d0ada1ffe07b5d46783a0be2746a1e06fa7946dd"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"28cb8b2222f8efc11cde414e7bead032522b37b9","unresolved":false,"context_lines":[{"line_number":92,"context_line":"  Attached* spec [2]_ can be used to maximize the chance that a chosen host"},{"line_number":93,"context_line":"  will have NUMA-affinitized PCI devices."},{"line_number":94,"context_line":""},{"line_number":95,"context_line":"**legacy**"},{"line_number":96,"context_line":""},{"line_number":97,"context_line":"  This is the default value and it describes the current nova behavior. Usually"},{"line_number":98,"context_line":"  we have information about association of PCI devices with NUMA nodes."},{"line_number":99,"context_line":"  However, some PCI devices do not provide such information. The ``legacy``"},{"line_number":100,"context_line":"  value will mean that nova will boot VMs with PCI device if either:"},{"line_number":101,"context_line":""},{"line_number":102,"context_line":"  * The PCI device is associated with at least one NUMA nodes on which the"},{"line_number":103,"context_line":"    instance will be booted"},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"  * There is no information about PCI-NUMA affinity available"},{"line_number":106,"context_line":""},{"line_number":107,"context_line":"If both image and flavor properties are not set (equals ``None``) the"},{"line_number":108,"context_line":"``legacy`` policy will be used. If one of image *or* flavor property is not set"}],"source_content_type":"text/x-rst","patch_set":26,"id":"7f287b81_c99760eb","line":105,"range":{"start_line":95,"start_character":0,"end_line":105,"end_character":61},"updated":"2017-09-06 16:16:53.000000000","message":"I\u0027ve had some ideas on this. Given that this is seen as broken behavior, how about we take a two-stage approach to ridding ourselves of the broken behavior?\n\nStage 1 (Queens): Default policy is \u0027required\u0027. However, \u0027required\u0027 continues to support hosts that do not provide NUMA topology information but with a WARNING-level log that this functionality is deprecated and will be removed in the next release.\n\nStage 2 (Rocky): We remove the log and start erroring out in this scenario. Operators should not see this because they\u0027ve either fixed their hardware or set \u0027hw_pci_numa_affinity_policy\u003dpreferred\u0027\n\nThoughts?","commit_id":"d0ada1ffe07b5d46783a0be2746a1e06fa7946dd"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"19c568871eb3e90d4c0a8c4f28c48098d028cd59","unresolved":false,"context_lines":[{"line_number":92,"context_line":"  Attached* spec [2]_ can be used to maximize the chance that a chosen host"},{"line_number":93,"context_line":"  will have NUMA-affinitized PCI devices."},{"line_number":94,"context_line":""},{"line_number":95,"context_line":"**legacy**"},{"line_number":96,"context_line":""},{"line_number":97,"context_line":"  This is the default value and it describes the current nova behavior. Usually"},{"line_number":98,"context_line":"  we have information about association of PCI devices with NUMA nodes."},{"line_number":99,"context_line":"  However, some PCI devices do not provide such information. The ``legacy``"},{"line_number":100,"context_line":"  value will mean that nova will boot VMs with PCI device if either:"},{"line_number":101,"context_line":""},{"line_number":102,"context_line":"  * The PCI device is associated with at least one NUMA nodes on which the"},{"line_number":103,"context_line":"    instance will be booted"},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"  * There is no information about PCI-NUMA affinity available"},{"line_number":106,"context_line":""},{"line_number":107,"context_line":"If both image and flavor properties are not set (equals ``None``) the"},{"line_number":108,"context_line":"``legacy`` policy will be used. If one of image *or* flavor property is not set"}],"source_content_type":"text/x-rst","patch_set":26,"id":"ff5d4b2f_65e91d7f","line":105,"range":{"start_line":95,"start_character":0,"end_line":105,"end_character":61},"in_reply_to":"7f287b81_44feb538","updated":"2017-09-19 12:56:41.000000000","message":"Based on the discussions at the PTG, it seems there is value to retaining this behavior after all. For example, PCI devices attached to the QPI bus (like FPGAs on newer Xeon chips)","commit_id":"d0ada1ffe07b5d46783a0be2746a1e06fa7946dd"},{"author":{"_account_id":8768,"name":"Chris Friesen","email":"chris.friesen@windriver.com","username":"cbf123"},"change_message_id":"101375ed26c02a64a2f782101335df237f12fb3b","unresolved":false,"context_lines":[{"line_number":92,"context_line":"  Attached* spec [2]_ can be used to maximize the chance that a chosen host"},{"line_number":93,"context_line":"  will have NUMA-affinitized PCI devices."},{"line_number":94,"context_line":""},{"line_number":95,"context_line":"**legacy**"},{"line_number":96,"context_line":""},{"line_number":97,"context_line":"  This is the default value and it describes the current nova behavior. Usually"},{"line_number":98,"context_line":"  we have information about association of PCI devices with NUMA nodes."},{"line_number":99,"context_line":"  However, some PCI devices do not provide such information. The ``legacy``"},{"line_number":100,"context_line":"  value will mean that nova will boot VMs with PCI device if either:"},{"line_number":101,"context_line":""},{"line_number":102,"context_line":"  * The PCI device is associated with at least one NUMA nodes on which the"},{"line_number":103,"context_line":"    instance will be booted"},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"  * There is no information about PCI-NUMA affinity available"},{"line_number":106,"context_line":""},{"line_number":107,"context_line":"If both image and flavor properties are not set (equals ``None``) the"},{"line_number":108,"context_line":"``legacy`` policy will be used. If one of image *or* flavor property is not set"}],"source_content_type":"text/x-rst","patch_set":26,"id":"7f287b81_44feb538","line":105,"range":{"start_line":95,"start_character":0,"end_line":105,"end_character":61},"in_reply_to":"7f287b81_c99760eb","updated":"2017-09-06 16:45:58.000000000","message":"Are there any cases where there really is no sensible NUMA information for a PCI device?  (Like maybe a PCI bus is attached to multiple NUMA zones.)  As long as this is not the case then this seems reasonable to me.","commit_id":"d0ada1ffe07b5d46783a0be2746a1e06fa7946dd"},{"author":{"_account_id":8768,"name":"Chris Friesen","email":"chris.friesen@windriver.com","username":"cbf123"},"change_message_id":"ff3c5a13ba021aa3cb60b8e64b907ce6102920dd","unresolved":false,"context_lines":[{"line_number":115,"context_line":"------------"},{"line_number":116,"context_line":""},{"line_number":117,"context_line":"Now we have no implemented alternatives. But we can change placement behavior"},{"line_number":118,"context_line":"to do *not* boot VMs which need no PCI devices on NUMA nodes with PCI devices"},{"line_number":119,"context_line":"if possible. Unfortunately it wouldn\u0027t fully solve the problem. For example"},{"line_number":120,"context_line":"compute host has two NUMA nodes. First node has PCI devices, second - hasn\u0027t."},{"line_number":121,"context_line":"Also first node is full by VMs with PCI devices. In this case user wouldn\u0027t be"}],"source_content_type":"text/x-rst","patch_set":26,"id":"7f287b81_f3132db7","line":118,"range":{"start_line":118,"start_character":0,"end_line":118,"end_character":16},"updated":"2017-09-06 15:08:39.000000000","message":"nit: should be \"...to *not* boot...\"","commit_id":"d0ada1ffe07b5d46783a0be2746a1e06fa7946dd"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"ff3129b5839757780584195f92cac38e36640730","unresolved":false,"context_lines":[{"line_number":115,"context_line":"------------"},{"line_number":116,"context_line":""},{"line_number":117,"context_line":"Now we have no implemented alternatives. But we can change placement behavior"},{"line_number":118,"context_line":"to do *not* boot VMs which need no PCI devices on NUMA nodes with PCI devices"},{"line_number":119,"context_line":"if possible. Unfortunately it wouldn\u0027t fully solve the problem. For example"},{"line_number":120,"context_line":"compute host has two NUMA nodes. First node has PCI devices, second - hasn\u0027t."},{"line_number":121,"context_line":"Also first node is full by VMs with PCI devices. In this case user wouldn\u0027t be"}],"source_content_type":"text/x-rst","patch_set":26,"id":"7f515b1d_f8a88b8c","line":118,"range":{"start_line":118,"start_character":0,"end_line":118,"end_character":16},"in_reply_to":"7f287b81_f3132db7","updated":"2017-09-27 14:45:30.000000000","message":"Done","commit_id":"d0ada1ffe07b5d46783a0be2746a1e06fa7946dd"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"885307124ea588345ce1d9841c3096aec84a5183","unresolved":false,"context_lines":[{"line_number":124,"context_line":"Data model impact"},{"line_number":125,"context_line":"-----------------"},{"line_number":126,"context_line":""},{"line_number":127,"context_line":"New field ``pci_numa_affinity_policy`` will be added to the InstanceNUMACell"},{"line_number":128,"context_line":"object. Fortunately this object is stored as JSON in DB. So we need no DB"},{"line_number":129,"context_line":"migrations to add new field to this object."},{"line_number":130,"context_line":""}],"source_content_type":"text/x-rst","patch_set":26,"id":"7f515b1d_ce167bf2","line":127,"range":{"start_line":127,"start_character":60,"end_line":127,"end_character":76},"updated":"2017-09-25 12:33:14.000000000","message":"Why would a new field be added to InstanceNUMACell? I think just storing the extra_spec for the flavor (which we do anyway) is fine for rescheduling/move operations.","commit_id":"d0ada1ffe07b5d46783a0be2746a1e06fa7946dd"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"ff3129b5839757780584195f92cac38e36640730","unresolved":false,"context_lines":[{"line_number":124,"context_line":"Data model impact"},{"line_number":125,"context_line":"-----------------"},{"line_number":126,"context_line":""},{"line_number":127,"context_line":"New field ``pci_numa_affinity_policy`` will be added to the InstanceNUMACell"},{"line_number":128,"context_line":"object. Fortunately this object is stored as JSON in DB. So we need no DB"},{"line_number":129,"context_line":"migrations to add new field to this object."},{"line_number":130,"context_line":""}],"source_content_type":"text/x-rst","patch_set":26,"id":"7f515b1d_38918348","line":127,"range":{"start_line":127,"start_character":60,"end_line":127,"end_character":76},"in_reply_to":"7f515b1d_ce167bf2","updated":"2017-09-27 14:45:30.000000000","message":"\u003e Why would a new field be added to InstanceNUMACell? I think just storing the extra_spec for the flavor (which we do anyway) is fine for rescheduling/move operations.\n\nIt\u0027s extra_specs and image metadata. This is what we did for CPU (thread) pinning, so I didn\u0027t see any reason to do something different here","commit_id":"d0ada1ffe07b5d46783a0be2746a1e06fa7946dd"},{"author":{"_account_id":8768,"name":"Chris Friesen","email":"chris.friesen@windriver.com","username":"cbf123"},"change_message_id":"1acb3ba4a3020f79cf0358e8177d1356a766673c","unresolved":false,"context_lines":[{"line_number":124,"context_line":"Data model impact"},{"line_number":125,"context_line":"-----------------"},{"line_number":126,"context_line":""},{"line_number":127,"context_line":"New field ``pci_numa_affinity_policy`` will be added to the InstanceNUMACell"},{"line_number":128,"context_line":"object. Fortunately this object is stored as JSON in DB. So we need no DB"},{"line_number":129,"context_line":"migrations to add new field to this object."},{"line_number":130,"context_line":""}],"source_content_type":"text/x-rst","patch_set":26,"id":"7f515b1d_d81aba7b","line":127,"range":{"start_line":127,"start_character":60,"end_line":127,"end_character":76},"in_reply_to":"7f515b1d_ce167bf2","updated":"2017-09-25 16:51:55.000000000","message":"numa_fit_instance_to_host() doesn\u0027t get passed the instance currently, so we\u0027d need to change the function signature.\n\nActually, I wonder if it might not make sense to make the PCI numa strictness part of the individual PCI device request.  That way you could represent things like \"I need to be strictly affined to this NIC, but I don\u0027t need to be strictly affined to this FPGA.\"  I think this could be done without changing the function signature since we could make it part of the PCI request(s).","commit_id":"d0ada1ffe07b5d46783a0be2746a1e06fa7946dd"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"a04f27406d089707eda2c6a55b4d044667c854a4","unresolved":false,"context_lines":[{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/nova/+spec/share-pci-between-numa-nodes"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"In the Juno release th \"I/O based NUMA scheduling\" spec was implemented [1]_."},{"line_number":14,"context_line":"This modified the scheduling algorithm such that users were only allowed to"},{"line_number":15,"context_line":"boot instances with PCI devices if the instance was being scheduled on at least"},{"line_number":16,"context_line":"one of the NUMA nodes is associated with the PCI devices or if the PCI devices"}],"source_content_type":"text/x-rst","patch_set":27,"id":"5f4e5783_adefef85","line":13,"range":{"start_line":13,"start_character":20,"end_line":13,"end_character":22},"updated":"2017-10-18 15:05:25.000000000","message":"the","commit_id":"056c630a8e563e8a655db47d8bd85471e2ad279e"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"a04f27406d089707eda2c6a55b4d044667c854a4","unresolved":false,"context_lines":[{"line_number":13,"context_line":"In the Juno release th \"I/O based NUMA scheduling\" spec was implemented [1]_."},{"line_number":14,"context_line":"This modified the scheduling algorithm such that users were only allowed to"},{"line_number":15,"context_line":"boot instances with PCI devices if the instance was being scheduled on at least"},{"line_number":16,"context_line":"one of the NUMA nodes is associated with the PCI devices or if the PCI devices"},{"line_number":17,"context_line":"have no information about NUMA nodes and PCI devices affinity. Before this,"},{"line_number":18,"context_line":"nova booted instances with PCI devices without checking NUMA affinity. However,"},{"line_number":19,"context_line":"such hardcoded behaviour causes problems if not every NUMA node has its own PCI"}],"source_content_type":"text/x-rst","patch_set":27,"id":"5f4e5783_0d85e3ad","line":16,"range":{"start_line":16,"start_character":22,"end_line":16,"end_character":40},"updated":"2017-10-18 15:05:25.000000000","message":"s/is associated/associated/","commit_id":"056c630a8e563e8a655db47d8bd85471e2ad279e"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"a04f27406d089707eda2c6a55b4d044667c854a4","unresolved":false,"context_lines":[{"line_number":14,"context_line":"This modified the scheduling algorithm such that users were only allowed to"},{"line_number":15,"context_line":"boot instances with PCI devices if the instance was being scheduled on at least"},{"line_number":16,"context_line":"one of the NUMA nodes is associated with the PCI devices or if the PCI devices"},{"line_number":17,"context_line":"have no information about NUMA nodes and PCI devices affinity. Before this,"},{"line_number":18,"context_line":"nova booted instances with PCI devices without checking NUMA affinity. However,"},{"line_number":19,"context_line":"such hardcoded behaviour causes problems if not every NUMA node has its own PCI"},{"line_number":20,"context_line":"device. In this case nova wouldn\u0027t allow to boot an instance on NUMA nodes"}],"source_content_type":"text/x-rst","patch_set":27,"id":"5f4e5783_adbdaf69","line":17,"range":{"start_line":17,"start_character":45,"end_line":17,"end_character":52},"updated":"2017-10-18 15:05:25.000000000","message":"device","commit_id":"056c630a8e563e8a655db47d8bd85471e2ad279e"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"a04f27406d089707eda2c6a55b4d044667c854a4","unresolved":false,"context_lines":[{"line_number":17,"context_line":"have no information about NUMA nodes and PCI devices affinity. Before this,"},{"line_number":18,"context_line":"nova booted instances with PCI devices without checking NUMA affinity. However,"},{"line_number":19,"context_line":"such hardcoded behaviour causes problems if not every NUMA node has its own PCI"},{"line_number":20,"context_line":"device. In this case nova wouldn\u0027t allow to boot an instance on NUMA nodes"},{"line_number":21,"context_line":"without PCI devices."},{"line_number":22,"context_line":""},{"line_number":23,"context_line":"Problem description"}],"source_content_type":"text/x-rst","patch_set":27,"id":"5f4e5783_cda7cb4f","line":20,"range":{"start_line":20,"start_character":41,"end_line":20,"end_character":48},"updated":"2017-10-18 15:05:25.000000000","message":"booting","commit_id":"056c630a8e563e8a655db47d8bd85471e2ad279e"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"a04f27406d089707eda2c6a55b4d044667c854a4","unresolved":false,"context_lines":[{"line_number":38,"context_line":"solution because an instance can use a PCI device if there is no information"},{"line_number":39,"context_line":"about affinity of NUMA nodes with this PCI device. This can lead to a situation"},{"line_number":40,"context_line":"whereby PCI device is not on the NUMA node which the CPU and RAM is on. The"},{"line_number":41,"context_line":"scheduling mechanism should be more flexible. The user should be able to choose"},{"line_number":42,"context_line":"between maximum performance behavior and maximum chance of successfully"},{"line_number":43,"context_line":"launching the instance."},{"line_number":44,"context_line":""},{"line_number":45,"context_line":"Of course this ability should be configurable and the current scheduling"},{"line_number":46,"context_line":"behaviour must remain as the default."}],"source_content_type":"text/x-rst","patch_set":27,"id":"5f4e5783_ad300f09","line":43,"range":{"start_line":41,"start_character":46,"end_line":43,"end_character":23},"updated":"2017-10-18 15:05:25.000000000","message":"Not sure a *user* should be able to choose this. Perhaps a *deployer* should be able to affect scheduling decisions with some toggle-able behaviour.","commit_id":"056c630a8e563e8a655db47d8bd85471e2ad279e"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"a04f27406d089707eda2c6a55b4d044667c854a4","unresolved":false,"context_lines":[{"line_number":59,"context_line":"- As an operator of a deployment with a mix of NUMA-aware and non-NUMA-aware"},{"line_number":60,"context_line":"  hosts, I want to ensure my PCI devices are always NUMA affinitized *if NUMA"},{"line_number":61,"context_line":"  information is available*. However, I still want to be able to schedule"},{"line_number":62,"context_line":"  instances of the non-NUMA-aware hosts."},{"line_number":63,"context_line":""},{"line_number":64,"context_line":"  Alternatively, as an operator with an existing deployment using PCI devices,"},{"line_number":65,"context_line":"  I don\u0027t want nova to pull the rug from under my feet and suddenly refuse to"}],"source_content_type":"text/x-rst","patch_set":27,"id":"5f4e5783_2d241fc6","line":62,"range":{"start_line":62,"start_character":12,"end_line":62,"end_character":14},"updated":"2017-10-18 15:05:25.000000000","message":"on","commit_id":"056c630a8e563e8a655db47d8bd85471e2ad279e"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"a04f27406d089707eda2c6a55b4d044667c854a4","unresolved":false,"context_lines":[{"line_number":75,"context_line":""},{"line_number":76,"context_line":"**required**"},{"line_number":77,"context_line":""},{"line_number":78,"context_line":"  This value will mean that nova will boot instances with PCI devices *only* if"},{"line_number":79,"context_line":"  at least one of the NUMA nodes is associated with these PCI devices. It means"},{"line_number":80,"context_line":"  that if NUMA node info for some PCI devices could not be determined, those"},{"line_number":81,"context_line":"  PCI devices wouldn\u0027t be consumable by the instance. This provides maximum"},{"line_number":82,"context_line":"  performance."}],"source_content_type":"text/x-rst","patch_set":27,"id":"5f4e5783_c8bb390c","line":79,"range":{"start_line":78,"start_character":2,"end_line":79,"end_character":70},"updated":"2017-10-18 15:05:25.000000000","message":"This is exceedingly difficult language to understand. How about this instead?\n\nIf the flavor or image specifies the required PCI NUMA affinity policy, then Nova will only boot the instance on compute hosts where NUMA affinity is known for host PCI devices *AND* the PCI devices requested by the flavor are affinitized to one or the NUMA nodes the instance is pinned to during scheduling.","commit_id":"056c630a8e563e8a655db47d8bd85471e2ad279e"},{"author":{"_account_id":14070,"name":"Eric Fried","email":"openstack@fried.cc","username":"efried"},"change_message_id":"161ce6e4fa72549ca73f8cd51332be672985578f","unresolved":false,"context_lines":[{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/nova/+spec/share-pci-between-numa-nodes"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"In the Juno release th \"I/O based NUMA scheduling\" spec was implemented [1]_."},{"line_number":14,"context_line":"This modified the scheduling algorithm such that users were only allowed to"},{"line_number":15,"context_line":"boot instances with PCI devices if the instance was being scheduled on at least"},{"line_number":16,"context_line":"one of the NUMA nodes is associated with the PCI devices or if the PCI devices"}],"source_content_type":"text/x-rst","patch_set":28,"id":"7f515b1d_bdf6775c","line":13,"range":{"start_line":13,"start_character":20,"end_line":13,"end_character":22},"updated":"2017-10-13 16:03:52.000000000","message":"the","commit_id":"008ad71c7d3e125aafe0667f87c2a81636bbae75"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"96a134ec3fd72355755587b39fdce5a970fd049b","unresolved":false,"context_lines":[{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/nova/+spec/share-pci-between-numa-nodes"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"In the Juno release th \"I/O based NUMA scheduling\" spec was implemented [1]_."},{"line_number":14,"context_line":"This modified the scheduling algorithm such that users were only allowed to"},{"line_number":15,"context_line":"boot instances with PCI devices if the instance was being scheduled on at least"},{"line_number":16,"context_line":"one of the NUMA nodes is associated with the PCI devices or if the PCI devices"}],"source_content_type":"text/x-rst","patch_set":28,"id":"3f4b6375_53502e33","line":13,"range":{"start_line":13,"start_character":20,"end_line":13,"end_character":22},"in_reply_to":"7f515b1d_bdf6775c","updated":"2017-10-18 09:48:04.000000000","message":"Done","commit_id":"008ad71c7d3e125aafe0667f87c2a81636bbae75"},{"author":{"_account_id":14070,"name":"Eric Fried","email":"openstack@fried.cc","username":"efried"},"change_message_id":"161ce6e4fa72549ca73f8cd51332be672985578f","unresolved":false,"context_lines":[{"line_number":13,"context_line":"In the Juno release th \"I/O based NUMA scheduling\" spec was implemented [1]_."},{"line_number":14,"context_line":"This modified the scheduling algorithm such that users were only allowed to"},{"line_number":15,"context_line":"boot instances with PCI devices if the instance was being scheduled on at least"},{"line_number":16,"context_line":"one of the NUMA nodes is associated with the PCI devices or if the PCI devices"},{"line_number":17,"context_line":"have no information about NUMA nodes and PCI devices affinity. Before this,"},{"line_number":18,"context_line":"nova booted instances with PCI devices without checking NUMA affinity. However,"},{"line_number":19,"context_line":"such hardcoded behaviour causes problems if not every NUMA node has its own PCI"}],"source_content_type":"text/x-rst","patch_set":28,"id":"5f4e5783_2454cc20","line":16,"range":{"start_line":16,"start_character":22,"end_line":16,"end_character":24},"updated":"2017-10-13 16:03:52.000000000","message":"strike","commit_id":"008ad71c7d3e125aafe0667f87c2a81636bbae75"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"96a134ec3fd72355755587b39fdce5a970fd049b","unresolved":false,"context_lines":[{"line_number":13,"context_line":"In the Juno release th \"I/O based NUMA scheduling\" spec was implemented [1]_."},{"line_number":14,"context_line":"This modified the scheduling algorithm such that users were only allowed to"},{"line_number":15,"context_line":"boot instances with PCI devices if the instance was being scheduled on at least"},{"line_number":16,"context_line":"one of the NUMA nodes is associated with the PCI devices or if the PCI devices"},{"line_number":17,"context_line":"have no information about NUMA nodes and PCI devices affinity. Before this,"},{"line_number":18,"context_line":"nova booted instances with PCI devices without checking NUMA affinity. However,"},{"line_number":19,"context_line":"such hardcoded behaviour causes problems if not every NUMA node has its own PCI"}],"source_content_type":"text/x-rst","patch_set":28,"id":"3f4b6375_134ab67f","line":16,"range":{"start_line":16,"start_character":22,"end_line":16,"end_character":24},"in_reply_to":"5f4e5783_2454cc20","updated":"2017-10-18 09:48:04.000000000","message":"Done","commit_id":"008ad71c7d3e125aafe0667f87c2a81636bbae75"},{"author":{"_account_id":14070,"name":"Eric Fried","email":"openstack@fried.cc","username":"efried"},"change_message_id":"161ce6e4fa72549ca73f8cd51332be672985578f","unresolved":false,"context_lines":[{"line_number":14,"context_line":"This modified the scheduling algorithm such that users were only allowed to"},{"line_number":15,"context_line":"boot instances with PCI devices if the instance was being scheduled on at least"},{"line_number":16,"context_line":"one of the NUMA nodes is associated with the PCI devices or if the PCI devices"},{"line_number":17,"context_line":"have no information about NUMA nodes and PCI devices affinity. Before this,"},{"line_number":18,"context_line":"nova booted instances with PCI devices without checking NUMA affinity. However,"},{"line_number":19,"context_line":"such hardcoded behaviour causes problems if not every NUMA node has its own PCI"},{"line_number":20,"context_line":"device. In this case nova wouldn\u0027t allow to boot an instance on NUMA nodes"}],"source_content_type":"text/x-rst","patch_set":28,"id":"5f4e5783_af0bf521","line":17,"range":{"start_line":17,"start_character":0,"end_line":17,"end_character":4},"updated":"2017-10-13 16:03:52.000000000","message":"had","commit_id":"008ad71c7d3e125aafe0667f87c2a81636bbae75"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"96a134ec3fd72355755587b39fdce5a970fd049b","unresolved":false,"context_lines":[{"line_number":14,"context_line":"This modified the scheduling algorithm such that users were only allowed to"},{"line_number":15,"context_line":"boot instances with PCI devices if the instance was being scheduled on at least"},{"line_number":16,"context_line":"one of the NUMA nodes is associated with the PCI devices or if the PCI devices"},{"line_number":17,"context_line":"have no information about NUMA nodes and PCI devices affinity. Before this,"},{"line_number":18,"context_line":"nova booted instances with PCI devices without checking NUMA affinity. However,"},{"line_number":19,"context_line":"such hardcoded behaviour causes problems if not every NUMA node has its own PCI"},{"line_number":20,"context_line":"device. In this case nova wouldn\u0027t allow to boot an instance on NUMA nodes"}],"source_content_type":"text/x-rst","patch_set":28,"id":"3f4b6375_7330ca08","line":17,"range":{"start_line":17,"start_character":0,"end_line":17,"end_character":4},"in_reply_to":"5f4e5783_af0bf521","updated":"2017-10-18 09:48:04.000000000","message":"Done","commit_id":"008ad71c7d3e125aafe0667f87c2a81636bbae75"},{"author":{"_account_id":14070,"name":"Eric Fried","email":"openstack@fried.cc","username":"efried"},"change_message_id":"161ce6e4fa72549ca73f8cd51332be672985578f","unresolved":false,"context_lines":[{"line_number":17,"context_line":"have no information about NUMA nodes and PCI devices affinity. Before this,"},{"line_number":18,"context_line":"nova booted instances with PCI devices without checking NUMA affinity. However,"},{"line_number":19,"context_line":"such hardcoded behaviour causes problems if not every NUMA node has its own PCI"},{"line_number":20,"context_line":"device. In this case nova wouldn\u0027t allow to boot an instance on NUMA nodes"},{"line_number":21,"context_line":"without PCI devices."},{"line_number":22,"context_line":""},{"line_number":23,"context_line":"Problem description"}],"source_content_type":"text/x-rst","patch_set":28,"id":"5f4e5783_8426e0cd","line":20,"range":{"start_line":20,"start_character":41,"end_line":20,"end_character":48},"updated":"2017-10-13 16:03:52.000000000","message":"booting","commit_id":"008ad71c7d3e125aafe0667f87c2a81636bbae75"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"96a134ec3fd72355755587b39fdce5a970fd049b","unresolved":false,"context_lines":[{"line_number":17,"context_line":"have no information about NUMA nodes and PCI devices affinity. Before this,"},{"line_number":18,"context_line":"nova booted instances with PCI devices without checking NUMA affinity. However,"},{"line_number":19,"context_line":"such hardcoded behaviour causes problems if not every NUMA node has its own PCI"},{"line_number":20,"context_line":"device. In this case nova wouldn\u0027t allow to boot an instance on NUMA nodes"},{"line_number":21,"context_line":"without PCI devices."},{"line_number":22,"context_line":""},{"line_number":23,"context_line":"Problem description"}],"source_content_type":"text/x-rst","patch_set":28,"id":"3f4b6375_f3239abd","line":20,"range":{"start_line":20,"start_character":41,"end_line":20,"end_character":48},"in_reply_to":"5f4e5783_8426e0cd","updated":"2017-10-18 09:48:04.000000000","message":"Done","commit_id":"008ad71c7d3e125aafe0667f87c2a81636bbae75"},{"author":{"_account_id":14070,"name":"Eric Fried","email":"openstack@fried.cc","username":"efried"},"change_message_id":"161ce6e4fa72549ca73f8cd51332be672985578f","unresolved":false,"context_lines":[{"line_number":24,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":25,"context_line":""},{"line_number":26,"context_line":"Booting instance with PCI devices only if at least one of the NUMA nodes is"},{"line_number":27,"context_line":"associated with the PCI devices is good for performance. This ensures there is"},{"line_number":28,"context_line":"no cross NUMA node memory traffic."},{"line_number":29,"context_line":""},{"line_number":30,"context_line":"However, if a user has an environment with two NUMA nodes and only one PCI"},{"line_number":31,"context_line":"device (for example SR-IOV card associated with first NUMA node) they would be"}],"source_content_type":"text/x-rst","patch_set":28,"id":"5f4e5783_4f3361f7","line":28,"range":{"start_line":27,"start_character":57,"end_line":28,"end_character":34},"updated":"2017-10-13 16:03:52.000000000","message":"Well, not *no* such traffic.  As I understand it, if multiple devices are specified, only one has to be on the NUMA node.  So the others still engender cross-node traffic.","commit_id":"008ad71c7d3e125aafe0667f87c2a81636bbae75"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"96a134ec3fd72355755587b39fdce5a970fd049b","unresolved":false,"context_lines":[{"line_number":24,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":25,"context_line":""},{"line_number":26,"context_line":"Booting instance with PCI devices only if at least one of the NUMA nodes is"},{"line_number":27,"context_line":"associated with the PCI devices is good for performance. This ensures there is"},{"line_number":28,"context_line":"no cross NUMA node memory traffic."},{"line_number":29,"context_line":""},{"line_number":30,"context_line":"However, if a user has an environment with two NUMA nodes and only one PCI"},{"line_number":31,"context_line":"device (for example SR-IOV card associated with first NUMA node) they would be"}],"source_content_type":"text/x-rst","patch_set":28,"id":"3f4b6375_fe112155","line":28,"range":{"start_line":27,"start_character":57,"end_line":28,"end_character":34},"in_reply_to":"5f4e5783_4f3361f7","updated":"2017-10-18 09:48:04.000000000","message":"I rephrased this whole paragraph (I didn\u0027t write any of this spec originally)","commit_id":"008ad71c7d3e125aafe0667f87c2a81636bbae75"},{"author":{"_account_id":14070,"name":"Eric Fried","email":"openstack@fried.cc","username":"efried"},"change_message_id":"161ce6e4fa72549ca73f8cd51332be672985578f","unresolved":false,"context_lines":[{"line_number":36,"context_line":""},{"line_number":37,"context_line":"In addition, the current behavior doesn\u0027t always provide the best performance"},{"line_number":38,"context_line":"solution because an instance can use a PCI device if there is no information"},{"line_number":39,"context_line":"about affinity of NUMA nodes with this PCI device. This can lead to a situation"},{"line_number":40,"context_line":"whereby PCI device is not on the NUMA node which the CPU and RAM is on. The"},{"line_number":41,"context_line":"scheduling mechanism should be more flexible. The user should be able to choose"},{"line_number":42,"context_line":"between maximum performance behavior and maximum chance of successfully"},{"line_number":43,"context_line":"launching the instance."}],"source_content_type":"text/x-rst","patch_set":28,"id":"5f4e5783_4f6121d3","line":40,"range":{"start_line":39,"start_character":51,"end_line":40,"end_character":71},"updated":"2017-10-13 16:03:52.000000000","message":"Who\u0027s responsible for setting up this association?  Seems like it was just missed in this scenario, which doesn\u0027t seem like our fault/problem.","commit_id":"008ad71c7d3e125aafe0667f87c2a81636bbae75"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"96a134ec3fd72355755587b39fdce5a970fd049b","unresolved":false,"context_lines":[{"line_number":36,"context_line":""},{"line_number":37,"context_line":"In addition, the current behavior doesn\u0027t always provide the best performance"},{"line_number":38,"context_line":"solution because an instance can use a PCI device if there is no information"},{"line_number":39,"context_line":"about affinity of NUMA nodes with this PCI device. This can lead to a situation"},{"line_number":40,"context_line":"whereby PCI device is not on the NUMA node which the CPU and RAM is on. The"},{"line_number":41,"context_line":"scheduling mechanism should be more flexible. The user should be able to choose"},{"line_number":42,"context_line":"between maximum performance behavior and maximum chance of successfully"},{"line_number":43,"context_line":"launching the instance."}],"source_content_type":"text/x-rst","patch_set":28,"id":"3f4b6375_7e4c1176","line":40,"range":{"start_line":39,"start_character":51,"end_line":40,"end_character":71},"in_reply_to":"5f4e5783_4f6121d3","updated":"2017-10-18 09:48:04.000000000","message":"Its not a manual configuration option, rather, it\u0027s because the devices itself doesn\u0027t offer the option (hardware or firmware issue) or the platform doesn\u0027t offer it (again, hardware or firmware issue). In both cases, we\u0027re not to blame for the issue. However, we are giving the impression of NUMA affinity despite not actually delivering this.","commit_id":"008ad71c7d3e125aafe0667f87c2a81636bbae75"},{"author":{"_account_id":14070,"name":"Eric Fried","email":"openstack@fried.cc","username":"efried"},"change_message_id":"161ce6e4fa72549ca73f8cd51332be672985578f","unresolved":false,"context_lines":[{"line_number":89,"context_line":"  however, if this is not possible then `nova-compute` will fall back to"},{"line_number":90,"context_line":"  scheduling on a NUMA node that is not associated with the PCI device."},{"line_number":91,"context_line":""},{"line_number":92,"context_line":"  Note that even though the ``NUMATopologyFilter`` will not consider  NUMA"},{"line_number":93,"context_line":"  affinity, the weigher proposed in the *Reserve NUMA Nodes with PCI Devices"},{"line_number":94,"context_line":"  Attached* spec [2]_ can be used to maximize the chance that a chosen host"},{"line_number":95,"context_line":"  will have NUMA-affinitized PCI devices."}],"source_content_type":"text/x-rst","patch_set":28,"id":"5f4e5783_0feb2913","line":92,"range":{"start_line":92,"start_character":68,"end_line":92,"end_character":70},"updated":"2017-10-13 16:03:52.000000000","message":"extra space","commit_id":"008ad71c7d3e125aafe0667f87c2a81636bbae75"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"96a134ec3fd72355755587b39fdce5a970fd049b","unresolved":false,"context_lines":[{"line_number":89,"context_line":"  however, if this is not possible then `nova-compute` will fall back to"},{"line_number":90,"context_line":"  scheduling on a NUMA node that is not associated with the PCI device."},{"line_number":91,"context_line":""},{"line_number":92,"context_line":"  Note that even though the ``NUMATopologyFilter`` will not consider  NUMA"},{"line_number":93,"context_line":"  affinity, the weigher proposed in the *Reserve NUMA Nodes with PCI Devices"},{"line_number":94,"context_line":"  Attached* spec [2]_ can be used to maximize the chance that a chosen host"},{"line_number":95,"context_line":"  will have NUMA-affinitized PCI devices."}],"source_content_type":"text/x-rst","patch_set":28,"id":"3f4b6375_3e569908","line":92,"range":{"start_line":92,"start_character":68,"end_line":92,"end_character":70},"in_reply_to":"5f4e5783_0feb2913","updated":"2017-10-18 09:48:04.000000000","message":"Done","commit_id":"008ad71c7d3e125aafe0667f87c2a81636bbae75"},{"author":{"_account_id":8768,"name":"Chris Friesen","email":"chris.friesen@windriver.com","username":"cbf123"},"change_message_id":"76f4877810728480568cbd9c990b127babc11741","unresolved":false,"context_lines":[{"line_number":94,"context_line":"  Attached* spec [2]_ can be used to maximize the chance that a chosen host"},{"line_number":95,"context_line":"  will have NUMA-affinitized PCI devices."},{"line_number":96,"context_line":""},{"line_number":97,"context_line":"**legacy**"},{"line_number":98,"context_line":""},{"line_number":99,"context_line":"  This is the default value and it describes the current nova behavior. Usually"},{"line_number":100,"context_line":"  we have information about association of PCI devices with NUMA nodes."},{"line_number":101,"context_line":"  However, some PCI devices do not provide such information. The ``legacy``"},{"line_number":102,"context_line":"  value will mean that nova will boot instances with PCI device if either:"},{"line_number":103,"context_line":""},{"line_number":104,"context_line":"  * The PCI device is associated with at least one NUMA nodes on which the"},{"line_number":105,"context_line":"    instance will be booted"},{"line_number":106,"context_line":""},{"line_number":107,"context_line":"  * There is no information about PCI-NUMA affinity available"},{"line_number":108,"context_line":""},{"line_number":109,"context_line":"If both image and flavor properties are not set (equals ``None``) the"},{"line_number":110,"context_line":"``legacy`` policy will be used. If one of image *or* flavor property is not set"},{"line_number":111,"context_line":"(equals ``None``) but the other property is set then the value of the set"},{"line_number":112,"context_line":"property will be used. In a case of conflicts between flavor and image"},{"line_number":113,"context_line":"properties (both properties are set and they are not equal) an exception will"},{"line_number":114,"context_line":"be raised."},{"line_number":115,"context_line":""},{"line_number":116,"context_line":"Alternatives"},{"line_number":117,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":28,"id":"5f4e5783_5a48732b","line":114,"range":{"start_line":97,"start_character":0,"end_line":114,"end_character":10},"updated":"2017-10-17 15:25:27.000000000","message":"There was a proposal at the PTG to drop the \"legacy\" value and instead have \"required\" support the legacy behaviour for one release while emitting a warning, then drop the legacy behaviour in favour of the more strict behaviour.  I think this is worth mentioning, if only to be explicit about how we want to deal with it.\n\nIf \"don\u0027t change behaviour for existing flavours\" reigns supreme, then defaulting to \"legacy\" is probably the best bet.  If instead we think the current behaviour is buggy then it makes sense to fix it.\n\nAre there any legitimate cases where we would not have NUMA affinity available?  (In which case we probably need to keep the legacy option around.)","commit_id":"008ad71c7d3e125aafe0667f87c2a81636bbae75"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"e051163800d83eda265ba1a81e0986f5fa1d53b2","unresolved":false,"context_lines":[{"line_number":94,"context_line":"  Attached* spec [2]_ can be used to maximize the chance that a chosen host"},{"line_number":95,"context_line":"  will have NUMA-affinitized PCI devices."},{"line_number":96,"context_line":""},{"line_number":97,"context_line":"**legacy**"},{"line_number":98,"context_line":""},{"line_number":99,"context_line":"  This is the default value and it describes the current nova behavior. Usually"},{"line_number":100,"context_line":"  we have information about association of PCI devices with NUMA nodes."},{"line_number":101,"context_line":"  However, some PCI devices do not provide such information. The ``legacy``"},{"line_number":102,"context_line":"  value will mean that nova will boot instances with PCI device if either:"},{"line_number":103,"context_line":""},{"line_number":104,"context_line":"  * The PCI device is associated with at least one NUMA nodes on which the"},{"line_number":105,"context_line":"    instance will be booted"},{"line_number":106,"context_line":""},{"line_number":107,"context_line":"  * There is no information about PCI-NUMA affinity available"},{"line_number":108,"context_line":""},{"line_number":109,"context_line":"If both image and flavor properties are not set (equals ``None``) the"},{"line_number":110,"context_line":"``legacy`` policy will be used. If one of image *or* flavor property is not set"},{"line_number":111,"context_line":"(equals ``None``) but the other property is set then the value of the set"},{"line_number":112,"context_line":"property will be used. In a case of conflicts between flavor and image"},{"line_number":113,"context_line":"properties (both properties are set and they are not equal) an exception will"},{"line_number":114,"context_line":"be raised."},{"line_number":115,"context_line":""},{"line_number":116,"context_line":"Alternatives"},{"line_number":117,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":28,"id":"3f4b6375_5ed71525","line":114,"range":{"start_line":97,"start_character":0,"end_line":114,"end_character":10},"in_reply_to":"5f4e5783_5a48732b","updated":"2017-10-18 09:44:43.000000000","message":"I\u0027m more in favor of deprecating the legacy value (so we could correctly write a release note for that) and ask operators to change that opt by the next cycle, rather than making the same opt value having a different behaviour between cycles honestly.\n\nLike, we could directly start with \u0027legacy\u0027 being a deprecated value for Queens, and properly communicate to operators they need to change it in Rocky.","commit_id":"008ad71c7d3e125aafe0667f87c2a81636bbae75"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"96a134ec3fd72355755587b39fdce5a970fd049b","unresolved":false,"context_lines":[{"line_number":94,"context_line":"  Attached* spec [2]_ can be used to maximize the chance that a chosen host"},{"line_number":95,"context_line":"  will have NUMA-affinitized PCI devices."},{"line_number":96,"context_line":""},{"line_number":97,"context_line":"**legacy**"},{"line_number":98,"context_line":""},{"line_number":99,"context_line":"  This is the default value and it describes the current nova behavior. Usually"},{"line_number":100,"context_line":"  we have information about association of PCI devices with NUMA nodes."},{"line_number":101,"context_line":"  However, some PCI devices do not provide such information. The ``legacy``"},{"line_number":102,"context_line":"  value will mean that nova will boot instances with PCI device if either:"},{"line_number":103,"context_line":""},{"line_number":104,"context_line":"  * The PCI device is associated with at least one NUMA nodes on which the"},{"line_number":105,"context_line":"    instance will be booted"},{"line_number":106,"context_line":""},{"line_number":107,"context_line":"  * There is no information about PCI-NUMA affinity available"},{"line_number":108,"context_line":""},{"line_number":109,"context_line":"If both image and flavor properties are not set (equals ``None``) the"},{"line_number":110,"context_line":"``legacy`` policy will be used. If one of image *or* flavor property is not set"},{"line_number":111,"context_line":"(equals ``None``) but the other property is set then the value of the set"},{"line_number":112,"context_line":"property will be used. In a case of conflicts between flavor and image"},{"line_number":113,"context_line":"properties (both properties are set and they are not equal) an exception will"},{"line_number":114,"context_line":"be raised."},{"line_number":115,"context_line":""},{"line_number":116,"context_line":"Alternatives"},{"line_number":117,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":28,"id":"3f4b6375_9ee22d2f","line":114,"range":{"start_line":97,"start_character":0,"end_line":114,"end_character":10},"in_reply_to":"5f4e5783_5a48732b","updated":"2017-10-18 09:48:04.000000000","message":"sean-k-mooney specifically called out Xeon\u0027s integrated FPGAs as one such device that would not have NUMA affinity available. I\u0027ve noted this in the doc","commit_id":"008ad71c7d3e125aafe0667f87c2a81636bbae75"},{"author":{"_account_id":14070,"name":"Eric Fried","email":"openstack@fried.cc","username":"efried"},"change_message_id":"161ce6e4fa72549ca73f8cd51332be672985578f","unresolved":false,"context_lines":[{"line_number":119,"context_line":"- Change placement behavior to *not* boot instances which need no PCI devices"},{"line_number":120,"context_line":"  on NUMA nodes with PCI devices if possible. This would maximize the"},{"line_number":121,"context_line":"  possibility that a instance could find a suitable host to boot on."},{"line_number":122,"context_line":"  Unfortunately it does not fully solve the problem. Consider the case where a"},{"line_number":123,"context_line":"  host has two NUMA nodes: the first node has PCI devices while the second one"},{"line_number":124,"context_line":"  does not. If the first node is fully occupied by instances with PCI devices,"},{"line_number":125,"context_line":"  then you would be unable to utilize the second node as it has no PCI devices."},{"line_number":126,"context_line":""},{"line_number":127,"context_line":"- Make the PCI NUMA strictness part of the individual PCI device request. This"},{"line_number":128,"context_line":"  would allow us to represent requests like \"I need to be strictly affined to"}],"source_content_type":"text/x-rst","patch_set":28,"id":"5f4e5783_af5c15ed","line":125,"range":{"start_line":122,"start_character":53,"end_line":125,"end_character":79},"updated":"2017-10-13 16:03:52.000000000","message":"I think this is being expressed backwards.  The full one is the one *without* PCI devices, and now you can\u0027t boot an instance that doesn\u0027t need a PCI device because of this rule.\n\nExcept \"if possible\" on L120 implies that you could still do this - just with the kind of semantic in **preferred** above.\n\nI think what I\u0027m getting at is that it would be a reasonable addition to this spec to say that: \"Instances with no PCI devices will be preferentially booted to NUMA nodes with no associated PCI devices.  If no such NUMA nodes are available, the instance will be booted to any available NUMA node.\"\n\nNot saying we should do this.  Just saying I don\u0027t see the issue described in this paragraph as an actual problem.","commit_id":"008ad71c7d3e125aafe0667f87c2a81636bbae75"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"96a134ec3fd72355755587b39fdce5a970fd049b","unresolved":false,"context_lines":[{"line_number":119,"context_line":"- Change placement behavior to *not* boot instances which need no PCI devices"},{"line_number":120,"context_line":"  on NUMA nodes with PCI devices if possible. This would maximize the"},{"line_number":121,"context_line":"  possibility that a instance could find a suitable host to boot on."},{"line_number":122,"context_line":"  Unfortunately it does not fully solve the problem. Consider the case where a"},{"line_number":123,"context_line":"  host has two NUMA nodes: the first node has PCI devices while the second one"},{"line_number":124,"context_line":"  does not. If the first node is fully occupied by instances with PCI devices,"},{"line_number":125,"context_line":"  then you would be unable to utilize the second node as it has no PCI devices."},{"line_number":126,"context_line":""},{"line_number":127,"context_line":"- Make the PCI NUMA strictness part of the individual PCI device request. This"},{"line_number":128,"context_line":"  would allow us to represent requests like \"I need to be strictly affined to"}],"source_content_type":"text/x-rst","patch_set":28,"id":"3f4b6375_1ebabddd","line":125,"range":{"start_line":122,"start_character":53,"end_line":125,"end_character":79},"in_reply_to":"5f4e5783_af5c15ed","updated":"2017-10-18 09:48:04.000000000","message":"I think I\u0027ve clarified this now by splitting it into two alternatives, one stricter than the other. Lemme know what you think","commit_id":"008ad71c7d3e125aafe0667f87c2a81636bbae75"},{"author":{"_account_id":14070,"name":"Eric Fried","email":"openstack@fried.cc","username":"efried"},"change_message_id":"161ce6e4fa72549ca73f8cd51332be672985578f","unresolved":false,"context_lines":[{"line_number":180,"context_line":""},{"line_number":181,"context_line":"If the ``legacy`` policy is selected, the existing nova behaviour will be"},{"line_number":182,"context_line":"retained and performance will remain unchanged."},{"line_number":183,"context_line":""},{"line_number":184,"context_line":"Other deployer impact"},{"line_number":185,"context_line":"---------------------"},{"line_number":186,"context_line":""}],"source_content_type":"text/x-rst","patch_set":28,"id":"5f4e5783_2f72654a","line":183,"updated":"2017-10-13 16:03:52.000000000","message":"Is there any appreciable performance impact to the scheduling process itself to process this new logic?","commit_id":"008ad71c7d3e125aafe0667f87c2a81636bbae75"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"96a134ec3fd72355755587b39fdce5a970fd049b","unresolved":false,"context_lines":[{"line_number":180,"context_line":""},{"line_number":181,"context_line":"If the ``legacy`` policy is selected, the existing nova behaviour will be"},{"line_number":182,"context_line":"retained and performance will remain unchanged."},{"line_number":183,"context_line":""},{"line_number":184,"context_line":"Other deployer impact"},{"line_number":185,"context_line":"---------------------"},{"line_number":186,"context_line":""}],"source_content_type":"text/x-rst","patch_set":28,"id":"3f4b6375_be76093e","line":183,"in_reply_to":"5f4e5783_2f72654a","updated":"2017-10-18 09:48:04.000000000","message":"Done","commit_id":"008ad71c7d3e125aafe0667f87c2a81636bbae75"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"e051163800d83eda265ba1a81e0986f5fa1d53b2","unresolved":false,"context_lines":[{"line_number":180,"context_line":""},{"line_number":181,"context_line":"If the ``legacy`` policy is selected, the existing nova behaviour will be"},{"line_number":182,"context_line":"retained and performance will remain unchanged."},{"line_number":183,"context_line":""},{"line_number":184,"context_line":"Other deployer impact"},{"line_number":185,"context_line":"---------------------"},{"line_number":186,"context_line":""}],"source_content_type":"text/x-rst","patch_set":28,"id":"3f4b6375_be6cc99c","line":183,"in_reply_to":"5f4e5783_2f72654a","updated":"2017-10-18 09:44:43.000000000","message":"I don\u0027t see any performance impact, but Stephen could confirm.","commit_id":"008ad71c7d3e125aafe0667f87c2a81636bbae75"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"a04f27406d089707eda2c6a55b4d044667c854a4","unresolved":false,"context_lines":[{"line_number":25,"context_line":""},{"line_number":26,"context_line":"In its current iteration, nova boots instances with PCI devices on the same"},{"line_number":27,"context_line":"NUMA nodes that these PCI devices are associated with. This is good for"},{"line_number":28,"context_line":"performances, as it ensures there is limited cross-NUMA node memory traffic."},{"line_number":29,"context_line":"However, if a user has an environment with two NUMA nodes and only one PCI"},{"line_number":30,"context_line":"device (for example SR-IOV card associated with first NUMA node) they would be"},{"line_number":31,"context_line":"able to boot instance with *one* NUMA node and SR-IOV ports only on the first"}],"source_content_type":"text/x-rst","patch_set":29,"id":"3f4b6375_3fee9088","line":28,"range":{"start_line":28,"start_character":0,"end_line":28,"end_character":12},"updated":"2017-10-18 15:05:25.000000000","message":"performance","commit_id":"74e9bf897d388a1d058b0122a0450c5f2d635cb5"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"a04f27406d089707eda2c6a55b4d044667c854a4","unresolved":false,"context_lines":[{"line_number":111,"context_line":"  integrated on to the dies of recent Intel Xeon chips, which hook into the QPI"},{"line_number":112,"context_line":"  bus and therefore have no NUMA affinity [3]_."},{"line_number":113,"context_line":""},{"line_number":114,"context_line":"If both image and flavor properties are not set (equals ``None``) the"},{"line_number":115,"context_line":"``legacy`` policy will be used. If one of image *or* flavor property is not set"},{"line_number":116,"context_line":"(equals ``None``) but the other property is set then the value of the set"},{"line_number":117,"context_line":"property will be used. In a case of conflicts between flavor and image"}],"source_content_type":"text/x-rst","patch_set":29,"id":"3f4b6375_b073cd4d","line":114,"range":{"start_line":114,"start_character":49,"end_line":114,"end_character":64},"updated":"2017-10-18 15:05:25.000000000","message":"I\u0027d just as soon leave this out, frankly. The lack of existence of an extra spec or metadata key does not mean the key equals None. :)","commit_id":"74e9bf897d388a1d058b0122a0450c5f2d635cb5"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"a04f27406d089707eda2c6a55b4d044667c854a4","unresolved":false,"context_lines":[{"line_number":113,"context_line":""},{"line_number":114,"context_line":"If both image and flavor properties are not set (equals ``None``) the"},{"line_number":115,"context_line":"``legacy`` policy will be used. If one of image *or* flavor property is not set"},{"line_number":116,"context_line":"(equals ``None``) but the other property is set then the value of the set"},{"line_number":117,"context_line":"property will be used. In a case of conflicts between flavor and image"},{"line_number":118,"context_line":"properties (both properties are set and they are not equal) an exception will"},{"line_number":119,"context_line":"be raised."}],"source_content_type":"text/x-rst","patch_set":29,"id":"3f4b6375_908e9125","line":116,"range":{"start_line":116,"start_character":0,"end_line":116,"end_character":17},"updated":"2017-10-18 15:05:25.000000000","message":"ditto above.","commit_id":"74e9bf897d388a1d058b0122a0450c5f2d635cb5"},{"author":{"_account_id":14070,"name":"Eric Fried","email":"openstack@fried.cc","username":"efried"},"change_message_id":"e38b55e925c40bf7bfd0fde83c340a66f887f7a7","unresolved":false,"context_lines":[{"line_number":128,"context_line":"  many instances without PCI devices would result in a large number of unused,"},{"line_number":129,"context_line":"  PCI device-having hosts."},{"line_number":130,"context_line":""},{"line_number":131,"context_line":"- Change placement behavior to *avoid* booting instances with PCI devices on"},{"line_number":132,"context_line":"  NUMA nodes with PCI devices *if possible*. This is a softer version of the"},{"line_number":133,"context_line":"  first alternative and has actually been address by the"},{"line_number":134,"context_line":"  \u0027reserve-numa-with-pci\u0027 spec [4]_. Unfortunately this does not fully solve"}],"source_content_type":"text/x-rst","patch_set":29,"id":"3f4b6375_64935f51","line":131,"range":{"start_line":131,"start_character":57,"end_line":131,"end_character":61},"updated":"2017-10-18 13:26:25.000000000","message":"without?","commit_id":"74e9bf897d388a1d058b0122a0450c5f2d635cb5"},{"author":{"_account_id":14070,"name":"Eric Fried","email":"openstack@fried.cc","username":"efried"},"change_message_id":"e38b55e925c40bf7bfd0fde83c340a66f887f7a7","unresolved":false,"context_lines":[{"line_number":130,"context_line":""},{"line_number":131,"context_line":"- Change placement behavior to *avoid* booting instances with PCI devices on"},{"line_number":132,"context_line":"  NUMA nodes with PCI devices *if possible*. This is a softer version of the"},{"line_number":133,"context_line":"  first alternative and has actually been address by the"},{"line_number":134,"context_line":"  \u0027reserve-numa-with-pci\u0027 spec [4]_. Unfortunately this does not fully solve"},{"line_number":135,"context_line":"  the problem. Consider the case where a host has two NUMA nodes: the first"},{"line_number":136,"context_line":"  node has PCI devices while the second one does not. If the first node is"}],"source_content_type":"text/x-rst","patch_set":29,"id":"3f4b6375_04214b4e","line":133,"range":{"start_line":133,"start_character":42,"end_line":133,"end_character":49},"updated":"2017-10-18 13:26:25.000000000","message":"addressed","commit_id":"74e9bf897d388a1d058b0122a0450c5f2d635cb5"},{"author":{"_account_id":14070,"name":"Eric Fried","email":"openstack@fried.cc","username":"efried"},"change_message_id":"e38b55e925c40bf7bfd0fde83c340a66f887f7a7","unresolved":false,"context_lines":[{"line_number":131,"context_line":"- Change placement behavior to *avoid* booting instances with PCI devices on"},{"line_number":132,"context_line":"  NUMA nodes with PCI devices *if possible*. This is a softer version of the"},{"line_number":133,"context_line":"  first alternative and has actually been address by the"},{"line_number":134,"context_line":"  \u0027reserve-numa-with-pci\u0027 spec [4]_. Unfortunately this does not fully solve"},{"line_number":135,"context_line":"  the problem. Consider the case where a host has two NUMA nodes: the first"},{"line_number":136,"context_line":"  node has PCI devices while the second one does not. If the first node is"},{"line_number":137,"context_line":"  fully occupied by instances with PCI devices then you would be unable to"},{"line_number":138,"context_line":"  utilize the second node as it has no PCI devices."},{"line_number":139,"context_line":""},{"line_number":140,"context_line":"- Make the PCI NUMA strictness part of the individual PCI device request. This"},{"line_number":141,"context_line":"  would allow us to represent requests like \"I need to be strictly affined to"}],"source_content_type":"text/x-rst","patch_set":29,"id":"3f4b6375_e4798f2f","line":138,"range":{"start_line":134,"start_character":37,"end_line":138,"end_character":51},"updated":"2017-10-18 13:26:25.000000000","message":"This still feels like a statement of a non-problem to me.  It\u0027s saying you\u0027re saturating the PCI-device-having NUMA node with instances that need PCI devices, and then you can\u0027t deploy any more instances that need PCI devices.  As we used to say in grade school, duh.  But you would still be able to deploy instances that *don\u0027t* need PCI devices there.\n\nSo this whole section still doesn\u0027t gel with me.\n\nI feel like there\u0027s only one obvious thing to be expressed here: *preferentially* boot PCI-needing instances on NUMA nodes *with* PCI devices; and *preferentially* boot non-PCI-needing instances on NUMA nodes *without* PCI devices.  Meaning if the preferred thingy is already saturated, you fall back to the other.  In the case of PCI-needing instances, the fallback scenario just means your devices aren\u0027t NUMA-affined - which is acceptable with the \"preferred\" option but rejected with the \"required\" option.  In the case of non-PCI-needing instances, the fallback means you\u0027re taking up space that would prevent PCI-needing instances from being nicely affined.\n\nThis is *almost* the solution proposed by the spec anyway.  The thing that\u0027s missing is how to treat non-PCI-needing instances - but it sounds like that\u0027s addressed by this other spec (which I didn\u0027t look at).","commit_id":"74e9bf897d388a1d058b0122a0450c5f2d635cb5"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"a04f27406d089707eda2c6a55b4d044667c854a4","unresolved":false,"context_lines":[{"line_number":131,"context_line":"- Change placement behavior to *avoid* booting instances with PCI devices on"},{"line_number":132,"context_line":"  NUMA nodes with PCI devices *if possible*. This is a softer version of the"},{"line_number":133,"context_line":"  first alternative and has actually been address by the"},{"line_number":134,"context_line":"  \u0027reserve-numa-with-pci\u0027 spec [4]_. Unfortunately this does not fully solve"},{"line_number":135,"context_line":"  the problem. Consider the case where a host has two NUMA nodes: the first"},{"line_number":136,"context_line":"  node has PCI devices while the second one does not. If the first node is"},{"line_number":137,"context_line":"  fully occupied by instances with PCI devices then you would be unable to"},{"line_number":138,"context_line":"  utilize the second node as it has no PCI devices."},{"line_number":139,"context_line":""},{"line_number":140,"context_line":"- Make the PCI NUMA strictness part of the individual PCI device request. This"},{"line_number":141,"context_line":"  would allow us to represent requests like \"I need to be strictly affined to"}],"source_content_type":"text/x-rst","patch_set":29,"id":"3f4b6375_d0d9a922","line":138,"range":{"start_line":134,"start_character":37,"end_line":138,"end_character":51},"in_reply_to":"3f4b6375_e4798f2f","updated":"2017-10-18 15:05:25.000000000","message":"\u003e This still feels like a statement of a non-problem to me.  It\u0027s\n \u003e saying you\u0027re saturating the PCI-device-having NUMA node with\n \u003e instances that need PCI devices, and then you can\u0027t deploy any more\n \u003e instances that need PCI devices.  As we used to say in grade\n \u003e school, duh.  But you would still be able to deploy instances that\n \u003e *don\u0027t* need PCI devices there.\n\nNo, actually Sergey is describing enforcing a policy that would *prevent* flavors that do *NOT* require PCI devices from being launched on any node that *DOES* having PCI devices.\n\nRegardless, though, the point stands... this is merely an alternative to the proposed solution.","commit_id":"74e9bf897d388a1d058b0122a0450c5f2d635cb5"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"a04f27406d089707eda2c6a55b4d044667c854a4","unresolved":false,"context_lines":[{"line_number":139,"context_line":""},{"line_number":140,"context_line":"- Make the PCI NUMA strictness part of the individual PCI device request. This"},{"line_number":141,"context_line":"  would allow us to represent requests like \"I need to be strictly affined to"},{"line_number":142,"context_line":"  this NIC, but I don\u0027t need to be strictly affined to this FPGA\". It is very"},{"line_number":143,"context_line":"  unlikely that this level of granularity of request would be required. In"},{"line_number":144,"context_line":"  addition, it\u0027s difficult to see how this would fit into the resource provider"},{"line_number":145,"context_line":"  world in the future as the problem is transformed from a scheduling one (at"},{"line_number":146,"context_line":"  the host level) to a placement one."}],"source_content_type":"text/x-rst","patch_set":29,"id":"3f4b6375_700af58a","line":143,"range":{"start_line":142,"start_character":67,"end_line":143,"end_character":71},"updated":"2017-10-18 15:05:25.000000000","message":"famous last words. another famous last word: NFV.","commit_id":"74e9bf897d388a1d058b0122a0450c5f2d635cb5"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"702b219eeb9e344fa48f4e1a6c33fc16fa9c90eb","unresolved":false,"context_lines":[{"line_number":80,"context_line":"  PCI devices wouldn\u0027t be consumable by the instance. This provides maximum"},{"line_number":81,"context_line":"  performance."},{"line_number":82,"context_line":""},{"line_number":83,"context_line":"**preferred**"},{"line_number":84,"context_line":""},{"line_number":85,"context_line":"  This value will mean that `nova-scheduler` will choose a compute host with"},{"line_number":86,"context_line":"  minimal consideration for the NUMA affinity of PCI devices. `nova-compute`"}],"source_content_type":"text/x-rst","patch_set":30,"id":"3f4b6375_ff1a3115","line":83,"range":{"start_line":83,"start_character":2,"end_line":83,"end_character":11},"updated":"2017-10-18 17:22:01.000000000","message":"im still think legacy should be called preferred and this should be called something else but since this is merged I wont belabour that further.","commit_id":"91fb90589d61bfcd489b2684742a0a926347460b"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"702b219eeb9e344fa48f4e1a6c33fc16fa9c90eb","unresolved":false,"context_lines":[{"line_number":108,"context_line":"  This is required because the configuration option will apply globally to an"},{"line_number":109,"context_line":"  instance which may have multiple devices attached, and not all of these"},{"line_number":110,"context_line":"  devices may have NUMA affinity. An example of such a device is the FPGAs"},{"line_number":111,"context_line":"  integrated on to the dies of recent Intel Xeon chips, which hook into the QPI"},{"line_number":112,"context_line":"  bus and therefore have no NUMA affinity [3]_."},{"line_number":113,"context_line":""},{"line_number":114,"context_line":"If both image and flavor properties are not set (equals ``None``) the"},{"line_number":115,"context_line":"``legacy`` policy will be used. If one of image *or* flavor property is not set"}],"source_content_type":"text/x-rst","patch_set":30,"id":"3f4b6375_7fb521ee","line":112,"range":{"start_line":111,"start_character":31,"end_line":112,"end_character":47},"updated":"2017-10-18 17:22:01.000000000","message":"they can have numa affinity in the same way pci deviced do. pci devices get there affinity from where the\nthere config space is mem mappend by there driver. but they are not required to have numa affinity either as the fpga may not use system memory at all.\n\na better example would have been pci devices that don\u0027t support mmio/dma. actully that is true of any device.\nif the device does not support mmio or dma it has no numa affinity as you do not interact with the device via a memory mapped region. it still may have socket affinity as it will be phyicall wired to the pins on the socket but not numa affintiy","commit_id":"91fb90589d61bfcd489b2684742a0a926347460b"}]}