)]}'
{"specs/rocky/approved/resource-class-affinity.rst":[{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"24cc50498ab310845652ca57e4f8a17286e6ac07","unresolved":false,"context_lines":[{"line_number":12,"context_line":""},{"line_number":13,"context_line":"This spec describes a limited but simple mechanism to support modeling and"},{"line_number":14,"context_line":"exploiting NUMA via placement. The design does not attempt to solve the entire"},{"line_number":15,"context_line":"issue of affinity and anti-affinity. Rather, it allows us to address the"},{"line_number":16,"context_line":"majority of use cases in a way that\u0027s easy to understand, model, and use; and"},{"line_number":17,"context_line":"which does not restrict future development of an all-encompassing generic"},{"line_number":18,"context_line":"affinity/anti-affinity solution."},{"line_number":19,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_67e06427","line":16,"range":{"start_line":15,"start_character":69,"end_line":16,"end_character":21},"updated":"2018-02-12 14:58:00.000000000","message":"Not sure I\u0027d say that NUMA scenarios are the \"majority\" of use cases for affinity/anti-affinity. IMHO, the majority of use cases for affinity/anti-affinity revolve around HA/disaster recovery (anti-affinity) and network latency requirements (affinity).\n\nNUMA is more about pinning (v)CPUs to a socket/package/node in order to prevent/reduce cross-bus memory accesses. Pinning isn\u0027t affinity/anti-affinity but rather a completely different scheduling algorithm that pre-supposes a particular topology and runs a NUMA node-placement algorithm against each possible topology exposed by a piece of hardware...","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":14070,"name":"Eric Fried","email":"openstack@fried.cc","username":"efried"},"change_message_id":"aa1a50b41f87b093aa395382e11cf203fa6ba88e","unresolved":false,"context_lines":[{"line_number":12,"context_line":""},{"line_number":13,"context_line":"This spec describes a limited but simple mechanism to support modeling and"},{"line_number":14,"context_line":"exploiting NUMA via placement. The design does not attempt to solve the entire"},{"line_number":15,"context_line":"issue of affinity and anti-affinity. Rather, it allows us to address the"},{"line_number":16,"context_line":"majority of use cases in a way that\u0027s easy to understand, model, and use; and"},{"line_number":17,"context_line":"which does not restrict future development of an all-encompassing generic"},{"line_number":18,"context_line":"affinity/anti-affinity solution."},{"line_number":19,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_6df7e55a","line":16,"range":{"start_line":15,"start_character":69,"end_line":16,"end_character":21},"in_reply_to":"3fa0c359_67e06427","updated":"2018-02-12 16:50:10.000000000","message":"\u003e Not sure I\u0027d say that NUMA scenarios are the \"majority\" of use\n \u003e cases for affinity/anti-affinity.\n\nSorry, I should say \"majority of NUMA use cases\".\n\n \u003e IMHO, the majority of use cases\n \u003e for affinity/anti-affinity revolve around HA/disaster recovery\n \u003e (anti-affinity) and network latency requirements (affinity).\n\nWhich is the stuff we noodled on Friday, which we (or at least I) concluded wasn\u0027t really in the same problem/solution domain as NUMA, which is why I felt comfortable proposing a spec to fill that space.\n\n \u003e NUMA is more about pinning (v)CPUs to a socket/package/node in\n \u003e order to prevent/reduce cross-bus memory accesses.\n\nCool, that gels with my understanding.\n\n \u003e Pinning isn\u0027t affinity/anti-affinity\n\nIn the sense that you have to pick CPUs, memory blocks, and devices that are \"stuck together\".  I asked sean-k-mooney for proper terminology and was told \"affinity\" [1].  Perhaps I misunderstood the nuance.  In any case, if \"affinity\" isn\u0027t the right term, I can s//g it, but the thrust of the design remains.\n\n \u003e pre-supposes a particular topology\n\nBingo, this is where we give virt (via update_provider_tree) the ability to model that topology in such a way that it can be understood and addressed for NUMA purposes.\n\n \u003e and runs a NUMA node-placement algorithm\n\nWhich we get (almost) for free from placement via this design.\n\nMore below.\n\n[1] http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2018-02-09.log.html#t2018-02-09T14:45:07","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"24cc50498ab310845652ca57e4f8a17286e6ac07","unresolved":false,"context_lines":[{"line_number":19,"context_line":""},{"line_number":20,"context_line":"The crux is the ability to declare, within a request for allocation candidates,"},{"line_number":21,"context_line":"a list of resource classes whose resources must come from providers in the same"},{"line_number":22,"context_line":"aggregate. Virt drivers implementing `update_provider_tree`_ model each NUMA"},{"line_number":23,"context_line":"node as a branch off of the root provider, associating each provider in the"},{"line_number":24,"context_line":"branch with a single aggregate. Operators design flavors containing the list of"},{"line_number":25,"context_line":"the resource classes which must come from the same NUMA node. The scheduler"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_4753e8e2","line":22,"range":{"start_line":22,"start_character":0,"end_line":22,"end_character":9},"updated":"2018-02-12 14:58:00.000000000","message":"aggregate? or subtree?","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":14070,"name":"Eric Fried","email":"openstack@fried.cc","username":"efried"},"change_message_id":"aa1a50b41f87b093aa395382e11cf203fa6ba88e","unresolved":false,"context_lines":[{"line_number":19,"context_line":""},{"line_number":20,"context_line":"The crux is the ability to declare, within a request for allocation candidates,"},{"line_number":21,"context_line":"a list of resource classes whose resources must come from providers in the same"},{"line_number":22,"context_line":"aggregate. Virt drivers implementing `update_provider_tree`_ model each NUMA"},{"line_number":23,"context_line":"node as a branch off of the root provider, associating each provider in the"},{"line_number":24,"context_line":"branch with a single aggregate. Operators design flavors containing the list of"},{"line_number":25,"context_line":"the resource classes which must come from the same NUMA node. The scheduler"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_18021d0b","line":22,"range":{"start_line":22,"start_character":0,"end_line":22,"end_character":9},"in_reply_to":"3fa0c359_4753e8e2","updated":"2018-02-12 16:50:10.000000000","message":"This is the key point of the design.  The group of providers comprising the NUMA node (socket? package?) is identified by associating those providers with an aggregate.  And it will usually happen to correspond to a subtree (it doesn\u0027t have to, but it would be kind of weird if it didn\u0027t).","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"24cc50498ab310845652ca57e4f8a17286e6ac07","unresolved":false,"context_lines":[{"line_number":20,"context_line":"The crux is the ability to declare, within a request for allocation candidates,"},{"line_number":21,"context_line":"a list of resource classes whose resources must come from providers in the same"},{"line_number":22,"context_line":"aggregate. Virt drivers implementing `update_provider_tree`_ model each NUMA"},{"line_number":23,"context_line":"node as a branch off of the root provider, associating each provider in the"},{"line_number":24,"context_line":"branch with a single aggregate. Operators design flavors containing the list of"},{"line_number":25,"context_line":"the resource classes which must come from the same NUMA node. The scheduler"},{"line_number":26,"context_line":"passes that list to placement, which returns only allocation candidates which"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_674e2439","line":23,"range":{"start_line":23,"start_character":10,"end_line":23,"end_character":16},"updated":"2018-02-12 14:58:00.000000000","message":"child provider of the compute node root provider.","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":14070,"name":"Eric Fried","email":"openstack@fried.cc","username":"efried"},"change_message_id":"aa1a50b41f87b093aa395382e11cf203fa6ba88e","unresolved":false,"context_lines":[{"line_number":20,"context_line":"The crux is the ability to declare, within a request for allocation candidates,"},{"line_number":21,"context_line":"a list of resource classes whose resources must come from providers in the same"},{"line_number":22,"context_line":"aggregate. Virt drivers implementing `update_provider_tree`_ model each NUMA"},{"line_number":23,"context_line":"node as a branch off of the root provider, associating each provider in the"},{"line_number":24,"context_line":"branch with a single aggregate. Operators design flavors containing the list of"},{"line_number":25,"context_line":"the resource classes which must come from the same NUMA node. The scheduler"},{"line_number":26,"context_line":"passes that list to placement, which returns only allocation candidates which"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_58b7453b","line":23,"range":{"start_line":23,"start_character":10,"end_line":23,"end_character":16},"in_reply_to":"3fa0c359_674e2439","updated":"2018-02-12 16:50:10.000000000","message":"No, a branch.  If all the resources in the NUMA node could be provided by a single RP, then as you imply below, we wouldn\u0027t need anything like this - we could just use granular resource groups.  But see the lovely ASCII diagrams - we want to be able to express e.g. NIC ports as having affinity to a NUMA node.","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"24cc50498ab310845652ca57e4f8a17286e6ac07","unresolved":false,"context_lines":[{"line_number":21,"context_line":"a list of resource classes whose resources must come from providers in the same"},{"line_number":22,"context_line":"aggregate. Virt drivers implementing `update_provider_tree`_ model each NUMA"},{"line_number":23,"context_line":"node as a branch off of the root provider, associating each provider in the"},{"line_number":24,"context_line":"branch with a single aggregate. Operators design flavors containing the list of"},{"line_number":25,"context_line":"the resource classes which must come from the same NUMA node. The scheduler"},{"line_number":26,"context_line":"passes that list to placement, which returns only allocation candidates which"},{"line_number":27,"context_line":"conform to that restriction."}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_0763a0b3","line":24,"range":{"start_line":24,"start_character":21,"end_line":24,"end_character":30},"updated":"2018-02-12 14:58:00.000000000","message":"why?","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":14070,"name":"Eric Fried","email":"openstack@fried.cc","username":"efried"},"change_message_id":"aa1a50b41f87b093aa395382e11cf203fa6ba88e","unresolved":false,"context_lines":[{"line_number":21,"context_line":"a list of resource classes whose resources must come from providers in the same"},{"line_number":22,"context_line":"aggregate. Virt drivers implementing `update_provider_tree`_ model each NUMA"},{"line_number":23,"context_line":"node as a branch off of the root provider, associating each provider in the"},{"line_number":24,"context_line":"branch with a single aggregate. Operators design flavors containing the list of"},{"line_number":25,"context_line":"the resource classes which must come from the same NUMA node. The scheduler"},{"line_number":26,"context_line":"passes that list to placement, which returns only allocation candidates which"},{"line_number":27,"context_line":"conform to that restriction."}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_d87cb573","line":24,"range":{"start_line":24,"start_character":21,"end_line":24,"end_character":30},"in_reply_to":"3fa0c359_0763a0b3","updated":"2018-02-12 16:50:10.000000000","message":"See above \u0026 below.","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"24cc50498ab310845652ca57e4f8a17286e6ac07","unresolved":false,"context_lines":[{"line_number":22,"context_line":"aggregate. Virt drivers implementing `update_provider_tree`_ model each NUMA"},{"line_number":23,"context_line":"node as a branch off of the root provider, associating each provider in the"},{"line_number":24,"context_line":"branch with a single aggregate. Operators design flavors containing the list of"},{"line_number":25,"context_line":"the resource classes which must come from the same NUMA node. The scheduler"},{"line_number":26,"context_line":"passes that list to placement, which returns only allocation candidates which"},{"line_number":27,"context_line":"conform to that restriction."},{"line_number":28,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_8776d06f","line":25,"range":{"start_line":25,"start_character":37,"end_line":25,"end_character":60},"updated":"2018-02-12 14:58:00.000000000","message":"in this paragraph, you\u0027re mixing aggregates with NUMA nodes with child providers/subtrees... it\u0027s not clear what you mean here.","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":14070,"name":"Eric Fried","email":"openstack@fried.cc","username":"efried"},"change_message_id":"aa1a50b41f87b093aa395382e11cf203fa6ba88e","unresolved":false,"context_lines":[{"line_number":22,"context_line":"aggregate. Virt drivers implementing `update_provider_tree`_ model each NUMA"},{"line_number":23,"context_line":"node as a branch off of the root provider, associating each provider in the"},{"line_number":24,"context_line":"branch with a single aggregate. Operators design flavors containing the list of"},{"line_number":25,"context_line":"the resource classes which must come from the same NUMA node. The scheduler"},{"line_number":26,"context_line":"passes that list to placement, which returns only allocation candidates which"},{"line_number":27,"context_line":"conform to that restriction."},{"line_number":28,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_b871a978","line":25,"range":{"start_line":25,"start_character":37,"end_line":25,"end_character":60},"in_reply_to":"3fa0c359_8776d06f","updated":"2018-02-12 16:50:10.000000000","message":"This paragraph is summarizing the solution.  Which is for virt to express a NUMA node by associating the providers in (usually) a subtree with an aggregate.  Then the admin/op only has to know which resource classes she wants to \"affinitize\".","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"24cc50498ab310845652ca57e4f8a17286e6ac07","unresolved":false,"context_lines":[{"line_number":60,"context_line":""},{"line_number":61,"context_line":"The problem being addressed is that, although the virt driver can model this,"},{"line_number":62,"context_line":"there is as yet no (good) way for the user to express a request to deploy an"},{"line_number":63,"context_line":"instance whose resources are all provided by the same NUMA node."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"Use Cases"},{"line_number":66,"context_line":"---------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_67fce4b4","line":63,"range":{"start_line":63,"start_character":41,"end_line":63,"end_character":64},"updated":"2018-02-12 14:58:00.000000000","message":"by the same child resource provider.","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"36221d8f5ca89deaba872a0f22181e24abad9e9a","unresolved":false,"context_lines":[{"line_number":60,"context_line":""},{"line_number":61,"context_line":"The problem being addressed is that, although the virt driver can model this,"},{"line_number":62,"context_line":"there is as yet no (good) way for the user to express a request to deploy an"},{"line_number":63,"context_line":"instance whose resources are all provided by the same NUMA node."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"Use Cases"},{"line_number":66,"context_line":"---------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_0b37264f","line":63,"range":{"start_line":63,"start_character":41,"end_line":63,"end_character":64},"in_reply_to":"3fa0c359_194f7f93","updated":"2018-02-12 18:49:27.000000000","message":"\u003e \u003e No, they *always* will be within a single subtree. When we were\n \u003e in\n \u003e \u003e Denver or maybe it was Atlanta, Sean was asked \"can NUMA\n \u003e resources\n \u003e \u003e potentially have multiple parents?\" and the answer we decided was\n \u003e \u003e \"perhaps but we\u0027re not going to support multi-parent scenarios\".\n \u003e \n \u003e Cool.  But note that this solution generalizes beyond NUMA.  If\n \u003e there\u0027s some other reason I want to group resources, and it happens\n \u003e to cross subtrees, it\u0027d have to be solved separately.\n\nBut we already have the concept of aggregates to group providers of resources. So I\u0027m not sure what this spec is proposing exactly.\n\n \u003e \u003e If what you\u0027re trying to model here is \"how do I determine the\n \u003e \u003e child providers that are \u0027affined to a NUMA node\u0027, then the way\n \u003e to\n \u003e \u003e do that is by limiting the request for resources to a subtree,\n \u003e not\n \u003e \u003e an aggregate.\n \u003e \n \u003e Fine, okay, but how do you *do* that?  How do you limit to a\n \u003e subtree?  How do you identify the subtree?\n\nYou don\u0027t need to... you need to identify the trees which have the capacity to meet resource and trait requirements together.\n\nWhen creating the list of allocation_request objects, you have the tree structure in memory and can limit the allocation_request objects in the returned list to only those which meet the original request conditions *and where the providers providing the resources are within the same subtree*.\n\n\u003e Are we going to force\n \u003e all virts to model the same way and then hardcode the meaning of\n \u003e certain structures?\n\nWe aren\u0027t going to force virt drivers to use trees, nor are we going to force virt drivers to represent NUMA affinity via child providers in trees. However, we ARE going to force all virt drivers to model resources using a single-parent model if they do decide to model their resource providers using trees.\n\n\u003e Are we going to enforce a naming convention\n \u003e for RPs, or insist on certain traits?  There has to be *some* way\n \u003e to say \"these providers - which okay, happen to comprise a subtree\n \u003e - represent a group from which we need to be able to \"affine\"\n \u003e resources\".\n\nThat\u0027s what the tree structure is all about, though. It\u0027s the tree structure that informs about the concept of what is \"next to\" or \"a child of\" something else. And that\u0027s what \"NUMA affinity\" is all about.\n\nAggregates are all about *grouping* of like providers and sharing of resources with providers in that group.\n\n\u003e Right now we simply don\u0027t have a way to say \"subtree\"\n \u003e explicitly.  And if we did, I\u0027m still not sold that that\u0027s the\n \u003e right thing to do without some additional indication by virt that\n \u003e \"this particular subtree is significant\".\n\nIt\u0027s not that *this particular subtree* is significant. The thing that is significant is that the tree structure *itself* is a piece of information that is important.\n\nBy making a NIC resource provider a child of a NUMA resource provider that itself is a child of a compute node provider, the virt driver is informing placement of the relationship between those resource providers.","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":14070,"name":"Eric Fried","email":"openstack@fried.cc","username":"efried"},"change_message_id":"9d0d6b5baafb1d92a50bb71f5aa9fc39c18d64da","unresolved":false,"context_lines":[{"line_number":60,"context_line":""},{"line_number":61,"context_line":"The problem being addressed is that, although the virt driver can model this,"},{"line_number":62,"context_line":"there is as yet no (good) way for the user to express a request to deploy an"},{"line_number":63,"context_line":"instance whose resources are all provided by the same NUMA node."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"Use Cases"},{"line_number":66,"context_line":"---------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_194f7f93","line":63,"range":{"start_line":63,"start_character":41,"end_line":63,"end_character":64},"in_reply_to":"3fa0c359_5e1acd16","updated":"2018-02-12 18:10:45.000000000","message":"\u003e No, they *always* will be within a single subtree. When we were in\n \u003e Denver or maybe it was Atlanta, Sean was asked \"can NUMA resources\n \u003e potentially have multiple parents?\" and the answer we decided was\n \u003e \"perhaps but we\u0027re not going to support multi-parent scenarios\".\n\nCool.  But note that this solution generalizes beyond NUMA.  If there\u0027s some other reason I want to group resources, and it happens to cross subtrees, it\u0027d have to be solved separately.\n\n \u003e If what you\u0027re trying to model here is \"how do I determine the\n \u003e child providers that are \u0027affined to a NUMA node\u0027, then the way to\n \u003e do that is by limiting the request for resources to a subtree, not\n \u003e an aggregate.\n\nFine, okay, but how do you *do* that?  How do you limit to a subtree?  How do you identify the subtree?  Are we going to force all virts to model the same way and then hardcode the meaning of certain structures?  Are we going to enforce a naming convention for RPs, or insist on certain traits?  There has to be *some* way to say \"these providers - which okay, happen to comprise a subtree - represent a group from which we need to be able to \"affine\" resources\".  Right now we simply don\u0027t have a way to say \"subtree\" explicitly.  And if we did, I\u0027m still not sold that that\u0027s the right thing to do without some additional indication by virt that \"this particular subtree is significant\".","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":14070,"name":"Eric Fried","email":"openstack@fried.cc","username":"efried"},"change_message_id":"aa1a50b41f87b093aa395382e11cf203fa6ba88e","unresolved":false,"context_lines":[{"line_number":60,"context_line":""},{"line_number":61,"context_line":"The problem being addressed is that, although the virt driver can model this,"},{"line_number":62,"context_line":"there is as yet no (good) way for the user to express a request to deploy an"},{"line_number":63,"context_line":"instance whose resources are all provided by the same NUMA node."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"Use Cases"},{"line_number":66,"context_line":"---------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_78c32158","line":63,"range":{"start_line":63,"start_character":41,"end_line":63,"end_character":64},"in_reply_to":"3fa0c359_67fce4b4","updated":"2018-02-12 16:50:10.000000000","message":"No, cause we have that (granular request groups).  \"the same NUMA node\" is *sometimes* a single child resource provider.  But as described above, sometimes you can\u0027t provide all your \"affinitized\" resources out of a single provider.  In that case \"the same NUMA node\" may span multiple providers, which are usually (but not necessarily) in a single subtree, which is why we need a mechanism to express that they\u0027re associated with each other.  This proposal uses aggregates for that.","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"8c21e217a8d74b671135e1478742b8d4d0c43249","unresolved":false,"context_lines":[{"line_number":60,"context_line":""},{"line_number":61,"context_line":"The problem being addressed is that, although the virt driver can model this,"},{"line_number":62,"context_line":"there is as yet no (good) way for the user to express a request to deploy an"},{"line_number":63,"context_line":"instance whose resources are all provided by the same NUMA node."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"Use Cases"},{"line_number":66,"context_line":"---------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_5e1acd16","line":63,"range":{"start_line":63,"start_character":41,"end_line":63,"end_character":64},"in_reply_to":"3fa0c359_78c32158","updated":"2018-02-12 17:16:11.000000000","message":"\u003e In that case \"the same NUMA\n \u003e node\" may span multiple providers, which are usually (but not\n \u003e necessarily) in a single subtree\n\nNo, they *always* will be within a single subtree. When we were in Denver or maybe it was Atlanta, Sean was asked \"can NUMA resources potentially have multiple parents?\" and the answer we decided was \"perhaps but we\u0027re not going to support multi-parent scenarios\".\n\nSo what I am saying is that it is *always* possible to model NUMA as the providers under a single parent -- which is called a subtree.\n\nIf what you\u0027re trying to model here is \"how do I determine the child providers that are \u0027affined to a NUMA node\u0027, then the way to do that is by limiting the request for resources to a subtree, not an aggregate.","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"24cc50498ab310845652ca57e4f8a17286e6ac07","unresolved":false,"context_lines":[{"line_number":64,"context_line":""},{"line_number":65,"context_line":"Use Cases"},{"line_number":66,"context_line":"---------"},{"line_number":67,"context_line":"As an end user, I wish to deploy an instance whose resources have affinity to"},{"line_number":68,"context_line":"the same NUMA node."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"Proposed change"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_471a0817","line":67,"range":{"start_line":67,"start_character":51,"end_line":67,"end_character":74},"updated":"2018-02-12 14:58:00.000000000","message":"I\u0027d prefer it if we didn\u0027t use the terms \"resources\" or \"affinity\" here. What you are saying here is \"deploy an instance whose vCPUs or huge memory pages are pinned to one or more specific NUMA nodes.\"","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"36221d8f5ca89deaba872a0f22181e24abad9e9a","unresolved":false,"context_lines":[{"line_number":64,"context_line":""},{"line_number":65,"context_line":"Use Cases"},{"line_number":66,"context_line":"---------"},{"line_number":67,"context_line":"As an end user, I wish to deploy an instance whose resources have affinity to"},{"line_number":68,"context_line":"the same NUMA node."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"Proposed change"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_eb9d1a0a","line":67,"range":{"start_line":67,"start_character":51,"end_line":67,"end_character":74},"in_reply_to":"3fa0c359_0b596633","updated":"2018-02-12 18:49:27.000000000","message":"Yes, it already \"handles\" this:\n\nhttps://github.com/openstack/nova/blob/master/nova/scheduler/filters/numa_topology_filter.py\n\nwhere the \"meat\" of the algorithm is here:\n\nhttps://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L1431-L1512\n\nThe primary problem with all the NUMA topology stuff is the coupling of CPU pinning, huge page pinning with the representation of the host and instance NUMA topologies.\n\nIn addition, the fact that the NUMA topology (the layout of VM sockets/cores/threads over the top of a physical CPU package/topology) is a complex \"resource\" that isn\u0027t integral in nature.\n\nThe NUMA topology filter and corresponding nova.virt.hardware.fit_instance_to_host() function is basically a mini-scheduler masquerading as \"just another filter\" inside our own filter scheduler.\n\nMy preference is to keep it there, not port it to placement, and focus instead on having the virt drivers properly constructing the ProviderTree so that the structure and relationship of resource providers the virt drivers know about is as accurate as possible.","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"8c21e217a8d74b671135e1478742b8d4d0c43249","unresolved":false,"context_lines":[{"line_number":64,"context_line":""},{"line_number":65,"context_line":"Use Cases"},{"line_number":66,"context_line":"---------"},{"line_number":67,"context_line":"As an end user, I wish to deploy an instance whose resources have affinity to"},{"line_number":68,"context_line":"the same NUMA node."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"Proposed change"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_9ece1561","line":67,"range":{"start_line":67,"start_character":51,"end_line":67,"end_character":74},"in_reply_to":"3fa0c359_18ca1d1a","updated":"2018-02-12 17:16:11.000000000","message":"And what I\u0027m saying is that in my experience, the people that are saying NUMA placement is important to them are hell-bent on ensuring that specific CPUs are pinned for their application VM\u0027s I/O threads. And therefore, a solution in placement for \"NUMA use cases\" that *doesn\u0027t* include CPU or huge page pinning isn\u0027t really solving any of the \"NUMA use cases\".\n\nNow, if you want to say that \"make sure the vNIC that my VM gets is associated with the same NUMA node that my VM ends up being associated with\", that\u0027s a completely different use case and problem. And, frankly, I do not believe the placement API needs to solve that problem. Rather, I believe that the placement API\u0027s GET /allocation_candidates can return the tree structure information and the Nova scheduler\u0027s existing NUMA topology filter can choose to use that information to determine which of the allocation requests it will try when claiming resources against multiple providers.","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":14070,"name":"Eric Fried","email":"openstack@fried.cc","username":"efried"},"change_message_id":"aa1a50b41f87b093aa395382e11cf203fa6ba88e","unresolved":false,"context_lines":[{"line_number":64,"context_line":""},{"line_number":65,"context_line":"Use Cases"},{"line_number":66,"context_line":"---------"},{"line_number":67,"context_line":"As an end user, I wish to deploy an instance whose resources have affinity to"},{"line_number":68,"context_line":"the same NUMA node."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"Proposed change"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_18ca1d1a","line":67,"range":{"start_line":67,"start_character":51,"end_line":67,"end_character":74},"in_reply_to":"3fa0c359_471a0817","updated":"2018-02-12 16:50:10.000000000","message":"That\u0027s not actually the use case I\u0027m addressing, which may be the main disconnect.\n\nIf we want to pin [whatever] to *specific* NUMA nodes, we can do that with what we have today (well, once update_provider_tree and NRP are finished) with traits.  I talk a little about that in Limitations/Anti-affinity [L260] and Alternatives/Traits [L348].\n\nIn contrast, this design is addressing the desire to make sure [whatever] and [another thing] come from the *same* NUMA node; but placement decides (or rather, presents alternatives for) which *specific* node that will be.  Which is much more cloud-y, I\u0027m led to understand.","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":14070,"name":"Eric Fried","email":"openstack@fried.cc","username":"efried"},"change_message_id":"9d0d6b5baafb1d92a50bb71f5aa9fc39c18d64da","unresolved":false,"context_lines":[{"line_number":64,"context_line":""},{"line_number":65,"context_line":"Use Cases"},{"line_number":66,"context_line":"---------"},{"line_number":67,"context_line":"As an end user, I wish to deploy an instance whose resources have affinity to"},{"line_number":68,"context_line":"the same NUMA node."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"Proposed change"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_0b596633","line":67,"range":{"start_line":67,"start_character":51,"end_line":67,"end_character":74},"in_reply_to":"3fa0c359_9ece1561","updated":"2018-02-12 18:10:45.000000000","message":"\u003e And what I\u0027m saying is that in my experience, the people that are\n \u003e saying NUMA placement is important to them are hell-bent on\n \u003e ensuring that specific CPUs are pinned for their application VM\u0027s\n \u003e I/O threads.\n\nCool, if this is all they care about, they can use traits and granular without any further placement support (beyond NRP/u_p_t).\n\n \u003e Now, if you want to say that \"make sure the vNIC that my VM gets is\n \u003e associated with the same NUMA node that my VM ends up being\n \u003e associated with\", that\u0027s a completely different use case and\n \u003e problem.\n\nYes, this.\n\n \u003e And, frankly, I do not believe the placement API needs to\n \u003e solve that problem. Rather, I believe that the placement API\u0027s GET\n \u003e /allocation_candidates can return the tree structure information\n \u003e and the Nova scheduler\u0027s existing NUMA topology filter can choose\n \u003e to use that information to determine which of the allocation\n \u003e requests it will try when claiming resources against multiple\n \u003e providers.\n\nI\u0027m not familiar with that filter.  Does it already handle this?  How does *it* identify subtrees, and know that a subtree corresponds to a NUMA node?","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":25625,"name":"Tetsuro Nakamura","email":"tetsuro.nakamura.bc@hco.ntt.co.jp","username":"tetsuro0907"},"change_message_id":"1c674ef71f19173c5a7b2dd6aa79b3c5e250cce3","unresolved":false,"context_lines":[{"line_number":64,"context_line":""},{"line_number":65,"context_line":"Use Cases"},{"line_number":66,"context_line":"---------"},{"line_number":67,"context_line":"As an end user, I wish to deploy an instance whose resources have affinity to"},{"line_number":68,"context_line":"the same NUMA node."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"Proposed change"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_c1abcb72","line":67,"range":{"start_line":67,"start_character":51,"end_line":67,"end_character":74},"in_reply_to":"3fa0c359_eb9d1a0a","updated":"2018-02-19 10:46:28.000000000","message":"\u003e My preference is to keep it there, not port it to placement.\n\nI agree. \n\nFirst, I think, the fact that \"the Nova today reports all the pCPUs to the Placement regardless of how we use it, that is, dedicated or shared.\" is making the things more complex.\n\nI\u0027m proposing in [1] to not report or claim to the Placement at all for dedicated vCPUs/pCPUs and totally leave it to Nova\u0027s internal mini-scheduler that Jay mentioned. Because we don\u0027t have the scheme in Placement to represent DEDICATED_CPU resource and I don\u0027t think it a good idea to model such a specific way of using resource as a standard resource class in the Placement.\n\nOne possible idea to solve the difficulty for modeling this DEDICATED_CPU is to decide and define in the Nova that \"the VCPUs of compute nodes are for shared(floating), and the VCPUs of NUMA are for dedicated(pinning).\" However, the MEMORY is even more complicated. We should manage each page size just as described in the test casein [2]. How do we manage this kind of resource class in the Placement?\n\nThis is why I think we can\u0027t leave this Nova\u0027s internal mini-scheduler to the Placement.\n\nI wrote in the spec[1] and above that modeling dedicated(pinning) CPUs in the Placement may offload some loads from the mini-scheduler to the Placement, but this is just in a sense that the allocation candidates to pass the mini-scheduler can be reduced, that is, this doesn\u0027t mean we can delete the code in the mini-scheduler. \n\nSo, my conclusion is \"I\u0027m not sure the cost warrants the benefit.\"\n\n[1] https://review.openstack.org/#/c/543805/\n\n[2] https://github.com/openstack/nova/blob/master/nova/tests/unit/virt/libvirt/test_driver.py#L2275-L2286","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"24cc50498ab310845652ca57e4f8a17286e6ac07","unresolved":false,"context_lines":[{"line_number":78,"context_line":"candidates`_ API (``GET /allocation_candidates``).  The value is a"},{"line_number":79,"context_line":"comma-separated list of string resource class names (see `Examples`_).  When it"},{"line_number":80,"context_line":"is specified, the API must return allocation candidates such that the listed"},{"line_number":81,"context_line":"resources come from providers in the same aggregate."},{"line_number":82,"context_line":""},{"line_number":83,"context_line":".. For purposes of this feature, resources provided by the same *resource"},{"line_number":84,"context_line":"   provider* are considered as coming from the same *aggregate*, even if that"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_67eaa4d8","line":81,"range":{"start_line":81,"start_character":42,"end_line":81,"end_character":51},"updated":"2018-02-12 14:58:00.000000000","message":"you haven\u0027t discussed aggregates at all in the Problem Description section above...","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":14070,"name":"Eric Fried","email":"openstack@fried.cc","username":"efried"},"change_message_id":"aa1a50b41f87b093aa395382e11cf203fa6ba88e","unresolved":false,"context_lines":[{"line_number":78,"context_line":"candidates`_ API (``GET /allocation_candidates``).  The value is a"},{"line_number":79,"context_line":"comma-separated list of string resource class names (see `Examples`_).  When it"},{"line_number":80,"context_line":"is specified, the API must return allocation candidates such that the listed"},{"line_number":81,"context_line":"resources come from providers in the same aggregate."},{"line_number":82,"context_line":""},{"line_number":83,"context_line":".. For purposes of this feature, resources provided by the same *resource"},{"line_number":84,"context_line":"   provider* are considered as coming from the same *aggregate*, even if that"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_38cd9930","line":81,"range":{"start_line":81,"start_character":42,"end_line":81,"end_character":51},"in_reply_to":"3fa0c359_67eaa4d8","updated":"2018-02-12 16:50:10.000000000","message":"Because aggregates aren\u0027t part of the problem.  They\u0027re part of the solution.","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"24cc50498ab310845652ca57e4f8a17286e6ac07","unresolved":false,"context_lines":[{"line_number":80,"context_line":"is specified, the API must return allocation candidates such that the listed"},{"line_number":81,"context_line":"resources come from providers in the same aggregate."},{"line_number":82,"context_line":""},{"line_number":83,"context_line":".. For purposes of this feature, resources provided by the same *resource"},{"line_number":84,"context_line":"   provider* are considered as coming from the same *aggregate*, even if that"},{"line_number":85,"context_line":"   resource provider has no aggregate associations."},{"line_number":86,"context_line":""},{"line_number":87,"context_line":"``group_resources`` only applies to the un-numbered request group, since"},{"line_number":88,"context_line":"a `numbered request group`_ is already restricted to a single resource"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_67c38458","line":85,"range":{"start_line":83,"start_character":3,"end_line":85,"end_character":51},"updated":"2018-02-12 14:58:00.000000000","message":"hmm, say what? this is very confusing indeed. Why not just say \"subtree of providers\" instead of re-using the known term \"aggregate\" to describe a subtree?","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":14070,"name":"Eric Fried","email":"openstack@fried.cc","username":"efried"},"change_message_id":"aa1a50b41f87b093aa395382e11cf203fa6ba88e","unresolved":false,"context_lines":[{"line_number":80,"context_line":"is specified, the API must return allocation candidates such that the listed"},{"line_number":81,"context_line":"resources come from providers in the same aggregate."},{"line_number":82,"context_line":""},{"line_number":83,"context_line":".. For purposes of this feature, resources provided by the same *resource"},{"line_number":84,"context_line":"   provider* are considered as coming from the same *aggregate*, even if that"},{"line_number":85,"context_line":"   resource provider has no aggregate associations."},{"line_number":86,"context_line":""},{"line_number":87,"context_line":"``group_resources`` only applies to the un-numbered request group, since"},{"line_number":88,"context_line":"a `numbered request group`_ is already restricted to a single resource"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_533406de","line":85,"range":{"start_line":83,"start_character":3,"end_line":85,"end_character":51},"in_reply_to":"3fa0c359_67c38458","updated":"2018-02-12 16:50:10.000000000","message":"\u003e hmm, say what? this is very confusing indeed.\n\nYeah, I commented it out (this paragraph doesn\u0027t show up in the rendered version) because I knew it would be confusing.\n\nThis is the reductive case where the model *happens* to be able to provide all of a NUMA node\u0027s resources out of a single provider.  In that model, we would still like to be able to support this syntax.  The reason I commented this out (and the sentence on L107-8) is because I think it\u0027s a better idea for virt to do the agg association anyway, even though it will be an \"aggregate of one\".\n\nNow, it\u0027s true that in this reductive case, the request could *also* be expressed using a numbered resource group.  But the supporting (and even encouraging) this syntax anyway makes the user experience more consistent.  And most importantly, it means the admin doesn\u0027t need to know what the model looks like.  Which IMO is a big plus.\n\n \u003e Why not just say\n \u003e \"subtree of providers\" instead of re-using the known term\n \u003e \"aggregate\" to describe a subtree?\n\nOh, let\u0027s make sure we\u0027re clear on this: I\u0027m not overloading the term \"aggregate\".  I\u0027m talking about actual placement aggregate UUIDs that you create via PUT /resource_providers/{u}/aggregates.  You will *usually* be creating them on a \"subtree of providers\" (though again, that doesn\u0027t have to be the case).  And the reason is because today placement has no concrete notion of \"subtree\" that can be applied to allocation candidate requests.","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"24cc50498ab310845652ca57e4f8a17286e6ac07","unresolved":false,"context_lines":[{"line_number":101,"context_line":"A virt driver wishing to support this feature (whether for NUMA or something"},{"line_number":102,"context_line":"else) should implement `update_provider_tree`_ and model its compute node such"},{"line_number":103,"context_line":"that more than one resource provider has inventories in the same resource"},{"line_number":104,"context_line":"class(es).  Then associate an aggregate UUID with each provider for which"},{"line_number":105,"context_line":"resource grouping is to be supported.  See `Examples`_."},{"line_number":106,"context_line":""},{"line_number":107,"context_line":".. If inventories in resource classes which need to be \"groupable\" are always"},{"line_number":108,"context_line":"   together (same resource provider), no aggregate associations are necessary."}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_c7829800","line":105,"range":{"start_line":104,"start_character":12,"end_line":105,"end_character":36},"updated":"2018-02-12 14:58:00.000000000","message":"but why?","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":14070,"name":"Eric Fried","email":"openstack@fried.cc","username":"efried"},"change_message_id":"aa1a50b41f87b093aa395382e11cf203fa6ba88e","unresolved":false,"context_lines":[{"line_number":101,"context_line":"A virt driver wishing to support this feature (whether for NUMA or something"},{"line_number":102,"context_line":"else) should implement `update_provider_tree`_ and model its compute node such"},{"line_number":103,"context_line":"that more than one resource provider has inventories in the same resource"},{"line_number":104,"context_line":"class(es).  Then associate an aggregate UUID with each provider for which"},{"line_number":105,"context_line":"resource grouping is to be supported.  See `Examples`_."},{"line_number":106,"context_line":""},{"line_number":107,"context_line":".. If inventories in resource classes which need to be \"groupable\" are always"},{"line_number":108,"context_line":"   together (same resource provider), no aggregate associations are necessary."}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_1350be0a","line":105,"range":{"start_line":104,"start_character":12,"end_line":105,"end_character":36},"in_reply_to":"3fa0c359_c7829800","updated":"2018-02-12 16:50:10.000000000","message":"See above.","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"24cc50498ab310845652ca57e4f8a17286e6ac07","unresolved":false,"context_lines":[{"line_number":144,"context_line":"  +-----------------+ +-----------------+ +-----------------+ +-----------------+"},{"line_number":145,"context_line":""},{"line_number":146,"context_line":"**Example 1:** \"Give me 2 VCPU and 1 GB of memory from the same NUMA node; and"},{"line_number":147,"context_line":"one VF from NET2 (which need not be from the same NUMA node).\""},{"line_number":148,"context_line":""},{"line_number":149,"context_line":"**Flavor:**"},{"line_number":150,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_47c44840","line":147,"updated":"2018-02-12 14:58:00.000000000","message":"But why use aggregates at all? The structure of the tree already provides the information you need for grouping the NICs by NUMA node (their parents are the NUMA node resource providers)","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":14070,"name":"Eric Fried","email":"openstack@fried.cc","username":"efried"},"change_message_id":"aa1a50b41f87b093aa395382e11cf203fa6ba88e","unresolved":false,"context_lines":[{"line_number":144,"context_line":"  +-----------------+ +-----------------+ +-----------------+ +-----------------+"},{"line_number":145,"context_line":""},{"line_number":146,"context_line":"**Example 1:** \"Give me 2 VCPU and 1 GB of memory from the same NUMA node; and"},{"line_number":147,"context_line":"one VF from NET2 (which need not be from the same NUMA node).\""},{"line_number":148,"context_line":""},{"line_number":149,"context_line":"**Flavor:**"},{"line_number":150,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_f3f432f3","line":147,"in_reply_to":"3fa0c359_47c44840","updated":"2018-02-12 16:50:10.000000000","message":"\u003e But why use aggregates at all? The structure of the tree already\n \u003e provides the information you need for grouping the NICs by NUMA\n \u003e node (their parents are the NUMA node resource providers)\n\nRight, but how does placement know that?  I would be opposed to any solution that e.g. keys off a substring in a RP\u0027s name (or trait) to identify a subtree as being a NUMA node.  And also to any solution that restricts the shape of the tree such that we can hardcode NUMA-ness by tree \"position\".  And using traits wholesale limits us to having to pin to a *specific* NUMA node as discussed above.\n\nAggregates aren\u0027t the only way to signify this grouping.  But they\u0027re (IMO) simple and well suited to the task.","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"24cc50498ab310845652ca57e4f8a17286e6ac07","unresolved":false,"context_lines":[{"line_number":168,"context_line":""},{"line_number":169,"context_line":".. code::"},{"line_number":170,"context_line":""},{"line_number":171,"context_line":"  ["},{"line_number":172,"context_line":"      { \"allocations\": {"},{"line_number":173,"context_line":"          NUMA1_rp: { \"resources\": {"},{"line_number":174,"context_line":"              \"VCPU\": 2,"},{"line_number":175,"context_line":"              \"MEMORY_MB\": 1024,"},{"line_number":176,"context_line":"          }}"},{"line_number":177,"context_line":"          NIC_PF2_rp: { \"resources\": {"},{"line_number":178,"context_line":"              \"SRIOV_NET_VF\": 1,"},{"line_number":179,"context_line":"          }}"},{"line_number":180,"context_line":"      }}"},{"line_number":181,"context_line":"      { \"allocations\": {"},{"line_number":182,"context_line":"          NUMA1_rp: { \"resources\": {"},{"line_number":183,"context_line":"              \"VCPU\": 2,"},{"line_number":184,"context_line":"              \"MEMORY_MB\": 1024,"},{"line_number":185,"context_line":"          }}"},{"line_number":186,"context_line":"          NIC_PF4_rp: { \"resources\": {"},{"line_number":187,"context_line":"              \"SRIOV_NET_VF\": 1,"},{"line_number":188,"context_line":"          }}"},{"line_number":189,"context_line":"      }}"},{"line_number":190,"context_line":"      { \"allocations\": {"},{"line_number":191,"context_line":"          NUMA2_rp: { \"resources\": {"},{"line_number":192,"context_line":"              \"VCPU\": 2,"},{"line_number":193,"context_line":"              \"MEMORY_MB\": 1024,"},{"line_number":194,"context_line":"          }}"},{"line_number":195,"context_line":"          NIC_PF2_rp: { \"resources\": {"},{"line_number":196,"context_line":"              \"SRIOV_NET_VF\": 1,"},{"line_number":197,"context_line":"          }}"},{"line_number":198,"context_line":"      }}"},{"line_number":199,"context_line":"      { \"allocations\": {"},{"line_number":200,"context_line":"          NUMA2_rp: { \"resources\": {"},{"line_number":201,"context_line":"              \"VCPU\": 2,"},{"line_number":202,"context_line":"              \"MEMORY_MB\": 1024,"},{"line_number":203,"context_line":"          }}"},{"line_number":204,"context_line":"          NIC_PF4_rp: { \"resources\": {"},{"line_number":205,"context_line":"              \"SRIOV_NET_VF\": 1,"},{"line_number":206,"context_line":"          }}"},{"line_number":207,"context_line":"      }}"},{"line_number":208,"context_line":"  ]"},{"line_number":209,"context_line":""},{"line_number":210,"context_line":"**Example 2:** \"Give me 2 VCPU, 1GB of memory, and one VF from NET2, all from"},{"line_number":211,"context_line":"the same NUMA node\""}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_67d4e4e9","line":208,"range":{"start_line":171,"start_character":0,"end_line":208,"end_character":3},"updated":"2018-02-12 14:58:00.000000000","message":"This doesn\u0027t show anything different from what is currently returned by GET /allocation_candidates. Is that intended?","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"36221d8f5ca89deaba872a0f22181e24abad9e9a","unresolved":false,"context_lines":[{"line_number":168,"context_line":""},{"line_number":169,"context_line":".. code::"},{"line_number":170,"context_line":""},{"line_number":171,"context_line":"  ["},{"line_number":172,"context_line":"      { \"allocations\": {"},{"line_number":173,"context_line":"          NUMA1_rp: { \"resources\": {"},{"line_number":174,"context_line":"              \"VCPU\": 2,"},{"line_number":175,"context_line":"              \"MEMORY_MB\": 1024,"},{"line_number":176,"context_line":"          }}"},{"line_number":177,"context_line":"          NIC_PF2_rp: { \"resources\": {"},{"line_number":178,"context_line":"              \"SRIOV_NET_VF\": 1,"},{"line_number":179,"context_line":"          }}"},{"line_number":180,"context_line":"      }}"},{"line_number":181,"context_line":"      { \"allocations\": {"},{"line_number":182,"context_line":"          NUMA1_rp: { \"resources\": {"},{"line_number":183,"context_line":"              \"VCPU\": 2,"},{"line_number":184,"context_line":"              \"MEMORY_MB\": 1024,"},{"line_number":185,"context_line":"          }}"},{"line_number":186,"context_line":"          NIC_PF4_rp: { \"resources\": {"},{"line_number":187,"context_line":"              \"SRIOV_NET_VF\": 1,"},{"line_number":188,"context_line":"          }}"},{"line_number":189,"context_line":"      }}"},{"line_number":190,"context_line":"      { \"allocations\": {"},{"line_number":191,"context_line":"          NUMA2_rp: { \"resources\": {"},{"line_number":192,"context_line":"              \"VCPU\": 2,"},{"line_number":193,"context_line":"              \"MEMORY_MB\": 1024,"},{"line_number":194,"context_line":"          }}"},{"line_number":195,"context_line":"          NIC_PF2_rp: { \"resources\": {"},{"line_number":196,"context_line":"              \"SRIOV_NET_VF\": 1,"},{"line_number":197,"context_line":"          }}"},{"line_number":198,"context_line":"      }}"},{"line_number":199,"context_line":"      { \"allocations\": {"},{"line_number":200,"context_line":"          NUMA2_rp: { \"resources\": {"},{"line_number":201,"context_line":"              \"VCPU\": 2,"},{"line_number":202,"context_line":"              \"MEMORY_MB\": 1024,"},{"line_number":203,"context_line":"          }}"},{"line_number":204,"context_line":"          NIC_PF4_rp: { \"resources\": {"},{"line_number":205,"context_line":"              \"SRIOV_NET_VF\": 1,"},{"line_number":206,"context_line":"          }}"},{"line_number":207,"context_line":"      }}"},{"line_number":208,"context_line":"  ]"},{"line_number":209,"context_line":""},{"line_number":210,"context_line":"**Example 2:** \"Give me 2 VCPU, 1GB of memory, and one VF from NET2, all from"},{"line_number":211,"context_line":"the same NUMA node\""}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_2bc1c2e5","line":208,"range":{"start_line":171,"start_character":0,"end_line":208,"end_character":3},"in_reply_to":"3fa0c359_137c3eea","updated":"2018-02-12 18:49:27.000000000","message":"No, I mean the above isn\u0027t any change to the existing response to GET /allocation_candidates.","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":14070,"name":"Eric Fried","email":"openstack@fried.cc","username":"efried"},"change_message_id":"aa1a50b41f87b093aa395382e11cf203fa6ba88e","unresolved":false,"context_lines":[{"line_number":168,"context_line":""},{"line_number":169,"context_line":".. code::"},{"line_number":170,"context_line":""},{"line_number":171,"context_line":"  ["},{"line_number":172,"context_line":"      { \"allocations\": {"},{"line_number":173,"context_line":"          NUMA1_rp: { \"resources\": {"},{"line_number":174,"context_line":"              \"VCPU\": 2,"},{"line_number":175,"context_line":"              \"MEMORY_MB\": 1024,"},{"line_number":176,"context_line":"          }}"},{"line_number":177,"context_line":"          NIC_PF2_rp: { \"resources\": {"},{"line_number":178,"context_line":"              \"SRIOV_NET_VF\": 1,"},{"line_number":179,"context_line":"          }}"},{"line_number":180,"context_line":"      }}"},{"line_number":181,"context_line":"      { \"allocations\": {"},{"line_number":182,"context_line":"          NUMA1_rp: { \"resources\": {"},{"line_number":183,"context_line":"              \"VCPU\": 2,"},{"line_number":184,"context_line":"              \"MEMORY_MB\": 1024,"},{"line_number":185,"context_line":"          }}"},{"line_number":186,"context_line":"          NIC_PF4_rp: { \"resources\": {"},{"line_number":187,"context_line":"              \"SRIOV_NET_VF\": 1,"},{"line_number":188,"context_line":"          }}"},{"line_number":189,"context_line":"      }}"},{"line_number":190,"context_line":"      { \"allocations\": {"},{"line_number":191,"context_line":"          NUMA2_rp: { \"resources\": {"},{"line_number":192,"context_line":"              \"VCPU\": 2,"},{"line_number":193,"context_line":"              \"MEMORY_MB\": 1024,"},{"line_number":194,"context_line":"          }}"},{"line_number":195,"context_line":"          NIC_PF2_rp: { \"resources\": {"},{"line_number":196,"context_line":"              \"SRIOV_NET_VF\": 1,"},{"line_number":197,"context_line":"          }}"},{"line_number":198,"context_line":"      }}"},{"line_number":199,"context_line":"      { \"allocations\": {"},{"line_number":200,"context_line":"          NUMA2_rp: { \"resources\": {"},{"line_number":201,"context_line":"              \"VCPU\": 2,"},{"line_number":202,"context_line":"              \"MEMORY_MB\": 1024,"},{"line_number":203,"context_line":"          }}"},{"line_number":204,"context_line":"          NIC_PF4_rp: { \"resources\": {"},{"line_number":205,"context_line":"              \"SRIOV_NET_VF\": 1,"},{"line_number":206,"context_line":"          }}"},{"line_number":207,"context_line":"      }}"},{"line_number":208,"context_line":"  ]"},{"line_number":209,"context_line":""},{"line_number":210,"context_line":"**Example 2:** \"Give me 2 VCPU, 1GB of memory, and one VF from NET2, all from"},{"line_number":211,"context_line":"the same NUMA node\""}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_137c3eea","line":208,"range":{"start_line":171,"start_character":0,"end_line":208,"end_character":3},"in_reply_to":"3fa0c359_67d4e4e9","updated":"2018-02-12 16:50:10.000000000","message":"When you say \"currently\", you mean once the NRP series is finished, right?\n\nSince we decided we wouldn\u0027t split inventory across RPs (e.g. one VCPU from NUMA1_rp and one from NUMA2_rp), it\u0027s true that this case will happen to be the same as if you left the group_resources part of the request off.\n\nAnd yes, that\u0027s intentional.  As I mentioned above, we would like the reductive case to be able to be expressed in a consistent fashion so the admin doesn\u0027t have to be aware of the tree structure.  This allows virt way more flexibility in their modeling.  For instance, what if virt wanted to model the NUMA node as a resource-less parent under which VCPU and MEMORY_MB are provided by *separate* providers?  Then the admin\u0027s flavor still works and affinitizes VCPU\u0026MEMORY_MB properly.  But she doesn\u0027t have to know that the model is weird/different.","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"24cc50498ab310845652ca57e4f8a17286e6ac07","unresolved":false,"context_lines":[{"line_number":287,"context_line":"Distance"},{"line_number":288,"context_line":"~~~~~~~~"},{"line_number":289,"context_line":"I can\u0027t say, \"Give me an instance with its VCPU fewer than two \u0027steps\u0027 away"},{"line_number":290,"context_line":"from its memory,\" whatever that means.  See `Tree Distance`_."},{"line_number":291,"context_line":""},{"line_number":292,"context_line":"Alternatives"},{"line_number":293,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_87dd70e1","line":290,"updated":"2018-02-12 14:58:00.000000000","message":"In addition to the above, this model doesn\u0027t fulfill the primary use case associated with NUMA topology in Nova: CPU and huge page pinning.\n\nThe fundamental incongruence between Placement modeling of resources and the needs of CPU pinning use cases is that Placement doesn\u0027t consider each individual pCPU core/thread as a separate resource. Instead, Placement considers all vCPUs to be equal.\n\nThis is why I don\u0027t believe it is useful to try to have Placement model pinning use cases, including NUMA topology fitting.","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":14070,"name":"Eric Fried","email":"openstack@fried.cc","username":"efried"},"change_message_id":"aa1a50b41f87b093aa395382e11cf203fa6ba88e","unresolved":false,"context_lines":[{"line_number":287,"context_line":"Distance"},{"line_number":288,"context_line":"~~~~~~~~"},{"line_number":289,"context_line":"I can\u0027t say, \"Give me an instance with its VCPU fewer than two \u0027steps\u0027 away"},{"line_number":290,"context_line":"from its memory,\" whatever that means.  See `Tree Distance`_."},{"line_number":291,"context_line":""},{"line_number":292,"context_line":"Alternatives"},{"line_number":293,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_734902b5","line":290,"in_reply_to":"3fa0c359_87dd70e1","updated":"2018-02-12 16:50:10.000000000","message":"I may need further explanation of this comment, but I think my response above [L67] applies.  Namely, we\u0027re not pinning to a *specific* NUMA node; we\u0027re requesting that resources be provided from the *same* NUMA node, whichever one that happens to be.  The separation of pCPUs happens via modeling.\n\nImportantly, once virt receives the spawn request, it knows which NUMA node to pin to based on the RP from which the resource was allocated.","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"0c04951bd8df57e59b13ccfc57be839039c69686","unresolved":false,"context_lines":[{"line_number":384,"context_line":""},{"line_number":385,"context_line":"(We could also implement this in ``GET /resource_providers``, since it supports"},{"line_number":386,"context_line":"the ``resources`` queryparam.  Is that necessary/desirable?  We didn\u0027t"},{"line_number":387,"context_line":"implement `numbered request group`_ there.)"},{"line_number":388,"context_line":""},{"line_number":389,"context_line":"Security impact"},{"line_number":390,"context_line":"---------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3fa0c359_7a368005","line":387,"range":{"start_line":387,"start_character":0,"end_line":387,"end_character":42},"updated":"2018-02-14 13:36:18.000000000","message":"tangent: someday it would be nice to have parity (where it makes any sense) between the two things. As sometimes what you want is just a list of providers. I\u0027m especially thinking of non-nova scenarios that haven\u0027t been explored yet where allocation_requests may not be a thing.","commit_id":"eac0955b0486e11327e7e9eb2575edc53ba49bf6"}]}