)]}'
{"/COMMIT_MSG":[{"author":{"_account_id":8768,"name":"Chris Friesen","email":"chris.friesen@windriver.com","username":"cbf123"},"change_message_id":"c939b62e6b500cfb24847af6b211238991a3988a","unresolved":false,"context_lines":[{"line_number":11,"context_line":"This patch fixes how instances are placed over NUMA nodes. The current"},{"line_number":12,"context_line":"behavior has undesired effects, as packing all instances to node 0: e.g:"},{"line_number":13,"context_line":"when hw:numa_nodes\u003d1, the result was to pack all instances on node 0,"},{"line_number":14,"context_line":"leaving the other nodes free (unused):"},{"line_number":15,"context_line":"instance1 --\u003e node0, instance2 --\u003e node0, instance3 --\u003e node0"},{"line_number":16,"context_line":""},{"line_number":17,"context_line":"The new criteria optimizes NUMA usage by spreading instances across all"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":2,"id":"1aa78d24_aa152c4a","line":14,"updated":"2016-07-14 16:22:13.000000000","message":"One potentially-nice thing about the current behaviour is that the other nodes are free to drop down to a low-power mode, which would save power in the datacenter.\n\nIf you spread across all NUMA nodes, then none of the sockets will be able to go to sleep.","commit_id":"afcb6833dc56ff7bbfe7a696e35b2e1b7a975fb4"},{"author":{"_account_id":8175,"name":"Rafael Folco","email":"rfolco@redhat.com","username":"rfolco"},"change_message_id":"233b154f2efbdc5e1bd4a2ad8dd106c1ad8e7462","unresolved":false,"context_lines":[{"line_number":11,"context_line":"This patch fixes how instances are placed over NUMA nodes. The current"},{"line_number":12,"context_line":"behavior has undesired effects, as packing all instances to node 0: e.g:"},{"line_number":13,"context_line":"when hw:numa_nodes\u003d1, the result was to pack all instances on node 0,"},{"line_number":14,"context_line":"leaving the other nodes free (unused):"},{"line_number":15,"context_line":"instance1 --\u003e node0, instance2 --\u003e node0, instance3 --\u003e node0"},{"line_number":16,"context_line":""},{"line_number":17,"context_line":"The new criteria optimizes NUMA usage by spreading instances across all"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":2,"id":"1aa78d24_bef1ea03","line":14,"in_reply_to":"1aa78d24_aa152c4a","updated":"2016-07-14 17:33:01.000000000","message":"The new criteria (spreading) focus in performance. Power-savings (packing) would be a nice feature to implement, though.\nThe current behavior is actually \"undefined\": It always use node0 until it reaches overall system limits. So you have all instances placed to node0. The other nodes are *never* used, unless you use numa_nodes \u003e 1.\nWith the current implementation, you can\u0027t effectively tell how many cpus are in use when you float over all cpus in the assigned numa node (no cpu pinning in place).","commit_id":"afcb6833dc56ff7bbfe7a696e35b2e1b7a975fb4"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"777d4ee7073135886bee404128b1a713d042d570","unresolved":false,"context_lines":[{"line_number":21,"context_line":"Some other possible criteria are: packing or random. A new config setting"},{"line_number":22,"context_line":"in the flavor or image metadata is required to implement those, though."},{"line_number":23,"context_line":"Also, for packing, there are some (non-trivial) changes required to make"},{"line_number":24,"context_line":"the cell node accountable, by updating host_cell.avail_cpus."},{"line_number":25,"context_line":""},{"line_number":26,"context_line":"Change-Id: I2bb9887184827340f059b1229ca2df146fe6a7c4"},{"line_number":27,"context_line":"Closes-Bug: #1601994"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":4,"id":"dada55a8_74098331","line":24,"updated":"2016-07-18 12:39:49.000000000","message":"The current behaviour *intentionally* packs NUMA nodes in order as this provides more efficient use of hardware resources. If you always spread guests across nodes, then you can end up in a situation where you have lots of unused resource across you hosts, but are unable to actually place new guests, because no single node has sufficient space.\n\nFor example, consider you have a host with 8 GB in node0 and 8 GB in node1. You launch a 4 GB guest, and you launch a 2 GB guest. Your patch puts them in node0 and node1 respectively. You now have 10 GB free space overall, but you are unable to launch an 8 GB guest, because no single node has enough space.\n\nThere are tradeoffs between different placement strategies and Nova has picked the strategy that maximises chances of being able to launch new VMs.\n\nIf you want to suggest alternative strategies, then this needs to be done via a blueprint and spec to outline the rational and design","commit_id":"75652814a7019d18cd8476e1ec6bc5ca627524d4"},{"author":{"_account_id":8175,"name":"Rafael Folco","email":"rfolco@redhat.com","username":"rfolco"},"change_message_id":"921b24ff3bd3ebbda1e429ca67c87344e4c688ec","unresolved":false,"context_lines":[{"line_number":21,"context_line":"Some other possible criteria are: packing or random. A new config setting"},{"line_number":22,"context_line":"in the flavor or image metadata is required to implement those, though."},{"line_number":23,"context_line":"Also, for packing, there are some (non-trivial) changes required to make"},{"line_number":24,"context_line":"the cell node accountable, by updating host_cell.avail_cpus."},{"line_number":25,"context_line":""},{"line_number":26,"context_line":"Change-Id: I2bb9887184827340f059b1229ca2df146fe6a7c4"},{"line_number":27,"context_line":"Closes-Bug: #1601994"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":4,"id":"dada55a8_ca6c049f","line":24,"in_reply_to":"dada55a8_74098331","updated":"2016-07-18 13:13:24.000000000","message":"The current code isn\u0027t packing when you do NOT set cpu pinning (hw:cpu_policy UNSET) and hw:numa_nodes\u003d1: the resulting behaviour is to always place instances to numa node[0]. The current code needs to be fixed regardless of the criteria adopted. This happens because the code path is not updating available cpus (in the scenario described above). When the node is fully packed, or the new instance does not fit in that node, then the instance needs to be place in the next node. This is not happening. I\u0027ll send a new patch to fix the bug using PACKING criteria instead (update host_cell.avail_cpus even when cpu pinning is not in place).\nMakes sense?","commit_id":"75652814a7019d18cd8476e1ec6bc5ca627524d4"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"364f4c9345a5dd93deaffd26405a12f62f0840c0","unresolved":false,"context_lines":[{"line_number":21,"context_line":"Some other possible criteria are: packing or random. A new config setting"},{"line_number":22,"context_line":"in the flavor or image metadata is required to implement those, though."},{"line_number":23,"context_line":"Also, for packing, there are some (non-trivial) changes required to make"},{"line_number":24,"context_line":"the cell node accountable, by updating host_cell.avail_cpus."},{"line_number":25,"context_line":""},{"line_number":26,"context_line":"Change-Id: I2bb9887184827340f059b1229ca2df146fe6a7c4"},{"line_number":27,"context_line":"Closes-Bug: #1601994"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":4,"id":"dada55a8_f3b432f3","line":24,"in_reply_to":"dada55a8_ca6c049f","updated":"2016-07-19 11:20:23.000000000","message":"I don\u0027t see that behaviour at all. I have a host with 3 NUMA nodes (8 GB, 4 GB and 4 GB respectively)\n\nSetup a flavour with 2 GB RAM and boot 10 guests\n\n$ glance image-update --property hw_numa_nodes\u003d1 f9a87240-a717-4909-92a9-8e130eeaa8be\n$ nova flavor-create m1.numa 20 2048 1 1\n$ for i in `seq 1 10` ; do nova boot --image f9a87240-a717-4909-92a9-8e130eeaa8be --flavor m1.numa demo$i ; done\n\nAnd see them spread across the nodes:\n\n# grep memnode /etc/libvirt//qemu/*xml\n/etc/libvirt//qemu/instance-00000039.xml:    \u003cmemnode cellid\u003d\u00270\u0027 mode\u003d\u0027strict\u0027 nodeset\u003d\u00270\u0027/\u003e\n/etc/libvirt//qemu/instance-0000003a.xml:    \u003cmemnode cellid\u003d\u00270\u0027 mode\u003d\u0027strict\u0027 nodeset\u003d\u00270\u0027/\u003e\n/etc/libvirt//qemu/instance-0000003b.xml:    \u003cmemnode cellid\u003d\u00270\u0027 mode\u003d\u0027strict\u0027 nodeset\u003d\u00270\u0027/\u003e\n/etc/libvirt//qemu/instance-0000003c.xml:    \u003cmemnode cellid\u003d\u00270\u0027 mode\u003d\u0027strict\u0027 nodeset\u003d\u00270\u0027/\u003e\n/etc/libvirt//qemu/instance-0000003d.xml:    \u003cmemnode cellid\u003d\u00270\u0027 mode\u003d\u0027strict\u0027 nodeset\u003d\u00270\u0027/\u003e\n/etc/libvirt//qemu/instance-0000003e.xml:    \u003cmemnode cellid\u003d\u00270\u0027 mode\u003d\u0027strict\u0027 nodeset\u003d\u00271\u0027/\u003e\n/etc/libvirt//qemu/instance-0000003f.xml:    \u003cmemnode cellid\u003d\u00270\u0027 mode\u003d\u0027strict\u0027 nodeset\u003d\u00271\u0027/\u003e\n/etc/libvirt//qemu/instance-00000040.xml:    \u003cmemnode cellid\u003d\u00270\u0027 mode\u003d\u0027strict\u0027 nodeset\u003d\u00272\u0027/\u003e\n/etc/libvirt//qemu/instance-00000041.xml:    \u003cmemnode cellid\u003d\u00270\u0027 mode\u003d\u0027strict\u0027 nodeset\u003d\u00272\u0027/\u003e\n\n\nNote that the default RAM overcommit ratio is 1.5, hence logical space on node 0 is 12 GB, hence why there are 5 guests  placed there before choosing to use node 1, and so on.","commit_id":"75652814a7019d18cd8476e1ec6bc5ca627524d4"},{"author":{"_account_id":8175,"name":"Rafael Folco","email":"rfolco@redhat.com","username":"rfolco"},"change_message_id":"f6e1b9c7d1ca26ece8ae7d2c1a732714dcf378fa","unresolved":false,"context_lines":[{"line_number":21,"context_line":"Some other possible criteria are: packing or random. A new config setting"},{"line_number":22,"context_line":"in the flavor or image metadata is required to implement those, though."},{"line_number":23,"context_line":"Also, for packing, there are some (non-trivial) changes required to make"},{"line_number":24,"context_line":"the cell node accountable, by updating host_cell.avail_cpus."},{"line_number":25,"context_line":""},{"line_number":26,"context_line":"Change-Id: I2bb9887184827340f059b1229ca2df146fe6a7c4"},{"line_number":27,"context_line":"Closes-Bug: #1601994"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":4,"id":"dada55a8_bc8ba21d","line":24,"in_reply_to":"dada55a8_f3b432f3","updated":"2016-07-19 13:30:38.000000000","message":"I figured out what is going on, explained here:\nhttps://bugzilla.redhat.com/show_bug.cgi?id\u003d1189906#c5\nhttps://bugs.launchpad.net/nova/+bug/1427772\n\nI believe we should set limits here:\nhttps://github.com/openstack/nova/blob/master/nova/scheduler/utils.py#L148\n\nTo avoid stacking all instances into node0:\nhttps://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L866\n\nHowever, I am not sure if this has any side effects.","commit_id":"75652814a7019d18cd8476e1ec6bc5ca627524d4"}]}