)]}'
{"specs/newton/approved/libvirt-emulator-threads-policy.rst":[{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"82dd07dbe26e67c5b4dd3beec272fedc921446ad","unresolved":false,"context_lines":[{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/nova/+spec/libvirt-emulator-threads-policy"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"The Nova schedular determines CPU resource utilization and instance"},{"line_number":14,"context_line":"CPU placement based on the number of vCPUs in the flavour. A number"},{"line_number":15,"context_line":"of hypervisors have work that is performed in the host OS on behalf"},{"line_number":16,"context_line":"of a guest instance, which does not take place in association with"}],"source_content_type":"text/x-rst","patch_set":1,"id":"dab17558_7b7a6178","line":13,"range":{"start_line":13,"start_character":9,"end_line":13,"end_character":18},"updated":"2016-05-23 14:26:19.000000000","message":"nit: s/schedular/scheduler/\n\n(rest of doc too)","commit_id":"f5ddac5ec9f140fd2d8c0265a295c1b1cf147468"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"e78864c5461c29320c0a53f139dd0fb603b87c1e","unresolved":false,"context_lines":[{"line_number":37,"context_line":""},{"line_number":38,"context_line":"Nova does not have any current mechanism to either track this extra"},{"line_number":39,"context_line":"guest instance compute requirement in order to measure utilization,"},{"line_number":40,"context_line":"nor to place any control over its execution policy."},{"line_number":41,"context_line":""},{"line_number":42,"context_line":"The libvirt driver has implemented a generic placement policy for"},{"line_number":43,"context_line":"KVM whereby the QEMU emulator threads are allowed to float across"}],"source_content_type":"text/x-rst","patch_set":1,"id":"dab17558_3c42bc3e","line":40,"updated":"2016-05-16 17:03:39.000000000","message":"The closest thing to this is the RAM overhead here:\nhttps://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L125\n\nBut I get that this is different, it feels like RAM overhead could be modelled in the same way.","commit_id":"f5ddac5ec9f140fd2d8c0265a295c1b1cf147468"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"e78864c5461c29320c0a53f139dd0fb603b87c1e","unresolved":false,"context_lines":[{"line_number":50,"context_line":"to be dedicated to a guest\u0027s emulator threads. This becomes even"},{"line_number":51,"context_line":"more important if Nova gains support for real-time workloads, as"},{"line_number":52,"context_line":"it will not be acceptable to allow emulator threads to steal time"},{"line_number":53,"context_line":"from real-time vCPUs."},{"line_number":54,"context_line":""},{"line_number":55,"context_line":"While it would be possible for the libvirt driver to add different"},{"line_number":56,"context_line":"placement policies, unless the concept of emulator threads is"}],"source_content_type":"text/x-rst","patch_set":1,"id":"dab17558_dc10d002","line":53,"updated":"2016-05-16 17:03:39.000000000","message":"You mean in addition to the vCPU-pin-set pinning?\nhttps://git.openstack.org/cgit/openstack/nova/tree/nova/conf/virt.py#n20\n\nIn Xen world, you basically give Dom0 some pCPUs on a per host basis, and they can do the work. You can pin Dom0\u0027s CPUs to the matching anti-set.\n\nI guess the key point here is we need different amounts for each flavor.","commit_id":"f5ddac5ec9f140fd2d8c0265a295c1b1cf147468"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"2774c2c1ff6e2d72847e421560b349bf117d9ddd","unresolved":false,"context_lines":[{"line_number":62,"context_line":"Use Cases"},{"line_number":63,"context_line":"----------"},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"The users of dedicated (no overcommit) CPU guests, particular"},{"line_number":66,"context_line":"those with real-time schedular policy, will want to configure"},{"line_number":67,"context_line":"guests such that emulator threads are placed separately from"},{"line_number":68,"context_line":"vCPUs. This will be desired by the NFV community, as well as"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_cac06e7f","line":65,"range":{"start_line":65,"start_character":51,"end_line":65,"end_character":61},"updated":"2016-05-24 19:38:39.000000000","message":"\"particularly\" or \"in particular\"","commit_id":"f5ddac5ec9f140fd2d8c0265a295c1b1cf147468"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"2774c2c1ff6e2d72847e421560b349bf117d9ddd","unresolved":false,"context_lines":[{"line_number":91,"context_line":""},{"line_number":92,"context_line":"would say that this instance is to be considered to have 2 system"},{"line_number":93,"context_line":"pseudo-CPUs. If this is omitted, it is assumed that the value is"},{"line_number":94,"context_line":"zero."},{"line_number":95,"context_line":""},{"line_number":96,"context_line":"In the virt drivers, when reporting the available resources to"},{"line_number":97,"context_line":"the schedular, the \u0027vcpu_total\u0027 field value will have to be"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_6a2b4234","line":94,"updated":"2016-05-24 19:38:39.000000000","message":"So are you saying that if my flavor says I get 8 vpus and I have 2 for this new property then my instance will actually consume 10 cpus on the host?\n\nI think the accounting for that is going to be difficult since a compute node is going to expose N cpus available and we\u0027re going to subtract from that amount in the scheduler resource pool filter. It would be a lot better if we said the flavor should have 10 and 2 of those will be dedicated to system tasks and not visible inside the guest.","commit_id":"f5ddac5ec9f140fd2d8c0265a295c1b1cf147468"},{"author":{"_account_id":7730,"name":"Sahid Orentino Ferdjaoui","email":"sahid.ferdjaoui@industrialdiscipline.com","username":"sahid"},"change_message_id":"7a921643c1ad6c61e2ec60eb407f3d4ef3fa17a4","unresolved":false,"context_lines":[{"line_number":91,"context_line":""},{"line_number":92,"context_line":"would say that this instance is to be considered to have 2 system"},{"line_number":93,"context_line":"pseudo-CPUs. If this is omitted, it is assumed that the value is"},{"line_number":94,"context_line":"zero."},{"line_number":95,"context_line":""},{"line_number":96,"context_line":"In the virt drivers, when reporting the available resources to"},{"line_number":97,"context_line":"the schedular, the \u0027vcpu_total\u0027 field value will have to be"}],"source_content_type":"text/x-rst","patch_set":1,"id":"fa7ab95a_6bc4cd5e","line":94,"in_reply_to":"bab6814e_6a2b4234","updated":"2016-09-01 11:11:06.000000000","message":"We can probably do that, I think we want to expose a distinction between resources which are going to be consumed by users which are vCPUs and resources consumed by system, but under the hood we are going to compute it as users are requesting only for vCPUs.","commit_id":"f5ddac5ec9f140fd2d8c0265a295c1b1cf147468"},{"author":{"_account_id":6772,"name":"Stephen Gordon","email":"sgordon@redhat.com","username":"sgordon"},"change_message_id":"0cad7406537c038af5db9d60f3c5ea9cdc13bbb7","unresolved":false,"context_lines":[{"line_number":94,"context_line":"zero."},{"line_number":95,"context_line":""},{"line_number":96,"context_line":"In the virt drivers, when reporting the available resources to"},{"line_number":97,"context_line":"the schedular, the \u0027vcpu_total\u0027 field value will have to be"},{"line_number":98,"context_line":"adjusted to include the hw:cpu_system value set for any running"},{"line_number":99,"context_line":"instances."},{"line_number":100,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"5aef4532_b76a417f","line":97,"updated":"2016-03-10 01:52:49.000000000","message":"I think this should actually be reflected in vcpu_used?","commit_id":"f5ddac5ec9f140fd2d8c0265a295c1b1cf147468"},{"author":{"_account_id":7730,"name":"Sahid Orentino Ferdjaoui","email":"sahid.ferdjaoui@industrialdiscipline.com","username":"sahid"},"change_message_id":"eba169b814acf0a15c6d31429fe6fb26561b3260","unresolved":false,"context_lines":[{"line_number":94,"context_line":"zero."},{"line_number":95,"context_line":""},{"line_number":96,"context_line":"In the virt drivers, when reporting the available resources to"},{"line_number":97,"context_line":"the schedular, the \u0027vcpu_total\u0027 field value will have to be"},{"line_number":98,"context_line":"adjusted to include the hw:cpu_system value set for any running"},{"line_number":99,"context_line":"instances."},{"line_number":100,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"5aef4532_412f973e","line":97,"in_reply_to":"5aef4532_b76a417f","updated":"2016-03-10 15:36:15.000000000","message":"Yes I think you\u0027re right, \u0027vcpu_total\u0027 reflects the number of cpus available on host, it\u0027s returned by virt\u0027s layer whereas \u0027vcpu_used\u0027 is the key maintained on compute node which will be something like:\n\n  compute_node[vcpu_used] +\u003d instance.vcpus + instance.cpu_system","commit_id":"f5ddac5ec9f140fd2d8c0265a295c1b1cf147468"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"2774c2c1ff6e2d72847e421560b349bf117d9ddd","unresolved":false,"context_lines":[{"line_number":96,"context_line":"In the virt drivers, when reporting the available resources to"},{"line_number":97,"context_line":"the schedular, the \u0027vcpu_total\u0027 field value will have to be"},{"line_number":98,"context_line":"adjusted to include the hw:cpu_system value set for any running"},{"line_number":99,"context_line":"instances."},{"line_number":100,"context_line":""},{"line_number":101,"context_line":"When the BaseCoreFilter schedular filter is checking a host\u0027s"},{"line_number":102,"context_line":"suitability, instead of checking just the flavour vCPU count,"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_6a87c212","line":99,"updated":"2016-05-24 19:38:39.000000000","message":"Well, remember that those legacy fields are going away RSN. We\u0027ll expose total as total and each instance will have a number of allocated vcpus. The amount available on a compute node at any time is the total minus the sum of the allocations. Hence, much easier if we say that the system cpus come out of the flavor\u0027s total because that math works the same way. If we don\u0027t then we basically have to have a pool of system cpus separate from the regular ones to decrement with allocations for instances that need this, which is of course not ideal.","commit_id":"f5ddac5ec9f140fd2d8c0265a295c1b1cf147468"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"2774c2c1ff6e2d72847e421560b349bf117d9ddd","unresolved":false,"context_lines":[{"line_number":101,"context_line":"When the BaseCoreFilter schedular filter is checking a host\u0027s"},{"line_number":102,"context_line":"suitability, instead of checking just the flavour vCPU count,"},{"line_number":103,"context_line":"it will have to include the hw:cpu_system value too. The over"},{"line_number":104,"context_line":"commit ratio will applied to the combined value."},{"line_number":105,"context_line":""},{"line_number":106,"context_line":"IOW, given a flavour with"},{"line_number":107,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_0a73964e","line":104,"updated":"2016-05-24 19:38:39.000000000","message":"We\u0027re hoping for the core filter to go away in Ocata or so.","commit_id":"f5ddac5ec9f140fd2d8c0265a295c1b1cf147468"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"e78864c5461c29320c0a53f139dd0fb603b87c1e","unresolved":false,"context_lines":[{"line_number":106,"context_line":"IOW, given a flavour with"},{"line_number":107,"context_line":""},{"line_number":108,"context_line":"* vcpus\u003d8"},{"line_number":109,"context_line":"* hw:cpu_system\u003d2"},{"line_number":110,"context_line":"* cpu_overcommit\u003d1.5"},{"line_number":111,"context_line":""},{"line_number":112,"context_line":"The instance will have a total of 10 vcpus required, which with"}],"source_content_type":"text/x-rst","patch_set":1,"id":"dab17558_5014c142","line":109,"updated":"2016-05-16 17:03:39.000000000","message":"is this actually a per disk and per vif amount, in reality?","commit_id":"f5ddac5ec9f140fd2d8c0265a295c1b1cf147468"},{"author":{"_account_id":7730,"name":"Sahid Orentino Ferdjaoui","email":"sahid.ferdjaoui@industrialdiscipline.com","username":"sahid"},"change_message_id":"7a921643c1ad6c61e2ec60eb407f3d4ef3fa17a4","unresolved":false,"context_lines":[{"line_number":106,"context_line":"IOW, given a flavour with"},{"line_number":107,"context_line":""},{"line_number":108,"context_line":"* vcpus\u003d8"},{"line_number":109,"context_line":"* hw:cpu_system\u003d2"},{"line_number":110,"context_line":"* cpu_overcommit\u003d1.5"},{"line_number":111,"context_line":""},{"line_number":112,"context_line":"The instance will have a total of 10 vcpus required, which with"}],"source_content_type":"text/x-rst","patch_set":1,"id":"fa7ab95a_9d5a3882","line":109,"in_reply_to":"bab6814e_250edd8b","updated":"2016-09-01 11:11:06.000000000","message":"Thinking again I don\u0027t think we should to make that option available with over-commitment and we don\u0027t want a ratio. We want to use that feature with pinning. So we request an amount of host CPUs, some of them are going to be pinned to be used as vCPUs and some other pinned to be used as \"system pseudo-CPUs\".\n\nAbout to make that feature working per disk or vif, I think that granularity can come after, if a real demand from users is done. The only one we may want so far is to separate vCPUs and system pseudo-CPUs on different NUMA nodes.","commit_id":"f5ddac5ec9f140fd2d8c0265a295c1b1cf147468"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"82dd07dbe26e67c5b4dd3beec272fedc921446ad","unresolved":false,"context_lines":[{"line_number":106,"context_line":"IOW, given a flavour with"},{"line_number":107,"context_line":""},{"line_number":108,"context_line":"* vcpus\u003d8"},{"line_number":109,"context_line":"* hw:cpu_system\u003d2"},{"line_number":110,"context_line":"* cpu_overcommit\u003d1.5"},{"line_number":111,"context_line":""},{"line_number":112,"context_line":"The instance will have a total of 10 vcpus required, which with"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_250edd8b","line":109,"in_reply_to":"dab17558_5014c142","updated":"2016-05-23 14:26:19.000000000","message":"A similar question: is the number of QEMU emulator threads expected to scale with the number of cores? If so, could we not specify this as a ratio, i.e. N guest vCPUs \u003d M psudo-CPUs?\n\nEDIT: See this is addressed below. Leaving here for posterity.","commit_id":"f5ddac5ec9f140fd2d8c0265a295c1b1cf147468"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"82dd07dbe26e67c5b4dd3beec272fedc921446ad","unresolved":false,"context_lines":[{"line_number":126,"context_line":"with vCPUs will only be used if hw:cpu_system is not set."},{"line_number":127,"context_line":""},{"line_number":128,"context_line":"When the guest instance has more than one NUMA node enabled there"},{"line_number":129,"context_line":"will need to be a way to indicate which NUMA node the system"},{"line_number":130,"context_line":"pseudo-CPUs should be associated with. The hw:cpu_system count is"},{"line_number":131,"context_line":"not referring to the number of threads the hypervisor uses, but"},{"line_number":132,"context_line":"rather the number of host CPUs we want to reserve. e.g. QEMU may"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_05e0f9cb","line":129,"range":{"start_line":129,"start_character":34,"end_line":129,"end_character":49},"updated":"2016-05-23 14:26:19.000000000","message":"Why only one NUMA node?","commit_id":"f5ddac5ec9f140fd2d8c0265a295c1b1cf147468"},{"author":{"_account_id":7730,"name":"Sahid Orentino Ferdjaoui","email":"sahid.ferdjaoui@industrialdiscipline.com","username":"sahid"},"change_message_id":"7a921643c1ad6c61e2ec60eb407f3d4ef3fa17a4","unresolved":false,"context_lines":[{"line_number":126,"context_line":"with vCPUs will only be used if hw:cpu_system is not set."},{"line_number":127,"context_line":""},{"line_number":128,"context_line":"When the guest instance has more than one NUMA node enabled there"},{"line_number":129,"context_line":"will need to be a way to indicate which NUMA node the system"},{"line_number":130,"context_line":"pseudo-CPUs should be associated with. The hw:cpu_system count is"},{"line_number":131,"context_line":"not referring to the number of threads the hypervisor uses, but"},{"line_number":132,"context_line":"rather the number of host CPUs we want to reserve. e.g. QEMU may"}],"source_content_type":"text/x-rst","patch_set":1,"id":"fa7ab95a_bda29c49","line":129,"in_reply_to":"bab6814e_05e0f9cb","updated":"2016-09-01 11:11:06.000000000","message":"What do you mean ? if guest only have one NUMA node so we can\u0027t really dispatch vCPUs and system pseudo-CPUs.","commit_id":"f5ddac5ec9f140fd2d8c0265a295c1b1cf147468"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"82dd07dbe26e67c5b4dd3beec272fedc921446ad","unresolved":false,"context_lines":[{"line_number":136,"context_line":"single guest NUMA node to associate the system psuedo-CPUs with"},{"line_number":137,"context_line":"e.g. the following says to associate with NUMA node 3."},{"line_number":138,"context_line":""},{"line_number":139,"context_line":"* hw:numa_cpu_system\u003d3"},{"line_number":140,"context_line":""},{"line_number":141,"context_line":"Alternatives"},{"line_number":142,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_85b34997","line":139,"range":{"start_line":139,"start_character":0,"end_line":139,"end_character":22},"updated":"2016-05-23 14:26:19.000000000","message":"Based on what I\u0027ve seen, so far we\u0027ve refrained from exposing the actual host topology to the user. For example: a user can ask for N NUMA nodes, and a given amount of CPUS on each node, but they cannot ask that specific host nodes be used. This seems to break that abstraction, which doesn\u0027t seem like something we should do.","commit_id":"f5ddac5ec9f140fd2d8c0265a295c1b1cf147468"},{"author":{"_account_id":7730,"name":"Sahid Orentino Ferdjaoui","email":"sahid.ferdjaoui@industrialdiscipline.com","username":"sahid"},"change_message_id":"7a921643c1ad6c61e2ec60eb407f3d4ef3fa17a4","unresolved":false,"context_lines":[{"line_number":136,"context_line":"single guest NUMA node to associate the system psuedo-CPUs with"},{"line_number":137,"context_line":"e.g. the following says to associate with NUMA node 3."},{"line_number":138,"context_line":""},{"line_number":139,"context_line":"* hw:numa_cpu_system\u003d3"},{"line_number":140,"context_line":""},{"line_number":141,"context_line":"Alternatives"},{"line_number":142,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"fa7ab95a_60ff8701","line":139,"in_reply_to":"bab6814e_0ad8b61c","updated":"2016-09-01 11:11:06.000000000","message":"That option is not used to specify on which host NUMA nodes we want system pseudo-CPUs to be associated but on guest NUMA nodes.\n\nSo if the flavor is requesting for 4 NUMA nodes, we may want to associate system pseudo-CPUs on NUMA node ID 3.\n\n  hw:numa_nodes \u003d 4\n  hw:numa_cpu_system\u003d3","commit_id":"f5ddac5ec9f140fd2d8c0265a295c1b1cf147468"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"2774c2c1ff6e2d72847e421560b349bf117d9ddd","unresolved":false,"context_lines":[{"line_number":136,"context_line":"single guest NUMA node to associate the system psuedo-CPUs with"},{"line_number":137,"context_line":"e.g. the following says to associate with NUMA node 3."},{"line_number":138,"context_line":""},{"line_number":139,"context_line":"* hw:numa_cpu_system\u003d3"},{"line_number":140,"context_line":""},{"line_number":141,"context_line":"Alternatives"},{"line_number":142,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_0ad8b61c","line":139,"range":{"start_line":139,"start_character":0,"end_line":139,"end_character":22},"in_reply_to":"bab6814e_85b34997","updated":"2016-05-24 19:38:39.000000000","message":"Agreed, this seems really detailed. Why not pick this?","commit_id":"f5ddac5ec9f140fd2d8c0265a295c1b1cf147468"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"82dd07dbe26e67c5b4dd3beec272fedc921446ad","unresolved":false,"context_lines":[{"line_number":149,"context_line":"are used, it will be critical to separate emulator threads for different"},{"line_number":150,"context_line":"KVM instances."},{"line_number":151,"context_line":""},{"line_number":152,"context_line":"Another option is to hardcode an assumption that the vCPUs number set"},{"line_number":153,"context_line":"against the flavour implicitly includes 1 vCPUs for emulator. eg a"},{"line_number":154,"context_line":"vCPU value of 5 would imply 4 actual vCPUs and 1 system pseudo-vCPU."},{"line_number":155,"context_line":"This would likely be extremely confusing to tenant users, and developers"},{"line_number":156,"context_line":"alike."},{"line_number":157,"context_line":""},{"line_number":158,"context_line":"Do nothing is always an option. If we did nothing, then it would limit"},{"line_number":159,"context_line":"the types of workload that can be run on Nova. This would have a negative"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_6523c5bd","line":156,"range":{"start_line":152,"start_character":0,"end_line":156,"end_character":6},"updated":"2016-05-23 14:26:19.000000000","message":"For most users, yes. However, using NUMA topologies already has implications for tracking CPU usage, e.g. overcommit ratios do not have effect. I think, if documented clearly, that this is a viable option.","commit_id":"f5ddac5ec9f140fd2d8c0265a295c1b1cf147468"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"2774c2c1ff6e2d72847e421560b349bf117d9ddd","unresolved":false,"context_lines":[{"line_number":149,"context_line":"are used, it will be critical to separate emulator threads for different"},{"line_number":150,"context_line":"KVM instances."},{"line_number":151,"context_line":""},{"line_number":152,"context_line":"Another option is to hardcode an assumption that the vCPUs number set"},{"line_number":153,"context_line":"against the flavour implicitly includes 1 vCPUs for emulator. eg a"},{"line_number":154,"context_line":"vCPU value of 5 would imply 4 actual vCPUs and 1 system pseudo-vCPU."},{"line_number":155,"context_line":"This would likely be extremely confusing to tenant users, and developers"},{"line_number":156,"context_line":"alike."},{"line_number":157,"context_line":""},{"line_number":158,"context_line":"Do nothing is always an option. If we did nothing, then it would limit"},{"line_number":159,"context_line":"the types of workload that can be run on Nova. This would have a negative"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_8a1026aa","line":156,"range":{"start_line":152,"start_character":0,"end_line":156,"end_character":6},"in_reply_to":"bab6814e_6523c5bd","updated":"2016-05-24 19:38:39.000000000","message":"We could also do one per four vcpus or something like that.","commit_id":"f5ddac5ec9f140fd2d8c0265a295c1b1cf147468"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"2774c2c1ff6e2d72847e421560b349bf117d9ddd","unresolved":false,"context_lines":[{"line_number":157,"context_line":""},{"line_number":158,"context_line":"Do nothing is always an option. If we did nothing, then it would limit"},{"line_number":159,"context_line":"the types of workload that can be run on Nova. This would have a negative"},{"line_number":160,"context_line":"impact inparticular on users making use of the dedicated vCPU feature,"},{"line_number":161,"context_line":"as there would be no way to guarantee their vCPUs are not pre-empted by"},{"line_number":162,"context_line":"emulator threads. It can be worked around to some degree with realtime"},{"line_number":163,"context_line":"by setting a fixed policy that the emulator threads only run on the vCPUs"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_0a1d56bd","line":160,"range":{"start_line":160,"start_character":7,"end_line":160,"end_character":19},"updated":"2016-05-24 19:38:39.000000000","message":"Need a space.","commit_id":"f5ddac5ec9f140fd2d8c0265a295c1b1cf147468"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"2774c2c1ff6e2d72847e421560b349bf117d9ddd","unresolved":false,"context_lines":[{"line_number":172,"context_line":""},{"line_number":173,"context_line":"* cpuset_system\u003dSetOfIntegersField()"},{"line_number":174,"context_line":""},{"line_number":175,"context_line":"This field will defualt to None, so a schema migration change just"},{"line_number":176,"context_line":"adds the new column with no data. It will only be populated if the"},{"line_number":177,"context_line":"hw:cpu_system flavour extra_spec is used and the NUMA filter is"},{"line_number":178,"context_line":"enabled, which will not be the case for any existing running instances."}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_ca698e27","line":175,"range":{"start_line":175,"start_character":16,"end_line":175,"end_character":23},"updated":"2016-05-24 19:38:39.000000000","message":"\"default\"","commit_id":"f5ddac5ec9f140fd2d8c0265a295c1b1cf147468"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"82dd07dbe26e67c5b4dd3beec272fedc921446ad","unresolved":false,"context_lines":[{"line_number":249,"context_line":"Testing"},{"line_number":250,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":251,"context_line":""},{"line_number":252,"context_line":"This can be tested in any CI system that is capable of testing the current"},{"line_number":253,"context_line":"NUMA and dedicated CPUs policy. ie, it requires ability to use KVM, not"},{"line_number":254,"context_line":"merely QEMU."},{"line_number":255,"context_line":""},{"line_number":256,"context_line":"Documentation Impact"},{"line_number":257,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_e0520312","line":254,"range":{"start_line":252,"start_character":0,"end_line":254,"end_character":12},"updated":"2016-05-23 14:26:19.000000000","message":"I can help with this, if this moves forward.","commit_id":"f5ddac5ec9f140fd2d8c0265a295c1b1cf147468"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"82dd07dbe26e67c5b4dd3beec272fedc921446ad","unresolved":false,"context_lines":[{"line_number":256,"context_line":"Documentation Impact"},{"line_number":257,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":258,"context_line":""},{"line_number":259,"context_line":"The documentation detailing NUMA and dedicated CPU policy usage will need"},{"line_number":260,"context_line":"to be extended to also describe the new options this work introduces."},{"line_number":261,"context_line":""},{"line_number":262,"context_line":"References"},{"line_number":263,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bab6814e_0050e709","line":260,"range":{"start_line":259,"start_character":0,"end_line":260,"end_character":69},"updated":"2016-05-23 14:26:19.000000000","message":"Please make sure this gets done :)","commit_id":"f5ddac5ec9f140fd2d8c0265a295c1b1cf147468"}],"specs/ocata/approved/libvirt-emulator-threads-policy.rst":[{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"f5e7877c72afa66243971d84395ead3f340f2ab6","unresolved":false,"context_lines":[{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/nova/+spec/libvirt-emulator-threads-policy"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"The Nova scheduler determines CPU resource utilization and instance"},{"line_number":14,"context_line":"CPU placement based on the number of vCPUs in the flavor. A number of"},{"line_number":15,"context_line":"hypervisors have operations that are being performed on behalf of the"},{"line_number":16,"context_line":"guest instance in the host OS. These operations should be acounted and"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a8bc5a2_73148d9f","line":13,"range":{"start_line":13,"start_character":4,"end_line":13,"end_character":8},"updated":"2016-09-26 10:36:54.000000000","message":"nova (too much time around docs people, clearly)","commit_id":"07518f1b504f993ed8073eeb6cff7e755cb90b79"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"f5e7877c72afa66243971d84395ead3f340f2ab6","unresolved":false,"context_lines":[{"line_number":14,"context_line":"CPU placement based on the number of vCPUs in the flavor. A number of"},{"line_number":15,"context_line":"hypervisors have operations that are being performed on behalf of the"},{"line_number":16,"context_line":"guest instance in the host OS. These operations should be acounted and"},{"line_number":17,"context_line":"scheduled separatelly, as well as have it\u0027s own placement policy"},{"line_number":18,"context_line":"controls applied."},{"line_number":19,"context_line":""},{"line_number":20,"context_line":"Problem description"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a8bc5a2_b3337561","line":17,"range":{"start_line":17,"start_character":10,"end_line":17,"end_character":21},"updated":"2016-09-26 10:36:54.000000000","message":"separately","commit_id":"07518f1b504f993ed8073eeb6cff7e755cb90b79"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"f5e7877c72afa66243971d84395ead3f340f2ab6","unresolved":false,"context_lines":[{"line_number":14,"context_line":"CPU placement based on the number of vCPUs in the flavor. A number of"},{"line_number":15,"context_line":"hypervisors have operations that are being performed on behalf of the"},{"line_number":16,"context_line":"guest instance in the host OS. These operations should be acounted and"},{"line_number":17,"context_line":"scheduled separatelly, as well as have it\u0027s own placement policy"},{"line_number":18,"context_line":"controls applied."},{"line_number":19,"context_line":""},{"line_number":20,"context_line":"Problem description"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a8bc5a2_f3417daa","line":17,"range":{"start_line":17,"start_character":39,"end_line":17,"end_character":43},"updated":"2016-09-26 10:36:54.000000000","message":"their?","commit_id":"07518f1b504f993ed8073eeb6cff7e755cb90b79"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"f5e7877c72afa66243971d84395ead3f340f2ab6","unresolved":false,"context_lines":[{"line_number":41,"context_line":""},{"line_number":42,"context_line":"The libvirt driver has implemented a generic placement policy for KVM"},{"line_number":43,"context_line":"whereby the QEMU emulator threads are allowed to float across the same"},{"line_number":44,"context_line":"pCPUs that the instance vCPUs are running on. IOW, the emulator"},{"line_number":45,"context_line":"threads will steal some time from the vCPUs whenever they have work to"},{"line_number":46,"context_line":"do. This is just about acceptable in the case where CPU over commit is"},{"line_number":47,"context_line":"being used. When guests want dedicated vCPU allocation though, there"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a8bc5a2_534a31ad","line":44,"range":{"start_line":44,"start_character":46,"end_line":44,"end_character":49},"updated":"2016-09-26 10:36:54.000000000","message":"In other words","commit_id":"07518f1b504f993ed8073eeb6cff7e755cb90b79"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"f5e7877c72afa66243971d84395ead3f340f2ab6","unresolved":false,"context_lines":[{"line_number":43,"context_line":"whereby the QEMU emulator threads are allowed to float across the same"},{"line_number":44,"context_line":"pCPUs that the instance vCPUs are running on. IOW, the emulator"},{"line_number":45,"context_line":"threads will steal some time from the vCPUs whenever they have work to"},{"line_number":46,"context_line":"do. This is just about acceptable in the case where CPU over commit is"},{"line_number":47,"context_line":"being used. When guests want dedicated vCPU allocation though, there"},{"line_number":48,"context_line":"is a desire to be able to express other placement policies. For"},{"line_number":49,"context_line":"example, to allocate one or more pCPUs to be dedicated to a guest\u0027s"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a8bc5a2_33e4c5aa","line":46,"range":{"start_line":46,"start_character":56,"end_line":46,"end_character":67},"updated":"2016-09-26 10:36:54.000000000","message":"overcommit?","commit_id":"07518f1b504f993ed8073eeb6cff7e755cb90b79"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"f5e7877c72afa66243971d84395ead3f340f2ab6","unresolved":false,"context_lines":[{"line_number":44,"context_line":"pCPUs that the instance vCPUs are running on. IOW, the emulator"},{"line_number":45,"context_line":"threads will steal some time from the vCPUs whenever they have work to"},{"line_number":46,"context_line":"do. This is just about acceptable in the case where CPU over commit is"},{"line_number":47,"context_line":"being used. When guests want dedicated vCPU allocation though, there"},{"line_number":48,"context_line":"is a desire to be able to express other placement policies. For"},{"line_number":49,"context_line":"example, to allocate one or more pCPUs to be dedicated to a guest\u0027s"},{"line_number":50,"context_line":"emulator threads. This becomes even more important if Nova gains"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a8bc5a2_53e15199","line":47,"updated":"2016-09-26 10:36:54.000000000","message":"However, when...","commit_id":"07518f1b504f993ed8073eeb6cff7e755cb90b79"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"f5e7877c72afa66243971d84395ead3f340f2ab6","unresolved":false,"context_lines":[{"line_number":44,"context_line":"pCPUs that the instance vCPUs are running on. IOW, the emulator"},{"line_number":45,"context_line":"threads will steal some time from the vCPUs whenever they have work to"},{"line_number":46,"context_line":"do. This is just about acceptable in the case where CPU over commit is"},{"line_number":47,"context_line":"being used. When guests want dedicated vCPU allocation though, there"},{"line_number":48,"context_line":"is a desire to be able to express other placement policies. For"},{"line_number":49,"context_line":"example, to allocate one or more pCPUs to be dedicated to a guest\u0027s"},{"line_number":50,"context_line":"emulator threads. This becomes even more important if Nova gains"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a8bc5a2_f3041d4b","line":47,"range":{"start_line":47,"start_character":55,"end_line":47,"end_character":61},"updated":"2016-09-26 10:36:54.000000000","message":"d/though/","commit_id":"07518f1b504f993ed8073eeb6cff7e755cb90b79"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"f5e7877c72afa66243971d84395ead3f340f2ab6","unresolved":false,"context_lines":[{"line_number":45,"context_line":"threads will steal some time from the vCPUs whenever they have work to"},{"line_number":46,"context_line":"do. This is just about acceptable in the case where CPU over commit is"},{"line_number":47,"context_line":"being used. When guests want dedicated vCPU allocation though, there"},{"line_number":48,"context_line":"is a desire to be able to express other placement policies. For"},{"line_number":49,"context_line":"example, to allocate one or more pCPUs to be dedicated to a guest\u0027s"},{"line_number":50,"context_line":"emulator threads. This becomes even more important if Nova gains"},{"line_number":51,"context_line":"support for real-time workloads, as it will not be acceptable to allow"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a8bc5a2_73be4d66","line":48,"range":{"start_line":48,"start_character":58,"end_line":48,"end_character":63},"updated":"2016-09-26 10:36:54.000000000","message":"policies, for example, to","commit_id":"07518f1b504f993ed8073eeb6cff7e755cb90b79"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"f5e7877c72afa66243971d84395ead3f340f2ab6","unresolved":false,"context_lines":[{"line_number":47,"context_line":"being used. When guests want dedicated vCPU allocation though, there"},{"line_number":48,"context_line":"is a desire to be able to express other placement policies. For"},{"line_number":49,"context_line":"example, to allocate one or more pCPUs to be dedicated to a guest\u0027s"},{"line_number":50,"context_line":"emulator threads. This becomes even more important if Nova gains"},{"line_number":51,"context_line":"support for real-time workloads, as it will not be acceptable to allow"},{"line_number":52,"context_line":"emulator threads to steal time from real-time vCPUs."},{"line_number":53,"context_line":""},{"line_number":54,"context_line":"While it would be possible for the libvirt driver to add different"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a8bc5a2_33a92525","line":51,"range":{"start_line":50,"start_character":51,"end_line":51,"end_character":11},"updated":"2016-09-26 10:36:54.000000000","message":"as nova continues to implement support for","commit_id":"07518f1b504f993ed8073eeb6cff7e755cb90b79"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"f5e7877c72afa66243971d84395ead3f340f2ab6","unresolved":false,"context_lines":[{"line_number":55,"context_line":"placement policies, unless the concept of emulator threads is exposed"},{"line_number":56,"context_line":"to the schedular in some manner, CPU usage cannot be expressed in a"},{"line_number":57,"context_line":"satisfactory manner. Thus there needs to be a way to describe to the"},{"line_number":58,"context_line":"schedular what other CPU usage may be associated with a guest, and"},{"line_number":59,"context_line":"account for that during placement."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"Use Cases"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a8bc5a2_537dd1af","line":58,"range":{"start_line":58,"start_character":0,"end_line":58,"end_character":9},"updated":"2016-09-26 10:36:54.000000000","message":":%s/schedular/scheduler","commit_id":"07518f1b504f993ed8073eeb6cff7e755cb90b79"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"f5e7877c72afa66243971d84395ead3f340f2ab6","unresolved":false,"context_lines":[{"line_number":91,"context_line":"Would say that this instance is to be considered to consumes 1 host"},{"line_number":92,"context_line":"CPU. Currently there is no desire to make customizable the number of"},{"line_number":93,"context_line":"host CPUs running emulator threads since only one should work for"},{"line_number":94,"context_line":"almost every usecases, an other limitation on that spec is that some"},{"line_number":95,"context_line":"usecases may need to isolate emulator threads on a separate host NUMA"},{"line_number":96,"context_line":"node which is currently not possible to achieve."},{"line_number":97,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a8bc5a2_ce7134aa","line":94,"range":{"start_line":94,"start_character":21,"end_line":94,"end_character":31},"updated":"2016-09-26 10:36:54.000000000","message":". Another","commit_id":"07518f1b504f993ed8073eeb6cff7e755cb90b79"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"f5e7877c72afa66243971d84395ead3f340f2ab6","unresolved":false,"context_lines":[{"line_number":98,"context_line":"Alternatives"},{"line_number":99,"context_line":"------------"},{"line_number":100,"context_line":""},{"line_number":101,"context_line":"We could use a host level tunable to just reserve a set of host pCPUs"},{"line_number":102,"context_line":"for running emulator threads globally, instead of trying to account"},{"line_number":103,"context_line":"for it per instance. This would work in the simple case, but when NUMA"},{"line_number":104,"context_line":"is used, it is highly desirable to have more fine grained config to"},{"line_number":105,"context_line":"control emulator thread placement. When real-time or dedicated CPUs"},{"line_number":106,"context_line":"are used, it will be critical to separate emulator threads for"},{"line_number":107,"context_line":"different KVM instances."},{"line_number":108,"context_line":""},{"line_number":109,"context_line":"Another option is to hardcode an assumption that the vCPUs number set"},{"line_number":110,"context_line":"against the flavour implicitly includes 1 vCPUs for emulator. eg a"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a8bc5a2_4ea60454","line":107,"range":{"start_line":101,"start_character":0,"end_line":107,"end_character":24},"updated":"2016-09-26 10:36:54.000000000","message":"I\u0027d like to dig a little more into this alternative. What\u0027s so special about emulator threads that they would require an additional host level tunable? There are many other examples of processes that would impact on high-performance workloads* and when the existing recommendation for configuring a host for these workloads is to use \u0027vcpu_pin_set\u0027 and \u0027reserved_host_memory_mb\u0027. Why are these not suitable options here and, if so unsuitable, is there anything we can do to make them better?\n\n* anything using DPDK\u0027s poll-mode driver, such as OVS w/ DPDK extensions, would be one. The kernel itself seems like another","commit_id":"07518f1b504f993ed8073eeb6cff7e755cb90b79"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"ff3a99632f2cf290619b96bf0516350bcf689c9e","unresolved":false,"context_lines":[{"line_number":104,"context_line":"is used, it is highly desirable to have more fine grained config to"},{"line_number":105,"context_line":"control emulator thread placement. When real-time or dedicated CPUs"},{"line_number":106,"context_line":"are used, it will be critical to separate emulator threads for"},{"line_number":107,"context_line":"different KVM instances."},{"line_number":108,"context_line":""},{"line_number":109,"context_line":"Another option is to hardcode an assumption that the vCPUs number set"},{"line_number":110,"context_line":"against the flavour implicitly includes 1 vCPUs for emulator. eg a"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a8bc5a2_63598d8e","line":107,"in_reply_to":"5a8bc5a2_4093a74f","updated":"2016-09-28 11:57:29.000000000","message":"Putting all the emulator threads on the same physical CPU is contrary to the user\u0027s request that the VM be given dedicated CPUs, as it means VMs will contend with each other. \n\nWe\u0027re trying to address the design limitation in real-time where you must reserve one or more vCPUs to run non-realtime, by allowing the emulator threads to be placed separately. This is a performance critical scenario, so we don\u0027t then want to make these emulator threads contend with other VMs by placing on a shared pCPU.","commit_id":"07518f1b504f993ed8073eeb6cff7e755cb90b79"},{"author":{"_account_id":7730,"name":"Sahid Orentino Ferdjaoui","email":"sahid.ferdjaoui@industrialdiscipline.com","username":"sahid"},"change_message_id":"900763d1dcf20b05fb9d40f5e9c40e5054fc601b","unresolved":false,"context_lines":[{"line_number":104,"context_line":"is used, it is highly desirable to have more fine grained config to"},{"line_number":105,"context_line":"control emulator thread placement. When real-time or dedicated CPUs"},{"line_number":106,"context_line":"are used, it will be critical to separate emulator threads for"},{"line_number":107,"context_line":"different KVM instances."},{"line_number":108,"context_line":""},{"line_number":109,"context_line":"Another option is to hardcode an assumption that the vCPUs number set"},{"line_number":110,"context_line":"against the flavour implicitly includes 1 vCPUs for emulator. eg a"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a8bc5a2_7148aecc","line":107,"in_reply_to":"5a8bc5a2_4ea60454","updated":"2016-09-27 15:07:51.000000000","message":"We can\u0027t really isolate emulator threads on a couple of host CPUs that would decrease performance for the vCPUs pinned on a different NUMA nodes.\n\nIn this implementation we will try to do the best-effort with that one host CPU used to isolate emulator threads, to make them running on the same host NUMA node than the guest vCPUs are pinned on.\n\nI think we can add a note to make predictable that if a guest is requesting two NUMA nodes so the pCPU used to isolate emulator threads would always be on node ID 0. So realtime users can configure their mask based on that truth.","commit_id":"07518f1b504f993ed8073eeb6cff7e755cb90b79"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"4ae8460b707347abb451017c72fef828c80e4ce3","unresolved":false,"context_lines":[{"line_number":104,"context_line":"is used, it is highly desirable to have more fine grained config to"},{"line_number":105,"context_line":"control emulator thread placement. When real-time or dedicated CPUs"},{"line_number":106,"context_line":"are used, it will be critical to separate emulator threads for"},{"line_number":107,"context_line":"different KVM instances."},{"line_number":108,"context_line":""},{"line_number":109,"context_line":"Another option is to hardcode an assumption that the vCPUs number set"},{"line_number":110,"context_line":"against the flavour implicitly includes 1 vCPUs for emulator. eg a"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a8bc5a2_4093a74f","line":107,"in_reply_to":"5a8bc5a2_7148aecc","updated":"2016-09-28 11:33:15.000000000","message":"\u003e We can\u0027t really isolate emulator threads on a couple of host CPUs that would decrease performance for the vCPUs pinned on a different NUMA nodes.\n \u003e In this implementation we will try to do the best-effort with that one host CPU used to isolate emulator threads, to make them running on the same host NUMA node than the guest vCPUs are pinned on.\n\nWouldn\u0027t it be as simple as making sure \u0027vcpu_pin_set\u0027 excludes at least one core from each node then?","commit_id":"07518f1b504f993ed8073eeb6cff7e755cb90b79"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"f5e7877c72afa66243971d84395ead3f340f2ab6","unresolved":false,"context_lines":[{"line_number":127,"context_line":""},{"line_number":128,"context_line":"The InstanceNUMATopology object will be extended to have a new field"},{"line_number":129,"context_line":""},{"line_number":130,"context_line":"* cpu_emulator_threads\u003dCPUEmulatorThreadsField()"},{"line_number":131,"context_line":""},{"line_number":132,"context_line":"This field will be implemented as an enum with two options:"},{"line_number":133,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a8bc5a2_0e25dcbf","line":130,"range":{"start_line":130,"start_character":2,"end_line":130,"end_character":48},"updated":"2016-09-26 10:36:54.000000000","message":"code","commit_id":"07518f1b504f993ed8073eeb6cff7e755cb90b79"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"f5e7877c72afa66243971d84395ead3f340f2ab6","unresolved":false,"context_lines":[{"line_number":168,"context_line":"---------------------"},{"line_number":169,"context_line":""},{"line_number":170,"context_line":"Deployers who want to use that new feature will have to configure"},{"line_number":171,"context_line":"thier flavor to use a dedicated cpu policy (hw:cpu_policy\u003ddedicated),"},{"line_number":172,"context_line":"in a same time set \u0027hw:cpu_emulator_threads\u0027 to \u0027isolate\u0027."},{"line_number":173,"context_line":""},{"line_number":174,"context_line":"Developer impact"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a8bc5a2_6e33c014","line":171,"range":{"start_line":171,"start_character":44,"end_line":171,"end_character":67},"updated":"2016-09-26 10:36:54.000000000","message":"format as code","commit_id":"07518f1b504f993ed8073eeb6cff7e755cb90b79"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"f5e7877c72afa66243971d84395ead3f340f2ab6","unresolved":false,"context_lines":[{"line_number":169,"context_line":""},{"line_number":170,"context_line":"Deployers who want to use that new feature will have to configure"},{"line_number":171,"context_line":"thier flavor to use a dedicated cpu policy (hw:cpu_policy\u003ddedicated),"},{"line_number":172,"context_line":"in a same time set \u0027hw:cpu_emulator_threads\u0027 to \u0027isolate\u0027."},{"line_number":173,"context_line":""},{"line_number":174,"context_line":"Developer impact"},{"line_number":175,"context_line":"----------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a8bc5a2_ae3948f4","line":172,"range":{"start_line":172,"start_character":19,"end_line":172,"end_character":44},"updated":"2016-09-26 10:36:54.000000000","message":"code","commit_id":"07518f1b504f993ed8073eeb6cff7e755cb90b79"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"f5e7877c72afa66243971d84395ead3f340f2ab6","unresolved":false,"context_lines":[{"line_number":169,"context_line":""},{"line_number":170,"context_line":"Deployers who want to use that new feature will have to configure"},{"line_number":171,"context_line":"thier flavor to use a dedicated cpu policy (hw:cpu_policy\u003ddedicated),"},{"line_number":172,"context_line":"in a same time set \u0027hw:cpu_emulator_threads\u0027 to \u0027isolate\u0027."},{"line_number":173,"context_line":""},{"line_number":174,"context_line":"Developer impact"},{"line_number":175,"context_line":"----------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a8bc5a2_ce3e94d9","line":172,"range":{"start_line":172,"start_character":49,"end_line":172,"end_character":57},"updated":"2016-09-26 10:36:54.000000000","message":"code","commit_id":"07518f1b504f993ed8073eeb6cff7e755cb90b79"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"f5e7877c72afa66243971d84395ead3f340f2ab6","unresolved":false,"context_lines":[{"line_number":213,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":214,"context_line":""},{"line_number":215,"context_line":"This can be tested in any CI system that is capable of testing the current"},{"line_number":216,"context_line":"NUMA and dedicated CPUs policy. ie, it requires ability to use KVM, not"},{"line_number":217,"context_line":"merely QEMU."},{"line_number":218,"context_line":""},{"line_number":219,"context_line":"Documentation Impact"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a8bc5a2_ce477475","line":216,"range":{"start_line":216,"start_character":30,"end_line":216,"end_character":35},"updated":"2016-09-26 10:36:54.000000000","message":", i.e. it","commit_id":"07518f1b504f993ed8073eeb6cff7e755cb90b79"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"f5e7877c72afa66243971d84395ead3f340f2ab6","unresolved":false,"context_lines":[{"line_number":213,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":214,"context_line":""},{"line_number":215,"context_line":"This can be tested in any CI system that is capable of testing the current"},{"line_number":216,"context_line":"NUMA and dedicated CPUs policy. ie, it requires ability to use KVM, not"},{"line_number":217,"context_line":"merely QEMU."},{"line_number":218,"context_line":""},{"line_number":219,"context_line":"Documentation Impact"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a8bc5a2_ee60b003","line":216,"range":{"start_line":216,"start_character":66,"end_line":216,"end_character":67},"updated":"2016-09-26 10:36:54.000000000","message":"and","commit_id":"07518f1b504f993ed8073eeb6cff7e755cb90b79"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"f5e7877c72afa66243971d84395ead3f340f2ab6","unresolved":false,"context_lines":[{"line_number":228,"context_line":"History"},{"line_number":229,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":230,"context_line":""},{"line_number":231,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a8bc5a2_4e544420","line":231,"updated":"2016-09-26 10:36:54.000000000","message":"Is this missing?","commit_id":"07518f1b504f993ed8073eeb6cff7e755cb90b79"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"26f7bb1d0e347f3dc7b4e807551aab0bce5a3137","unresolved":false,"context_lines":[{"line_number":47,"context_line":"overcommit is being used. However, when guests want dedicated vCPU"},{"line_number":48,"context_line":"allocation though, there is a desire to be able to express other"},{"line_number":49,"context_line":"placement policies, for example, to allocate one or more pCPUs to be"},{"line_number":50,"context_line":"dedicated to a guest\u0027s emulator threads. This becomes even as Nova"},{"line_number":51,"context_line":"continues to implement support for real-time workloads, as it will not"},{"line_number":52,"context_line":"be acceptable to allow emulator threads to steal time from real-time"},{"line_number":53,"context_line":"vCPUs."}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a95cdbc_dc71a50d","line":50,"updated":"2016-10-10 13:21:09.000000000","message":"s/This becomes even as Nova/This becomes critical as Nova/","commit_id":"d178fe6cfa3de253b2ee04b38e2ab296285d7807"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"26f7bb1d0e347f3dc7b4e807551aab0bce5a3137","unresolved":false,"context_lines":[{"line_number":66,"context_line":"with real-time scheduler policy, will want to configure guests such"},{"line_number":67,"context_line":"that emulator threads are placed separately from vCPUs. This will be"},{"line_number":68,"context_line":"desired by the NFV community, as well as the other user groups"},{"line_number":69,"context_line":"described for the real-time spec."},{"line_number":70,"context_line":""},{"line_number":71,"context_line":"Project Priority"},{"line_number":72,"context_line":"----------------"}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a95cdbc_f004c515","line":69,"updated":"2016-10-10 13:21:09.000000000","message":"We should probably spell out the problem with real time explicitly to make it clear why we need this feature. eg\n\n\"With current Nova real time support in libvirt, there is a requirement to reserve one vCPU for running non-realtime workloads. The QEMU emulator threads are pinned to run on the same host pCPU as this vCPU. While this requirement is just about acceptable for Linux guests, it prevents use of Nova to run other real time operating systems which require realtime response for all vCPUs. To broaden the realtime support it is necessary to pin emulator threads separately from vCPUs, which requires that the schedular be able to account for extra pCPU usage per guest.\"","commit_id":"d178fe6cfa3de253b2ee04b38e2ab296285d7807"},{"author":{"_account_id":8802,"name":"Vladik Romanovsky","email":"vromanso@redhat.com","username":"vladikr"},"change_message_id":"12906c93259933f9c7fc3974b4fa826f83cc231b","unresolved":false,"context_lines":[{"line_number":87,"context_line":"during claim.  An user which expresses the desire to isolate emulator"},{"line_number":88,"context_line":"threads must use a flavor configured to accept that specification:"},{"line_number":89,"context_line":""},{"line_number":90,"context_line":"* hw:cpu_emulator_threads\u003dshared|isolate"},{"line_number":91,"context_line":""},{"line_number":92,"context_line":"Would say that this instance is to be considered to consumes 1 host"},{"line_number":93,"context_line":"CPU. That pCPU used to make running emulator threads is going to"}],"source_content_type":"text/x-rst","patch_set":5,"id":"3a98d1f4_86f16b98","line":90,"range":{"start_line":90,"start_character":26,"end_line":90,"end_character":32},"updated":"2016-10-04 14:06:58.000000000","message":"Maybe it\u0027ll be better to rename cpu_emulator_threads to anything that emphasises whether the feature is enabled or disabled. I just think that seeing \"cpu_emulator_threads\u003dshared\" is a bit misleading","commit_id":"d178fe6cfa3de253b2ee04b38e2ab296285d7807"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"26f7bb1d0e347f3dc7b4e807551aab0bce5a3137","unresolved":false,"context_lines":[{"line_number":87,"context_line":"during claim.  An user which expresses the desire to isolate emulator"},{"line_number":88,"context_line":"threads must use a flavor configured to accept that specification:"},{"line_number":89,"context_line":""},{"line_number":90,"context_line":"* hw:cpu_emulator_threads\u003dshared|isolate"},{"line_number":91,"context_line":""},{"line_number":92,"context_line":"Would say that this instance is to be considered to consumes 1 host"},{"line_number":93,"context_line":"CPU. That pCPU used to make running emulator threads is going to"}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a95cdbc_703fd5b3","line":90,"range":{"start_line":90,"start_character":26,"end_line":90,"end_character":32},"in_reply_to":"3a98d1f4_86f16b98","updated":"2016-10-10 13:21:09.000000000","message":"Emulator threads are always enabled - what we\u0027re changing here is merely how we place them, so I think this proposal of sahid\u0027s is correct as is.","commit_id":"d178fe6cfa3de253b2ee04b38e2ab296285d7807"},{"author":{"_account_id":7730,"name":"Sahid Orentino Ferdjaoui","email":"sahid.ferdjaoui@industrialdiscipline.com","username":"sahid"},"change_message_id":"933e3280f53efe84b83cd4cc7743809f74ea374f","unresolved":false,"context_lines":[{"line_number":87,"context_line":"during claim.  An user which expresses the desire to isolate emulator"},{"line_number":88,"context_line":"threads must use a flavor configured to accept that specification:"},{"line_number":89,"context_line":""},{"line_number":90,"context_line":"* hw:cpu_emulator_threads\u003dshared|isolate"},{"line_number":91,"context_line":""},{"line_number":92,"context_line":"Would say that this instance is to be considered to consumes 1 host"},{"line_number":93,"context_line":"CPU. That pCPU used to make running emulator threads is going to"}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a95cdbc_e4b5d07e","line":90,"in_reply_to":"3a98d1f4_86f16b98","updated":"2016-10-04 14:46:33.000000000","message":"Yeah, I just tried to stay coherent with what we do with other options","commit_id":"d178fe6cfa3de253b2ee04b38e2ab296285d7807"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"26f7bb1d0e347f3dc7b4e807551aab0bce5a3137","unresolved":false,"context_lines":[{"line_number":160,"context_line":"Since we are in a context of a CPU policy dedicated, an user starting"},{"line_number":161,"context_line":"an instance from a flavor providing the option"},{"line_number":162,"context_line":"`hw:cpu_emulator_threads` set to `isolate` is going to consume as host"},{"line_number":163,"context_line":"CPUs the number of vCPUs + 1."},{"line_number":164,"context_line":""},{"line_number":165,"context_line":"Performance Impact"},{"line_number":166,"context_line":"------------------"}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a95cdbc_50b2d94c","line":163,"updated":"2016-10-10 13:21:09.000000000","message":"This is probably something for deployer impact - users don\u0027t really care what host CPUs are consumed - they\u0027re just getting a guest with nVCPUs.","commit_id":"d178fe6cfa3de253b2ee04b38e2ab296285d7807"},{"author":{"_account_id":7730,"name":"Sahid Orentino Ferdjaoui","email":"sahid.ferdjaoui@industrialdiscipline.com","username":"sahid"},"change_message_id":"5cc1d5e6c6108264bd52e603c29fec2dedf7a070","unresolved":false,"context_lines":[{"line_number":160,"context_line":"Since we are in a context of a CPU policy dedicated, an user starting"},{"line_number":161,"context_line":"an instance from a flavor providing the option"},{"line_number":162,"context_line":"`hw:cpu_emulator_threads` set to `isolate` is going to consume as host"},{"line_number":163,"context_line":"CPUs the number of vCPUs + 1."},{"line_number":164,"context_line":""},{"line_number":165,"context_line":"Performance Impact"},{"line_number":166,"context_line":"------------------"}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a95cdbc_95928bf4","line":163,"in_reply_to":"1a95cdbc_50b2d94c","updated":"2016-10-10 15:29:04.000000000","message":"Yes I basically repeating same thing in \"Deployer impact\"","commit_id":"d178fe6cfa3de253b2ee04b38e2ab296285d7807"},{"author":{"_account_id":6772,"name":"Stephen Gordon","email":"sgordon@redhat.com","username":"sgordon"},"change_message_id":"9b940a7e779bb038a34b1161fe4c8b97039c9ce9","unresolved":false,"context_lines":[{"line_number":87,"context_line":"Each hypervisor has a different architecture, for example QEMU has"},{"line_number":88,"context_line":"emulator threads, while Xen has stub-domains. To avoid favouring any"},{"line_number":89,"context_line":"specific implementation, the idea is to extend"},{"line_number":90,"context_line":"`estimate_instance_overhead` to return 1 host CPU to take into account"},{"line_number":91,"context_line":"during claim.  An user which expresses the desire to isolate emulator"},{"line_number":92,"context_line":"threads must use a flavor configured to accept that specification:"},{"line_number":93,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"fa6399be_b8b1a003","line":90,"range":{"start_line":90,"start_character":39,"end_line":90,"end_character":40},"updated":"2016-10-20 18:01:23.000000000","message":"I would suggest saying 1 additional host CPU or something of that ilk to explicitly state that this is in addition to those used for the guest\u0027s vCPU threads.\n\nOne other thought, my guest spans multiple NUMA nodes, do we still reserve one core for emulator threads or one core per NUMA node? How do we decide which node in the former case?","commit_id":"285bbbdc471b76d0ed1ed7099d6fc4828accce4e"},{"author":{"_account_id":6772,"name":"Stephen Gordon","email":"sgordon@redhat.com","username":"sgordon"},"change_message_id":"c1e23738edde827497f34104bfe8563e6ee61bcd","unresolved":false,"context_lines":[{"line_number":87,"context_line":"Each hypervisor has a different architecture, for example QEMU has"},{"line_number":88,"context_line":"emulator threads, while Xen has stub-domains. To avoid favouring any"},{"line_number":89,"context_line":"specific implementation, the idea is to extend"},{"line_number":90,"context_line":"`estimate_instance_overhead` to return 1 host CPU to take into account"},{"line_number":91,"context_line":"during claim.  An user which expresses the desire to isolate emulator"},{"line_number":92,"context_line":"threads must use a flavor configured to accept that specification:"},{"line_number":93,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"da6895a0_61b425f2","line":90,"in_reply_to":"da6895a0_132671ba","updated":"2016-10-21 17:14:54.000000000","message":"Right, what I\u0027m getting at is this is not immediately clear from the text - in particular that it is an *additional* claim.","commit_id":"285bbbdc471b76d0ed1ed7099d6fc4828accce4e"},{"author":{"_account_id":7730,"name":"Sahid Orentino Ferdjaoui","email":"sahid.ferdjaoui@industrialdiscipline.com","username":"sahid"},"change_message_id":"cfe953c99d04155a70deba4109586d3adcefb4a0","unresolved":false,"context_lines":[{"line_number":87,"context_line":"Each hypervisor has a different architecture, for example QEMU has"},{"line_number":88,"context_line":"emulator threads, while Xen has stub-domains. To avoid favouring any"},{"line_number":89,"context_line":"specific implementation, the idea is to extend"},{"line_number":90,"context_line":"`estimate_instance_overhead` to return 1 host CPU to take into account"},{"line_number":91,"context_line":"during claim.  An user which expresses the desire to isolate emulator"},{"line_number":92,"context_line":"threads must use a flavor configured to accept that specification:"},{"line_number":93,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"da6895a0_132671ba","line":90,"in_reply_to":"fa6399be_b8b1a003","updated":"2016-10-21 10:09:29.000000000","message":"Currently the spec reserves one host CPU on the same host NUMA node than the guest NUMA node 0 is related.\n\nSo if a guest has 2 NUMA nodes with this topology:\n\n  guest_numa_0 \u003d host_numa_3\n  guest_numa_1 \u003d host_numa_8\n\nSo the additional host CPU reserved will be from host_numa_3.","commit_id":"285bbbdc471b76d0ed1ed7099d6fc4828accce4e"},{"author":{"_account_id":6772,"name":"Stephen Gordon","email":"sgordon@redhat.com","username":"sgordon"},"change_message_id":"9b940a7e779bb038a34b1161fe4c8b97039c9ce9","unresolved":false,"context_lines":[{"line_number":93,"context_line":""},{"line_number":94,"context_line":"* hw:cpu_emulator_threads\u003dshared|isolate"},{"line_number":95,"context_line":""},{"line_number":96,"context_line":"Would say that this instance is to be considered to consumes 1 host"},{"line_number":97,"context_line":"CPU. That pCPU used to make running emulator threads is going to"},{"line_number":98,"context_line":"always be configured on the related guest NUMA node ID 0, to make it"},{"line_number":99,"context_line":"predictable for users."},{"line_number":100,"context_line":"Currently there is no desire to make customizable the number of host"}],"source_content_type":"text/x-rst","patch_set":6,"id":"fa6399be_1b1b981c","line":97,"range":{"start_line":96,"start_character":0,"end_line":97,"end_character":4},"updated":"2016-10-20 18:01:23.000000000","message":"Not clear to me what this sentence refers to, shared or isolate scenario (or both)?","commit_id":"285bbbdc471b76d0ed1ed7099d6fc4828accce4e"},{"author":{"_account_id":7730,"name":"Sahid Orentino Ferdjaoui","email":"sahid.ferdjaoui@industrialdiscipline.com","username":"sahid"},"change_message_id":"cfe953c99d04155a70deba4109586d3adcefb4a0","unresolved":false,"context_lines":[{"line_number":94,"context_line":"* hw:cpu_emulator_threads\u003dshared|isolate"},{"line_number":95,"context_line":""},{"line_number":96,"context_line":"Would say that this instance is to be considered to consumes 1 host"},{"line_number":97,"context_line":"CPU. That pCPU used to make running emulator threads is going to"},{"line_number":98,"context_line":"always be configured on the related guest NUMA node ID 0, to make it"},{"line_number":99,"context_line":"predictable for users."},{"line_number":100,"context_line":"Currently there is no desire to make customizable the number of host"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da6895a0_30f36b75","line":97,"in_reply_to":"fa6399be_1b1b981c","updated":"2016-10-21 10:09:29.000000000","message":"Yes, it refers to \"isolate, I will update that point.","commit_id":"285bbbdc471b76d0ed1ed7099d6fc4828accce4e"},{"author":{"_account_id":6772,"name":"Stephen Gordon","email":"sgordon@redhat.com","username":"sgordon"},"change_message_id":"9b940a7e779bb038a34b1161fe4c8b97039c9ce9","unresolved":false,"context_lines":[{"line_number":97,"context_line":"CPU. That pCPU used to make running emulator threads is going to"},{"line_number":98,"context_line":"always be configured on the related guest NUMA node ID 0, to make it"},{"line_number":99,"context_line":"predictable for users."},{"line_number":100,"context_line":"Currently there is no desire to make customizable the number of host"},{"line_number":101,"context_line":"CPUs running emulator threads since only one should work for almost"},{"line_number":102,"context_line":"every usecases."},{"line_number":103,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"fa6399be_98a77cbc","line":100,"range":{"start_line":100,"start_character":22,"end_line":100,"end_character":28},"updated":"2016-10-20 18:01:23.000000000","message":"Although this has been re-hashed in similar specification reviews in the past, it might be worth explicitly outlining why.","commit_id":"285bbbdc471b76d0ed1ed7099d6fc4828accce4e"},{"author":{"_account_id":6772,"name":"Stephen Gordon","email":"sgordon@redhat.com","username":"sgordon"},"change_message_id":"9b940a7e779bb038a34b1161fe4c8b97039c9ce9","unresolved":false,"context_lines":[{"line_number":137,"context_line":""},{"line_number":138,"context_line":"This field will be implemented as an enum with two options:"},{"line_number":139,"context_line":""},{"line_number":140,"context_line":"* shared - The emulator threads float across the pCPUs asscoaited to"},{"line_number":141,"context_line":"  the guest."},{"line_number":142,"context_line":"* isolate - The emulator threads are isolated on a single pCPU."},{"line_number":143,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"fa6399be_58a88477","line":140,"range":{"start_line":140,"start_character":32,"end_line":140,"end_character":37},"updated":"2016-10-20 18:01:23.000000000","message":"Probably worth noting that if you use isolcpus they will actually get stuck to the same core as vCPU 0?","commit_id":"285bbbdc471b76d0ed1ed7099d6fc4828accce4e"},{"author":{"_account_id":6772,"name":"Stephen Gordon","email":"sgordon@redhat.com","username":"sgordon"},"change_message_id":"9b940a7e779bb038a34b1161fe4c8b97039c9ce9","unresolved":false,"context_lines":[{"line_number":137,"context_line":""},{"line_number":138,"context_line":"This field will be implemented as an enum with two options:"},{"line_number":139,"context_line":""},{"line_number":140,"context_line":"* shared - The emulator threads float across the pCPUs asscoaited to"},{"line_number":141,"context_line":"  the guest."},{"line_number":142,"context_line":"* isolate - The emulator threads are isolated on a single pCPU."},{"line_number":143,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"fa6399be_bb12ac07","line":140,"range":{"start_line":140,"start_character":55,"end_line":140,"end_character":65},"updated":"2016-10-20 18:01:23.000000000","message":"associated","commit_id":"285bbbdc471b76d0ed1ed7099d6fc4828accce4e"},{"author":{"_account_id":6772,"name":"Stephen Gordon","email":"sgordon@redhat.com","username":"sgordon"},"change_message_id":"c1e23738edde827497f34104bfe8563e6ee61bcd","unresolved":false,"context_lines":[{"line_number":137,"context_line":""},{"line_number":138,"context_line":"This field will be implemented as an enum with two options:"},{"line_number":139,"context_line":""},{"line_number":140,"context_line":"* shared - The emulator threads float across the pCPUs asscoaited to"},{"line_number":141,"context_line":"  the guest."},{"line_number":142,"context_line":"* isolate - The emulator threads are isolated on a single pCPU."},{"line_number":143,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"da6895a0_81ecd9c8","line":140,"in_reply_to":"da6895a0_9368c1bc","updated":"2016-10-21 17:14:54.000000000","message":"Yeah, the current behaviour is impacted by a kernel commit [1] that makes isolcpus work the way it was (apparently) always supposed to work.\n\nThe impact of this for Nova is if you follow the instructions for creating a guest with CPU pinning [2] - which include the use of isolcpus the emulator threads do not actually float as advertized but instead end up on the pCPU with vCPU 0 on it - always. The workaround is to use Systemd\u0027s CPUAffinity setting instead of isolcpus for non-RT workloads (for RT workloads the use of another core entirely or the pCPU aligned with vCPU 0 is actually expected behaviour) albeit this does not constrain kernel processes.\n\nHence I think it\u0027s important to note that the ability to \"float\" the emulator threads is somewhat constrained by some external factors, and what will happen if these factors (isolcpus) are present.\n\n[1] https://kernel.googlesource.com/pub/scm/linux/kernel/git/stable/linux-stable/+/47b8ea7186aae7f474ec4c98f43eaa8da719cd83\n[2] http://redhatstackblog.redhat.com/2015/05/05/cpu-pinning-and-numa-topology-awareness-in-openstack-compute/","commit_id":"285bbbdc471b76d0ed1ed7099d6fc4828accce4e"},{"author":{"_account_id":7730,"name":"Sahid Orentino Ferdjaoui","email":"sahid.ferdjaoui@industrialdiscipline.com","username":"sahid"},"change_message_id":"cfe953c99d04155a70deba4109586d3adcefb4a0","unresolved":false,"context_lines":[{"line_number":137,"context_line":""},{"line_number":138,"context_line":"This field will be implemented as an enum with two options:"},{"line_number":139,"context_line":""},{"line_number":140,"context_line":"* shared - The emulator threads float across the pCPUs asscoaited to"},{"line_number":141,"context_line":"  the guest."},{"line_number":142,"context_line":"* isolate - The emulator threads are isolated on a single pCPU."},{"line_number":143,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"da6895a0_9368c1bc","line":140,"in_reply_to":"fa6399be_58a88477","updated":"2016-10-21 10:09:29.000000000","message":"Not sure to understand that point, \u0027shared\u0027 is the current behavior. Can you explain me more on why emulator threads are going to stuck on the same vCPU 0 when using isolcpus ?","commit_id":"285bbbdc471b76d0ed1ed7099d6fc4828accce4e"},{"author":{"_account_id":6772,"name":"Stephen Gordon","email":"sgordon@redhat.com","username":"sgordon"},"change_message_id":"9b940a7e779bb038a34b1161fe4c8b97039c9ce9","unresolved":false,"context_lines":[{"line_number":161,"context_line":"Other end user impact"},{"line_number":162,"context_line":"---------------------"},{"line_number":163,"context_line":""},{"line_number":164,"context_line":"None"},{"line_number":165,"context_line":""},{"line_number":166,"context_line":"Performance Impact"},{"line_number":167,"context_line":"------------------"}],"source_content_type":"text/x-rst","patch_set":6,"id":"fa6399be_78054878","line":164,"range":{"start_line":164,"start_character":0,"end_line":164,"end_character":4},"updated":"2016-10-20 18:01:23.000000000","message":"The feature is obviously opt-in, but should this section explicitly highlight that if they opt in their guest will take +1 CPU quota? There is probably a question of how this is reflected in the API/CLI to address as well. Will this just be a \"phantom\" CPU taken from their quota?\n\nE.g. my operator configures a flavor with isolate set. When I\u0027m launching that does it show that it needs 4 vCPUs per the flavor, or 5 vCPUs per the flavor + the one I need for emulator threads.\n\nI guess we can say the user can extrapolate this from extra_specs listed but that seems a bit messy from a UX perspective?","commit_id":"285bbbdc471b76d0ed1ed7099d6fc4828accce4e"},{"author":{"_account_id":7730,"name":"Sahid Orentino Ferdjaoui","email":"sahid.ferdjaoui@industrialdiscipline.com","username":"sahid"},"change_message_id":"cfe953c99d04155a70deba4109586d3adcefb4a0","unresolved":false,"context_lines":[{"line_number":161,"context_line":"Other end user impact"},{"line_number":162,"context_line":"---------------------"},{"line_number":163,"context_line":""},{"line_number":164,"context_line":"None"},{"line_number":165,"context_line":""},{"line_number":166,"context_line":"Performance Impact"},{"line_number":167,"context_line":"------------------"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da6895a0_73cfadb0","line":164,"in_reply_to":"fa6399be_78054878","updated":"2016-10-21 10:09:29.000000000","message":"Yes, it makes sense to clearly explain that a flavor configured with isolate is going to consume the number of (vCPUs + 1) host CPUs.","commit_id":"285bbbdc471b76d0ed1ed7099d6fc4828accce4e"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"45f946e32670059c066ad4e39cc9127681e1a1f2","unresolved":false,"context_lines":[{"line_number":13,"context_line":"The Nova scheduler determines CPU resource utilization and instance"},{"line_number":14,"context_line":"CPU placement based on the number of vCPUs in the flavor. A number of"},{"line_number":15,"context_line":"hypervisors have operations that are being performed on behalf of the"},{"line_number":16,"context_line":"guest instance in the host OS. These operations should be acounted and"},{"line_number":17,"context_line":"scheduled separately, as well as have their own placement policy"},{"line_number":18,"context_line":"controls applied."},{"line_number":19,"context_line":""}],"source_content_type":"text/x-rst","patch_set":7,"id":"ba5da102_2dc95bad","line":16,"updated":"2016-11-04 15:53:13.000000000","message":"s/acounted/accounted for/","commit_id":"aaa5bff651531e40b3af34c12097c27f6e7decec"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"45f946e32670059c066ad4e39cc9127681e1a1f2","unresolved":false,"context_lines":[{"line_number":37,"context_line":""},{"line_number":38,"context_line":"Nova does not have any current mechanism to either track this extra"},{"line_number":39,"context_line":"guest instance compute requirement in order to measure utilization,"},{"line_number":40,"context_line":"nor to place any control over its execution policy."},{"line_number":41,"context_line":""},{"line_number":42,"context_line":"The libvirt driver has implemented a generic placement policy for KVM"},{"line_number":43,"context_line":"whereby the QEMU emulator threads are allowed to float across the same"}],"source_content_type":"text/x-rst","patch_set":7,"id":"ba5da102_cdd7ffeb","line":40,"updated":"2016-11-04 15:53:13.000000000","message":"Right. The closest thing we have to something like this right now is the \"workload\" stats number which counts operations like migrations that may impact a host\u0027s performance relative to normal state.","commit_id":"aaa5bff651531e40b3af34c12097c27f6e7decec"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"45f946e32670059c066ad4e39cc9127681e1a1f2","unresolved":false,"context_lines":[{"line_number":88,"context_line":"emulator threads, while Xen has stub-domains. To avoid favouring any"},{"line_number":89,"context_line":"specific implementation, the idea is to extend"},{"line_number":90,"context_line":"`estimate_instance_overhead` to return 1 additional host CPU to take"},{"line_number":91,"context_line":"into account during claim. An user which expresses the desire to"},{"line_number":92,"context_line":"isolate emulator threads must use a flavor configured to accept that"},{"line_number":93,"context_line":"specification as:"},{"line_number":94,"context_line":""}],"source_content_type":"text/x-rst","patch_set":7,"id":"ba5da102_1062eaad","line":91,"updated":"2016-11-04 15:53:13.000000000","message":"a/An user/A user/","commit_id":"aaa5bff651531e40b3af34c12097c27f6e7decec"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"45f946e32670059c066ad4e39cc9127681e1a1f2","unresolved":false,"context_lines":[{"line_number":94,"context_line":""},{"line_number":95,"context_line":"* hw:cpu_emulator_threads\u003disolate"},{"line_number":96,"context_line":""},{"line_number":97,"context_line":"Would say that this instance is to be considered to consumes 1"},{"line_number":98,"context_line":"additional host CPU. That pCPU used to make running emulator threads"},{"line_number":99,"context_line":"is going to always be configured on the related guest NUMA node ID 0,"},{"line_number":100,"context_line":"to make it predictable for users. Currently there is no desire to make"}],"source_content_type":"text/x-rst","patch_set":7,"id":"ba5da102_50af0294","line":97,"updated":"2016-11-04 15:53:13.000000000","message":"s/consumes/consume/","commit_id":"aaa5bff651531e40b3af34c12097c27f6e7decec"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"762449e9561441572a2114875487c21d902c5b19","unresolved":false,"context_lines":[{"line_number":96,"context_line":""},{"line_number":97,"context_line":"Would say that this instance is to be considered to consumes 1"},{"line_number":98,"context_line":"additional host CPU. That pCPU used to make running emulator threads"},{"line_number":99,"context_line":"is going to always be configured on the related guest NUMA node ID 0,"},{"line_number":100,"context_line":"to make it predictable for users. Currently there is no desire to make"},{"line_number":101,"context_line":"customizable the number of host CPUs running emulator threads since"},{"line_number":102,"context_line":"only one should work for almost every usecases. If in the future a"},{"line_number":103,"context_line":"desire to isolate more than one host CPU to run emulator threads comes"}],"source_content_type":"text/x-rst","patch_set":7,"id":"ba5da102_799231a1","line":100,"range":{"start_line":99,"start_character":48,"end_line":100,"end_character":32},"updated":"2016-11-04 16:59:29.000000000","message":"This confused my slightly, but its just about picking which physical CPU to put the emulator thread. Totally wants to be the same NUMA node, as the guest, for memory locality. If we have to respin, it might be good to make that clearer.","commit_id":"aaa5bff651531e40b3af34c12097c27f6e7decec"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"45f946e32670059c066ad4e39cc9127681e1a1f2","unresolved":false,"context_lines":[{"line_number":99,"context_line":"is going to always be configured on the related guest NUMA node ID 0,"},{"line_number":100,"context_line":"to make it predictable for users. Currently there is no desire to make"},{"line_number":101,"context_line":"customizable the number of host CPUs running emulator threads since"},{"line_number":102,"context_line":"only one should work for almost every usecases. If in the future a"},{"line_number":103,"context_line":"desire to isolate more than one host CPU to run emulator threads comes"},{"line_number":104,"context_line":"so we would implement instead I/O threads to add granularity on"},{"line_number":105,"context_line":"dedicating used resources to run guests on host CPUs."}],"source_content_type":"text/x-rst","patch_set":7,"id":"ba5da102_10baeac0","line":102,"updated":"2016-11-04 15:53:13.000000000","message":"s/usecases/use case/\ns/in the future a/in the future there is a/","commit_id":"aaa5bff651531e40b3af34c12097c27f6e7decec"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"45f946e32670059c066ad4e39cc9127681e1a1f2","unresolved":false,"context_lines":[{"line_number":100,"context_line":"to make it predictable for users. Currently there is no desire to make"},{"line_number":101,"context_line":"customizable the number of host CPUs running emulator threads since"},{"line_number":102,"context_line":"only one should work for almost every usecases. If in the future a"},{"line_number":103,"context_line":"desire to isolate more than one host CPU to run emulator threads comes"},{"line_number":104,"context_line":"so we would implement instead I/O threads to add granularity on"},{"line_number":105,"context_line":"dedicating used resources to run guests on host CPUs."},{"line_number":106,"context_line":""}],"source_content_type":"text/x-rst","patch_set":7,"id":"ba5da102_30fd2e6f","line":103,"range":{"start_line":103,"start_character":60,"end_line":103,"end_character":61},"updated":"2016-11-04 15:53:13.000000000","message":"s/threads comes so we/threads, we/","commit_id":"aaa5bff651531e40b3af34c12097c27f6e7decec"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"45f946e32670059c066ad4e39cc9127681e1a1f2","unresolved":false,"context_lines":[{"line_number":111,"context_line":"for running emulator threads globally, instead of trying to account"},{"line_number":112,"context_line":"for it per instance. This would work in the simple case, but when NUMA"},{"line_number":113,"context_line":"is used, it is highly desirable to have more fine grained config to"},{"line_number":114,"context_line":"control emulator thread placement. When real-time or dedicated CPUs"},{"line_number":115,"context_line":"are used, it will be critical to separate emulator threads for"},{"line_number":116,"context_line":"different KVM instances."},{"line_number":117,"context_line":""}],"source_content_type":"text/x-rst","patch_set":7,"id":"ba5da102_50ec628f","line":114,"updated":"2016-11-04 15:53:13.000000000","message":"\"when NUMA is used, it is highly desirable to have more fine-grained config to control emulator thread placement\"\n\nBut the section above states that \"pCPU used to make running emulator threads is going to always be configured on the related guest NUMA node ID 0\".\n\nThe two statements seem contradictory, no?","commit_id":"aaa5bff651531e40b3af34c12097c27f6e7decec"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"4564ee393894af07684b4cf6da8517e754431733","unresolved":false,"context_lines":[{"line_number":111,"context_line":"for running emulator threads globally, instead of trying to account"},{"line_number":112,"context_line":"for it per instance. This would work in the simple case, but when NUMA"},{"line_number":113,"context_line":"is used, it is highly desirable to have more fine grained config to"},{"line_number":114,"context_line":"control emulator thread placement. When real-time or dedicated CPUs"},{"line_number":115,"context_line":"are used, it will be critical to separate emulator threads for"},{"line_number":116,"context_line":"different KVM instances."},{"line_number":117,"context_line":""}],"source_content_type":"text/x-rst","patch_set":7,"id":"ba5da102_646c4837","line":114,"in_reply_to":"ba5da102_50ec628f","updated":"2016-11-04 15:58:21.000000000","message":"This \"alternative\" is describing a scenario where the emulator threads for every guest are always placed on a pCPU on *host* NUMA node 0.\n\nThe propsoed solution described earlier is saying ti place the emulator threads on the host NUMA node associated with *guest* NUMA node 0.\n\nie if guest NUMA node 0 is running on host NUMA node 3, the then the emualator threads would go on a pCPU in host NUMA node 3.","commit_id":"aaa5bff651531e40b3af34c12097c27f6e7decec"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"84b2c0c7223c43c7d1ef8e68e946c87718980c04","unresolved":false,"context_lines":[{"line_number":111,"context_line":"for running emulator threads globally, instead of trying to account"},{"line_number":112,"context_line":"for it per instance. This would work in the simple case, but when NUMA"},{"line_number":113,"context_line":"is used, it is highly desirable to have more fine grained config to"},{"line_number":114,"context_line":"control emulator thread placement. When real-time or dedicated CPUs"},{"line_number":115,"context_line":"are used, it will be critical to separate emulator threads for"},{"line_number":116,"context_line":"different KVM instances."},{"line_number":117,"context_line":""}],"source_content_type":"text/x-rst","patch_set":7,"id":"ba5da102_ff854159","line":114,"in_reply_to":"ba5da102_646c4837","updated":"2016-11-04 16:05:26.000000000","message":"Ah, sorry I missed that subtlelty, thanks for the explanation Dan :)","commit_id":"aaa5bff651531e40b3af34c12097c27f6e7decec"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"762449e9561441572a2114875487c21d902c5b19","unresolved":false,"context_lines":[{"line_number":113,"context_line":"is used, it is highly desirable to have more fine grained config to"},{"line_number":114,"context_line":"control emulator thread placement. When real-time or dedicated CPUs"},{"line_number":115,"context_line":"are used, it will be critical to separate emulator threads for"},{"line_number":116,"context_line":"different KVM instances."},{"line_number":117,"context_line":""},{"line_number":118,"context_line":"Another option is to hardcode an assumption that the vCPUs number set"},{"line_number":119,"context_line":"against the flavour implicitly includes 1 vCPUs for emulator. eg a"}],"source_content_type":"text/x-rst","patch_set":7,"id":"ba5da102_a19b1d17","line":116,"updated":"2016-11-04 16:59:29.000000000","message":"Right, for Xen without stub domains, you would pin the Dom0 vCPUs then use CONF.vcpu_pin_set to stop other guests wondering onto those CPUs.\n\nIt feels a bit like one pCPU for each VM is too much, but for real time, you want emulator independence, so I guess it makes sense.","commit_id":"aaa5bff651531e40b3af34c12097c27f6e7decec"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"45f946e32670059c066ad4e39cc9127681e1a1f2","unresolved":false,"context_lines":[{"line_number":129,"context_line":"with realtime by setting a fixed policy that the emulator threads only"},{"line_number":130,"context_line":"run on the vCPUs that have non-realtime policy. This requires that all"},{"line_number":131,"context_line":"guest OS using realtime are SMP, but some guest OS want realtime, but"},{"line_number":132,"context_line":"are only UP."},{"line_number":133,"context_line":""},{"line_number":134,"context_line":"Data model impact"},{"line_number":135,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":7,"id":"ba5da102_f372e0af","line":132,"updated":"2016-11-04 15:53:13.000000000","message":"Another option would be to introduce a new resource class called HOST_EMULATOR_CPU (or PCPU, though that might be confused as indicating a physical CPU socket/core) and have the flavor request 4 VCPU and 1 HOST_EMULATOR_CPU. The compute nodes would have an amount of HOST_EMULATOR_CPU inventory that would correspond to a number of physical cores that are \"set aside\" for pinning emulator threads to. We would thus handle the emulator CPU resources in the exact same way as we handle any other type of quantitative resource in the system.","commit_id":"aaa5bff651531e40b3af34c12097c27f6e7decec"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"d7dbf674d3e5e3110e1ffd56d73104d1960d5009","unresolved":false,"context_lines":[{"line_number":129,"context_line":"with realtime by setting a fixed policy that the emulator threads only"},{"line_number":130,"context_line":"run on the vCPUs that have non-realtime policy. This requires that all"},{"line_number":131,"context_line":"guest OS using realtime are SMP, but some guest OS want realtime, but"},{"line_number":132,"context_line":"are only UP."},{"line_number":133,"context_line":""},{"line_number":134,"context_line":"Data model impact"},{"line_number":135,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":7,"id":"ba5da102_944a6726","line":132,"in_reply_to":"ba5da102_44eca48f","updated":"2016-11-07 15:47:08.000000000","message":"I do quite like the semantic request to isolate the emulator resources. Its an implementation detail that you loose a pCPU in the same of libvirt and how we implement this.","commit_id":"aaa5bff651531e40b3af34c12097c27f6e7decec"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"4564ee393894af07684b4cf6da8517e754431733","unresolved":false,"context_lines":[{"line_number":129,"context_line":"with realtime by setting a fixed policy that the emulator threads only"},{"line_number":130,"context_line":"run on the vCPUs that have non-realtime policy. This requires that all"},{"line_number":131,"context_line":"guest OS using realtime are SMP, but some guest OS want realtime, but"},{"line_number":132,"context_line":"are only UP."},{"line_number":133,"context_line":""},{"line_number":134,"context_line":"Data model impact"},{"line_number":135,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":7,"id":"ba5da102_44eca48f","line":132,"in_reply_to":"ba5da102_f372e0af","updated":"2016-11-04 15:58:21.000000000","message":"This was in fact how an earlier version of the spec proposed todo it - we\u0027d have a flavour property to set a reserved CPU count for emulator threads. The review feedback was against exposing this concept in the flavour as too QEMU specific, hence this spec hides it from view by having it visible only between scheduler \u0026 virt driver, not in flavour / API anywhere.","commit_id":"aaa5bff651531e40b3af34c12097c27f6e7decec"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"762449e9561441572a2114875487c21d902c5b19","unresolved":false,"context_lines":[{"line_number":149,"context_line":"command line using \u0027isolcpus\u003d\u0027 has been removed. It means that the"},{"line_number":150,"context_line":"emulator threads are not going to float on the union of pCPUs"},{"line_number":151,"context_line":"dedicated to the guest but instead be constrained to the pCPU running"},{"line_number":152,"context_line":"vCPU 0."},{"line_number":153,"context_line":""},{"line_number":154,"context_line":"[1] https://kernel.googlesource.com/pub/scm/linux/kernel/git/stable/linux-stable/+/47b8ea7186aae7f474ec4c98f43eaa8da719cd83%5E%21/#F0"},{"line_number":155,"context_line":""}],"source_content_type":"text/x-rst","patch_set":7,"id":"ba5da102_d9d17d31","line":152,"updated":"2016-11-04 16:59:29.000000000","message":"Nit: Given this, I would say shared really means uses the same pCPU as the guest, rather than isolated from the guest. But all the information is here, so its just a nit.","commit_id":"aaa5bff651531e40b3af34c12097c27f6e7decec"},{"author":{"_account_id":7730,"name":"Sahid Orentino Ferdjaoui","email":"sahid.ferdjaoui@industrialdiscipline.com","username":"sahid"},"change_message_id":"125bc8014e073dbaccf06aa0de3728d21c281075","unresolved":false,"context_lines":[{"line_number":149,"context_line":"command line using \u0027isolcpus\u003d\u0027 has been removed. It means that the"},{"line_number":150,"context_line":"emulator threads are not going to float on the union of pCPUs"},{"line_number":151,"context_line":"dedicated to the guest but instead be constrained to the pCPU running"},{"line_number":152,"context_line":"vCPU 0."},{"line_number":153,"context_line":""},{"line_number":154,"context_line":"[1] https://kernel.googlesource.com/pub/scm/linux/kernel/git/stable/linux-stable/+/47b8ea7186aae7f474ec4c98f43eaa8da719cd83%5E%21/#F0"},{"line_number":155,"context_line":""}],"source_content_type":"text/x-rst","patch_set":7,"id":"9a629dbe_2ce7a713","line":152,"in_reply_to":"ba5da102_d9d17d31","updated":"2016-11-07 11:24:11.000000000","message":"Yes, please see line 145, that is what I\u0027m trying to say line 145. If it\u0027s not enough clear I can try to reformulate","commit_id":"aaa5bff651531e40b3af34c12097c27f6e7decec"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"762449e9561441572a2114875487c21d902c5b19","unresolved":false,"context_lines":[{"line_number":173,"context_line":""},{"line_number":174,"context_line":"For end users, using the option \u0027cpu_emulator_threads\u0027 is going to"},{"line_number":175,"context_line":"consume an additional host CPU on the resources quota regarding the"},{"line_number":176,"context_line":"guest vCPUs allocated."},{"line_number":177,"context_line":""},{"line_number":178,"context_line":"Performance Impact"},{"line_number":179,"context_line":"------------------"}],"source_content_type":"text/x-rst","patch_set":7,"id":"ba5da102_f9bd61d8","line":176,"updated":"2016-11-04 16:59:29.000000000","message":"Sorry, to check is that vCPU user quota that will be consumed, or do you mean the resource tracking in the scheduler generally?\n\nI think \"host CPU\" is confusing me here, for some reason.","commit_id":"aaa5bff651531e40b3af34c12097c27f6e7decec"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"d7dbf674d3e5e3110e1ffd56d73104d1960d5009","unresolved":false,"context_lines":[{"line_number":173,"context_line":""},{"line_number":174,"context_line":"For end users, using the option \u0027cpu_emulator_threads\u0027 is going to"},{"line_number":175,"context_line":"consume an additional host CPU on the resources quota regarding the"},{"line_number":176,"context_line":"guest vCPUs allocated."},{"line_number":177,"context_line":""},{"line_number":178,"context_line":"Performance Impact"},{"line_number":179,"context_line":"------------------"}],"source_content_type":"text/x-rst","patch_set":7,"id":"9a629dbe_18ca0fa2","line":176,"in_reply_to":"9a629dbe_eca4cf09","updated":"2016-11-07 15:47:08.000000000","message":"OK, I see where you are coming from.","commit_id":"aaa5bff651531e40b3af34c12097c27f6e7decec"},{"author":{"_account_id":7730,"name":"Sahid Orentino Ferdjaoui","email":"sahid.ferdjaoui@industrialdiscipline.com","username":"sahid"},"change_message_id":"872097ee64634a513d2013e4ccb36a14af745be8","unresolved":false,"context_lines":[{"line_number":173,"context_line":""},{"line_number":174,"context_line":"For end users, using the option \u0027cpu_emulator_threads\u0027 is going to"},{"line_number":175,"context_line":"consume an additional host CPU on the resources quota regarding the"},{"line_number":176,"context_line":"guest vCPUs allocated."},{"line_number":177,"context_line":""},{"line_number":178,"context_line":"Performance Impact"},{"line_number":179,"context_line":"------------------"}],"source_content_type":"text/x-rst","patch_set":7,"id":"9a629dbe_eca4cf09","line":176,"in_reply_to":"ba5da102_f9bd61d8","updated":"2016-11-07 11:22:46.000000000","message":"We are in a context where a pCPU is a vCPU (cpu_policy: dedicated). We can also say an additional vCPU is going to be consumed from the user quota. I don\u0027t want to talk about vCPU here since it\u0027s really related to the host.","commit_id":"aaa5bff651531e40b3af34c12097c27f6e7decec"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"762449e9561441572a2114875487c21d902c5b19","unresolved":false,"context_lines":[{"line_number":220,"context_line":"* Enchance InstanceNUMATopology to take into account cpu_emulator_threads"},{"line_number":221,"context_line":"* Make resource tracker to handle \u0027estimate_instance_overhead\u0027 with vcpus"},{"line_number":222,"context_line":"* Extend estimate_instance_overhead for libvirt"},{"line_number":223,"context_line":"* Make libvirt to corretly pin emulator threads if requested."},{"line_number":224,"context_line":""},{"line_number":225,"context_line":"Dependencies"},{"line_number":226,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":7,"id":"ba5da102_99daa516","line":223,"updated":"2016-11-04 16:59:29.000000000","message":"This doesn\u0027t mention about consuming quota, and I think thats fine, but I guess the other wording about made me thing we would consume quota.","commit_id":"aaa5bff651531e40b3af34c12097c27f6e7decec"},{"author":{"_account_id":7730,"name":"Sahid Orentino Ferdjaoui","email":"sahid.ferdjaoui@industrialdiscipline.com","username":"sahid"},"change_message_id":"872097ee64634a513d2013e4ccb36a14af745be8","unresolved":false,"context_lines":[{"line_number":220,"context_line":"* Enchance InstanceNUMATopology to take into account cpu_emulator_threads"},{"line_number":221,"context_line":"* Make resource tracker to handle \u0027estimate_instance_overhead\u0027 with vcpus"},{"line_number":222,"context_line":"* Extend estimate_instance_overhead for libvirt"},{"line_number":223,"context_line":"* Make libvirt to corretly pin emulator threads if requested."},{"line_number":224,"context_line":""},{"line_number":225,"context_line":"Dependencies"},{"line_number":226,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":7,"id":"9a629dbe_6ce7df40","line":223,"in_reply_to":"ba5da102_99daa516","updated":"2016-11-07 11:22:46.000000000","message":"When we are going to extend estimate_instance_overhead we are going to make that resource to be consumed from the user quota.","commit_id":"aaa5bff651531e40b3af34c12097c27f6e7decec"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"762449e9561441572a2114875487c21d902c5b19","unresolved":false,"context_lines":[{"line_number":236,"context_line":""},{"line_number":237,"context_line":"This can be tested in any CI system that is capable of testing the"},{"line_number":238,"context_line":"current NUMA and dedicated CPUs policy. i.e. It requires ability to"},{"line_number":239,"context_line":"use KVM and not merely QEMU."},{"line_number":240,"context_line":""},{"line_number":241,"context_line":"Documentation Impact"},{"line_number":242,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":7,"id":"ba5da102_21f8edc2","line":239,"updated":"2016-11-04 16:59:29.000000000","message":"Didn\u0027t we have a plan for a functional test for the scheduler bits that mock out bits of the driver needed to simulate these things?","commit_id":"aaa5bff651531e40b3af34c12097c27f6e7decec"},{"author":{"_account_id":7730,"name":"Sahid Orentino Ferdjaoui","email":"sahid.ferdjaoui@industrialdiscipline.com","username":"sahid"},"change_message_id":"872097ee64634a513d2013e4ccb36a14af745be8","unresolved":false,"context_lines":[{"line_number":236,"context_line":""},{"line_number":237,"context_line":"This can be tested in any CI system that is capable of testing the"},{"line_number":238,"context_line":"current NUMA and dedicated CPUs policy. i.e. It requires ability to"},{"line_number":239,"context_line":"use KVM and not merely QEMU."},{"line_number":240,"context_line":""},{"line_number":241,"context_line":"Documentation Impact"},{"line_number":242,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":7,"id":"9a629dbe_ac0497c6","line":239,"in_reply_to":"ba5da102_21f8edc2","updated":"2016-11-07 11:22:46.000000000","message":"Yes we can probably add functional tests fro the scheduling and the driver bits","commit_id":"aaa5bff651531e40b3af34c12097c27f6e7decec"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"f862ecdbb5881e127ed40844ecf832750ae95d45","unresolved":false,"context_lines":[{"line_number":13,"context_line":"The Nova scheduler determines CPU resource utilization and instance"},{"line_number":14,"context_line":"CPU placement based on the number of vCPUs in the flavor. A number of"},{"line_number":15,"context_line":"hypervisors have operations that are being performed on behalf of the"},{"line_number":16,"context_line":"guest instance in the host OS. These operations should be acounted and"},{"line_number":17,"context_line":"scheduled separately, as well as have their own placement policy"},{"line_number":18,"context_line":"controls applied."},{"line_number":19,"context_line":""}],"source_content_type":"text/x-rst","patch_set":8,"id":"7a77a97e_19c4ef90","line":16,"range":{"start_line":16,"start_character":58,"end_line":16,"end_character":66},"updated":"2016-11-14 18:35:52.000000000","message":"\"accounted\"","commit_id":"9953d320408ec3364b2cf0a3d88aff90eb98c6e7"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"f862ecdbb5881e127ed40844ecf832750ae95d45","unresolved":false,"context_lines":[{"line_number":69,"context_line":"guests, it prevents use of Nova to run other real time operating"},{"line_number":70,"context_line":"systems which require realtime response for all vCPUs. To broaden the"},{"line_number":71,"context_line":"realtime support it is necessary to pin emulator threads separately"},{"line_number":72,"context_line":"from vCPUs, which requires that the schedular be able to account for"},{"line_number":73,"context_line":"extra pCPU usage per guest."},{"line_number":74,"context_line":""},{"line_number":75,"context_line":"Project Priority"}],"source_content_type":"text/x-rst","patch_set":8,"id":"7a77a97e_591ae718","line":72,"range":{"start_line":72,"start_character":36,"end_line":72,"end_character":45},"updated":"2016-11-14 18:35:52.000000000","message":"\"scheduler\"","commit_id":"9953d320408ec3364b2cf0a3d88aff90eb98c6e7"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"f862ecdbb5881e127ed40844ecf832750ae95d45","unresolved":false,"context_lines":[{"line_number":85,"context_line":"‘dedicated’."},{"line_number":86,"context_line":""},{"line_number":87,"context_line":"Each hypervisor has a different architecture, for example QEMU has"},{"line_number":88,"context_line":"emulator threads, while Xen has stub-domains. To avoid favouring any"},{"line_number":89,"context_line":"specific implementation, the idea is to extend"},{"line_number":90,"context_line":"`estimate_instance_overhead` to return 1 additional host CPU to take"},{"line_number":91,"context_line":"into account during claim. A user which expresses the desire to"}],"source_content_type":"text/x-rst","patch_set":8,"id":"7a77a97e_f9ebb3ae","line":88,"range":{"start_line":88,"start_character":55,"end_line":88,"end_character":64},"updated":"2016-11-14 18:35:52.000000000","message":"\"favoring\"","commit_id":"9953d320408ec3364b2cf0a3d88aff90eb98c6e7"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"f862ecdbb5881e127ed40844ecf832750ae95d45","unresolved":false,"context_lines":[{"line_number":102,"context_line":"only one should work for almost every use case. If in the future there"},{"line_number":103,"context_line":"is a desire to isolate more than one host CPU to run emulator threads,"},{"line_number":104,"context_line":"we would implement instead I/O threads to add granularity on"},{"line_number":105,"context_line":"dedicating used resources to run guests on host CPUs."},{"line_number":106,"context_line":""},{"line_number":107,"context_line":"Alternatives"},{"line_number":108,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":8,"id":"7a77a97e_79efa3b0","line":105,"updated":"2016-11-14 18:35:52.000000000","message":"Do we charge them for this in quota? If so, how do we heal that for instances that are currently running?\n\nAlso, does this properly make it to the placement api as additional allocation for the instance? When we start exposing that to users for capacity planning in some fashion, it\u0027s going to be a little confusing that my 4-vcpu guest shows as taking 5....","commit_id":"9953d320408ec3364b2cf0a3d88aff90eb98c6e7"},{"author":{"_account_id":7730,"name":"Sahid Orentino Ferdjaoui","email":"sahid.ferdjaoui@industrialdiscipline.com","username":"sahid"},"change_message_id":"6310cde84e1412f740fe94cd2e6f717e781247f8","unresolved":false,"context_lines":[{"line_number":102,"context_line":"only one should work for almost every use case. If in the future there"},{"line_number":103,"context_line":"is a desire to isolate more than one host CPU to run emulator threads,"},{"line_number":104,"context_line":"we would implement instead I/O threads to add granularity on"},{"line_number":105,"context_line":"dedicating used resources to run guests on host CPUs."},{"line_number":106,"context_line":""},{"line_number":107,"context_line":"Alternatives"},{"line_number":108,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":8,"id":"7a77a97e_9359655d","line":105,"in_reply_to":"7a77a97e_79efa3b0","updated":"2016-11-16 11:23:36.000000000","message":"About the quota since we are in a cpu policy dedicated, a vCPU is basically a pCPU I guess we should consume 1 vCPU on the user quota to avoid user to be able to consume twice the number of CPUs allowed.\n\nI agree it might looks strange but we do not have other choice, we should expect that users which are going to use this feature will understand it. The other way to see that is to make the flavor requesting e.g: 5vCPUs with one reserved for the emulator threads but it\u0027s even more strange for the user to request 5vCPUs and only see 4 in its VM, also doing that is just mixing the concept of vCPUs/pCPUs which we probably don\u0027t want.","commit_id":"9953d320408ec3364b2cf0a3d88aff90eb98c6e7"},{"author":{"_account_id":7730,"name":"Sahid Orentino Ferdjaoui","email":"sahid.ferdjaoui@industrialdiscipline.com","username":"sahid"},"change_message_id":"f642a3e025c12269fa2873250ed2ef3c791e0394","unresolved":false,"context_lines":[{"line_number":102,"context_line":"only one should work for almost every use case. If in the future there"},{"line_number":103,"context_line":"is a desire to isolate more than one host CPU to run emulator threads,"},{"line_number":104,"context_line":"we would implement instead I/O threads to add granularity on"},{"line_number":105,"context_line":"dedicating used resources to run guests on host CPUs."},{"line_number":106,"context_line":""},{"line_number":107,"context_line":"Alternatives"},{"line_number":108,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":8,"id":"7a77a97e_d6790bfe","line":105,"in_reply_to":"7a77a97e_9359655d","updated":"2016-11-16 11:38:44.000000000","message":"Ok so after further discussion on IRC with John, we were agree that the current quotas is not so strict and already \"leak\" in several use-cases. We may want to keep the process simple in a first attempt and so do not touch them.\n\nI will add a note in the spec about that the quotas are not going to be touch, but, for sure community can have different judgments on how we should handle it,  John and me do not have strong opinions and are open on discussions.","commit_id":"9953d320408ec3364b2cf0a3d88aff90eb98c6e7"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"f862ecdbb5881e127ed40844ecf832750ae95d45","unresolved":false,"context_lines":[{"line_number":138,"context_line":""},{"line_number":139,"context_line":"* cpu_emulator_threads\u003dCPUEmulatorThreadsField()"},{"line_number":140,"context_line":""},{"line_number":141,"context_line":"This field will be implemented as an enum with two options:"},{"line_number":142,"context_line":""},{"line_number":143,"context_line":"* shared - The emulator threads float across the pCPUs associated to"},{"line_number":144,"context_line":"  the guest."}],"source_content_type":"text/x-rst","patch_set":8,"id":"7a77a97e_19a30fa0","line":141,"updated":"2016-11-14 18:35:52.000000000","message":"Hmm, I thought above you said that you would specify this on the flavor. Can we not just look at the flavor to make the determination? Or do we _need_ to persist another flag?","commit_id":"9953d320408ec3364b2cf0a3d88aff90eb98c6e7"},{"author":{"_account_id":7730,"name":"Sahid Orentino Ferdjaoui","email":"sahid.ferdjaoui@industrialdiscipline.com","username":"sahid"},"change_message_id":"2ecc674dd881fde33b347efa1a35f9a72186fed3","unresolved":false,"context_lines":[{"line_number":138,"context_line":""},{"line_number":139,"context_line":"* cpu_emulator_threads\u003dCPUEmulatorThreadsField()"},{"line_number":140,"context_line":""},{"line_number":141,"context_line":"This field will be implemented as an enum with two options:"},{"line_number":142,"context_line":""},{"line_number":143,"context_line":"* shared - The emulator threads float across the pCPUs associated to"},{"line_number":144,"context_line":"  the guest."}],"source_content_type":"text/x-rst","patch_set":8,"id":"7a77a97e_6be25005","line":141,"in_reply_to":"7a77a97e_19a30fa0","updated":"2016-11-15 07:22:40.000000000","message":"When we compute usage of instances we pass the instance NUMA topology It\u0027s more convenient/coherent to continue on this way.","commit_id":"9953d320408ec3364b2cf0a3d88aff90eb98c6e7"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"6cbef3c6b64926a0e3ef74100bc7af66bd8499fa","unresolved":false,"context_lines":[{"line_number":138,"context_line":""},{"line_number":139,"context_line":"* cpu_emulator_threads\u003dCPUEmulatorThreadsField()"},{"line_number":140,"context_line":""},{"line_number":141,"context_line":"This field will be implemented as an enum with two options:"},{"line_number":142,"context_line":""},{"line_number":143,"context_line":"* shared - The emulator threads float across the pCPUs associated to"},{"line_number":144,"context_line":"  the guest."}],"source_content_type":"text/x-rst","patch_set":8,"id":"7a77a97e_c9f79ef3","line":141,"in_reply_to":"7a77a97e_6be25005","updated":"2016-11-16 15:43:54.000000000","message":"Why? we have the flavor on the instance very easily. You\u0027re calculating usage for an instance so you must have the instance object already anyway, right? It just seems like duplication of information to me and more change in data models we serialize.","commit_id":"9953d320408ec3364b2cf0a3d88aff90eb98c6e7"},{"author":{"_account_id":7730,"name":"Sahid Orentino Ferdjaoui","email":"sahid.ferdjaoui@industrialdiscipline.com","username":"sahid"},"change_message_id":"e3a359df8802d276036497a648d8ab152eef5a89","unresolved":false,"context_lines":[{"line_number":138,"context_line":""},{"line_number":139,"context_line":"* cpu_emulator_threads\u003dCPUEmulatorThreadsField()"},{"line_number":140,"context_line":""},{"line_number":141,"context_line":"This field will be implemented as an enum with two options:"},{"line_number":142,"context_line":""},{"line_number":143,"context_line":"* shared - The emulator threads float across the pCPUs associated to"},{"line_number":144,"context_line":"  the guest."}],"source_content_type":"text/x-rst","patch_set":8,"id":"7a77a97e_9c5b02f6","line":141,"in_reply_to":"7a77a97e_c9f79ef3","updated":"2016-11-17 08:36:25.000000000","message":"No, in that method we only pass the instance topology, I don\u0027t say we can\u0027t change the code to also pass the instance, but I\u0027m trying to stay coherent with what we already have. - As you said on IRC, let\u0027s have this technical debate on the implementation.","commit_id":"9953d320408ec3364b2cf0a3d88aff90eb98c6e7"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"63248e05a2188dd57255cd541dc3e119ac829ea9","unresolved":false,"context_lines":[{"line_number":173,"context_line":""},{"line_number":174,"context_line":"For end users, using the option \u0027cpu_emulator_threads\u0027 is going to"},{"line_number":175,"context_line":"consume an additional host CPU on the resources quota regarding the"},{"line_number":176,"context_line":"guest vCPUs allocated."},{"line_number":177,"context_line":""},{"line_number":178,"context_line":"Performance Impact"},{"line_number":179,"context_line":"------------------"}],"source_content_type":"text/x-rst","patch_set":8,"id":"9a629dbe_51c60168","line":176,"updated":"2016-11-08 10:38:15.000000000","message":"Nit: after discussion I believe this means, we consume extra user quota, due to the request.\n\nI could see arguments either way, and practically, I think this is a bit tricky code wise, but I don\u0027t mind if no quota is consumed.\n\nThe code is probably the best place to debate that at this point.","commit_id":"9953d320408ec3364b2cf0a3d88aff90eb98c6e7"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"a20af9257b0ca5daa23b91fcbb7fbd24056cfef9","unresolved":false,"context_lines":[{"line_number":186,"context_line":"---------------------"},{"line_number":187,"context_line":""},{"line_number":188,"context_line":"Deployers who want to use that new feature will have to configure"},{"line_number":189,"context_line":"thier flavors to use a dedicated cpu policy (hw:cpu_policy\u003ddedicated),"},{"line_number":190,"context_line":"in a same time set \u0027hw:cpu_emulator_threads\u0027 to \u0027isolate\u0027."},{"line_number":191,"context_line":""},{"line_number":192,"context_line":"Developer impact"}],"source_content_type":"text/x-rst","patch_set":8,"id":"9a629dbe_296c4bcf","line":189,"updated":"2016-11-10 14:33:46.000000000","message":"their","commit_id":"9953d320408ec3364b2cf0a3d88aff90eb98c6e7"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"f862ecdbb5881e127ed40844ecf832750ae95d45","unresolved":false,"context_lines":[{"line_number":193,"context_line":"----------------"},{"line_number":194,"context_line":""},{"line_number":195,"context_line":"* Developers of other virtualization drivers may wish to make use of"},{"line_number":196,"context_line":"  the new flavour extra spec property and scheduler accounting. This"},{"line_number":197,"context_line":"  will be of particular interest to the Xen hypervisor, if using the"},{"line_number":198,"context_line":"  stub domain feature."},{"line_number":199,"context_line":""}],"source_content_type":"text/x-rst","patch_set":8,"id":"7a77a97e_19ca2f50","line":196,"range":{"start_line":196,"start_character":10,"end_line":196,"end_character":17},"updated":"2016-11-14 18:35:52.000000000","message":"\"flavor\"","commit_id":"9953d320408ec3364b2cf0a3d88aff90eb98c6e7"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"f862ecdbb5881e127ed40844ecf832750ae95d45","unresolved":false,"context_lines":[{"line_number":217,"context_line":"----------"},{"line_number":218,"context_line":""},{"line_number":219,"context_line":"* Enhance flavor extra spec to take into account hw:cpu_emulator_threads"},{"line_number":220,"context_line":"* Enchance InstanceNUMATopology to take into account cpu_emulator_threads"},{"line_number":221,"context_line":"* Make resource tracker to handle \u0027estimate_instance_overhead\u0027 with vcpus"},{"line_number":222,"context_line":"* Extend estimate_instance_overhead for libvirt"},{"line_number":223,"context_line":"* Make libvirt to corretly pin emulator threads if requested."}],"source_content_type":"text/x-rst","patch_set":8,"id":"7a77a97e_7946c3c0","line":220,"range":{"start_line":220,"start_character":2,"end_line":220,"end_character":10},"updated":"2016-11-14 18:35:52.000000000","message":"Enchance sounds cool, butt his should be \"enhance\"","commit_id":"9953d320408ec3364b2cf0a3d88aff90eb98c6e7"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"d14aa080b50e61d68d64defe79c265f545195a5b","unresolved":false,"context_lines":[{"line_number":237,"context_line":"This can be tested in any CI system that is capable of testing the"},{"line_number":238,"context_line":"current NUMA and dedicated CPUs policy. i.e. It requires ability to"},{"line_number":239,"context_line":"use KVM and not merely QEMU. Functionnal tests for the scheduling and"},{"line_number":240,"context_line":"driver bits (libvirt) are going to be added."},{"line_number":241,"context_line":""},{"line_number":242,"context_line":"Documentation Impact"},{"line_number":243,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":8,"id":"9a629dbe_ddb409d6","line":240,"updated":"2016-11-14 10:28:53.000000000","message":"Just as a heads up, seems I miss read this. We really need to get a CI that can test this. I would by happier if this was a commitment to get a CI system up and running.","commit_id":"9953d320408ec3364b2cf0a3d88aff90eb98c6e7"},{"author":{"_account_id":7730,"name":"Sahid Orentino Ferdjaoui","email":"sahid.ferdjaoui@industrialdiscipline.com","username":"sahid"},"change_message_id":"2ecc674dd881fde33b347efa1a35f9a72186fed3","unresolved":false,"context_lines":[{"line_number":237,"context_line":"This can be tested in any CI system that is capable of testing the"},{"line_number":238,"context_line":"current NUMA and dedicated CPUs policy. i.e. It requires ability to"},{"line_number":239,"context_line":"use KVM and not merely QEMU. Functionnal tests for the scheduling and"},{"line_number":240,"context_line":"driver bits (libvirt) are going to be added."},{"line_number":241,"context_line":""},{"line_number":242,"context_line":"Documentation Impact"},{"line_number":243,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":8,"id":"7a77a97e_8ba32c9e","line":240,"in_reply_to":"7a77a97e_592027f8","updated":"2016-11-15 07:22:40.000000000","message":"I do not have so much information about the Intel NFV CI. Basically what we want here is to ensure the placement, I\u0027m not completely convinced to have it into Intel CI would bring a lot of robustness. But I can try to see with the group to incorporate it yes.","commit_id":"9953d320408ec3364b2cf0a3d88aff90eb98c6e7"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"f862ecdbb5881e127ed40844ecf832750ae95d45","unresolved":false,"context_lines":[{"line_number":237,"context_line":"This can be tested in any CI system that is capable of testing the"},{"line_number":238,"context_line":"current NUMA and dedicated CPUs policy. i.e. It requires ability to"},{"line_number":239,"context_line":"use KVM and not merely QEMU. Functionnal tests for the scheduling and"},{"line_number":240,"context_line":"driver bits (libvirt) are going to be added."},{"line_number":241,"context_line":""},{"line_number":242,"context_line":"Documentation Impact"},{"line_number":243,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":8,"id":"7a77a97e_592027f8","line":240,"in_reply_to":"9a629dbe_ddb409d6","updated":"2016-11-14 18:35:52.000000000","message":"Yeah, especially because of all the resource tracking implications here.\n\nCan the intel nfv ci system handle this?","commit_id":"9953d320408ec3364b2cf0a3d88aff90eb98c6e7"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"cd79b1fe944b458ab650209ac50843bdd2143429","unresolved":false,"context_lines":[{"line_number":106,"context_line":""},{"line_number":107,"context_line":"As we said an additional pCPU is going to be consumed but this first"},{"line_number":108,"context_line":"implementation is not going to update the user quotas, that in a"},{"line_number":109,"context_line":"spirit of simplicity since quotas already leak on different scenarios."},{"line_number":110,"context_line":""},{"line_number":111,"context_line":"Alternatives"},{"line_number":112,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":10,"id":"7a77a97e_a8ea31aa","line":109,"updated":"2016-11-17 11:04:09.000000000","message":"Nit: For me its more to do with the flavor vCPU count matching the quota usage. Its confusing if we break that. (also the code is hard/messy, due to virt specific-ness)","commit_id":"ad39cf6b8785e5019b0453e7421c5d3b064429ba"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"cd79b1fe944b458ab650209ac50843bdd2143429","unresolved":false,"context_lines":[{"line_number":241,"context_line":"This can be tested in any CI system that is capable of testing the"},{"line_number":242,"context_line":"current NUMA and dedicated CPUs policy. i.e. It requires ability to"},{"line_number":243,"context_line":"use KVM and not merely QEMU. Functionnal tests for the scheduling and"},{"line_number":244,"context_line":"driver bits (libvirt) are going to be added."},{"line_number":245,"context_line":""},{"line_number":246,"context_line":"Documentation Impact"},{"line_number":247,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":10,"id":"7a77a97e_c87e8d18","line":244,"updated":"2016-11-17 11:04:09.000000000","message":"Nit: I wish we were more committed to the integration testing here.","commit_id":"ad39cf6b8785e5019b0453e7421c5d3b064429ba"}]}