)]}'
{"specs/mitaka/approved/virt-driver-cpu-pinning.rst":[{"author":{"_account_id":19212,"name":"Antonio López","email":"antonio.lopezgracia@telefonica.com","username":"alogra"},"change_message_id":"99189f42816cfef389c48e27b6e1e86c44e2c75c","unresolved":false,"context_lines":[{"line_number":58,"context_line":" - avoid: the scheduler will not place the guest on a host which has"},{"line_number":59,"context_line":"   hyperthreads."},{"line_number":60,"context_line":" - separate: if the host has threads, each vCPU will be placed on a"},{"line_number":61,"context_line":"   different core. ie no two vCPUs will be placed on thread siblings"},{"line_number":62,"context_line":" - isolate: if the host has threads, each vCPU will be placed on a"},{"line_number":63,"context_line":"   different core and no vCPUs from other guests will be able to be"},{"line_number":64,"context_line":"   placed on the same core. ie one thread sibling is guaranteed to"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da85f559_35ba616f","line":61,"updated":"2015-11-10 16:43:22.000000000","message":"[TEF] The \"separate\" option will allow to pin the vCPUs on different NUMAs if required by the current resource usage? We understand the answer is yes.","commit_id":"b56d8017000bbdb73286c132917604aed53ee339"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"fc7a1506b19a369ed86d709745da869e99c76229","unresolved":false,"context_lines":[{"line_number":58,"context_line":" - avoid: the scheduler will not place the guest on a host which has"},{"line_number":59,"context_line":"   hyperthreads."},{"line_number":60,"context_line":" - separate: if the host has threads, each vCPU will be placed on a"},{"line_number":61,"context_line":"   different core. ie no two vCPUs will be placed on thread siblings"},{"line_number":62,"context_line":" - isolate: if the host has threads, each vCPU will be placed on a"},{"line_number":63,"context_line":"   different core and no vCPUs from other guests will be able to be"},{"line_number":64,"context_line":"   placed on the same core. ie one thread sibling is guaranteed to"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da85f559_9153eae1","line":61,"in_reply_to":"da85f559_35ba616f","updated":"2015-11-11 16:29:12.000000000","message":"You will not be able to pin vCPUs of one instance to pCPUs spread across different NUMA nodes. This is a limitation of the underlying data model and, while a bug has been reported against this, fixing this issue is outside of the scope of this spec.","commit_id":"b56d8017000bbdb73286c132917604aed53ee339"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"bec770159f4518f45b62cde050ec89deefcfb7f2","unresolved":false,"context_lines":[{"line_number":62,"context_line":" - isolate: if the host has threads, each vCPU will be placed on a"},{"line_number":63,"context_line":"   different core and no vCPUs from other guests will be able to be"},{"line_number":64,"context_line":"   placed on the same core. ie one thread sibling is guaranteed to"},{"line_number":65,"context_line":"   always be unused,"},{"line_number":66,"context_line":" - require: the scheduler will only place the guest on a host which has"},{"line_number":67,"context_line":"   hyperthreads, and vCPUs will be placed on the thread siblings of a single"},{"line_number":68,"context_line":"   core. This implies that you cannot have more vCPUs in the guest than the"}],"source_content_type":"text/x-rst","patch_set":1,"id":"fa80f949_e5f535cb","line":65,"updated":"2015-11-04 20:48:32.000000000","message":"If we rework, I think the isolate and separate should also be clarified. In my mind, all of the \u0027if the host had threads\u0027 statements had an implied \u0027if the host does not have threads, the scheduler won\u0027t place a guest on it\u0027 addition. My rationale for this is two fold:\n\n1) Allowing the \u0027separate\u0027 and \u0027isolate\u0027 cases to work on non-SMT hosts is a lie. By definition, the cases are designed to allow non SMT-like performance on SMT hosts. This means SMT must be present for the cases (per specification) to make sense.\n\n2) Regarding the operator impact, I think changing this would actually be detrimental for operators. In all likelihood, non-SMT hosts are going to be a \"premium resource\". Most server platforms available today are SMT-enabled so SMT will need to be disabled. Disabling SMT on a host will cut the \"core count\" of said platform in half (or worse, for \u003e2 thread sibling, non-Intel platforms like POWER). While the missing \"cores\" might not be true cores, that\u0027s still a lot of potential compute power to be sacrificing. If SMT has been disabled then it has been done so for a very good reason and a specific case (\u0027avoid\u0027) should be used.\n\nNote that I\u0027m using SMT as a generic term for tech like Intel Hyper-Threading, and SMP as a term for multi-core architectures.","commit_id":"b56d8017000bbdb73286c132917604aed53ee339"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"6ff5699daef6dba618eb58c596347e2de20f0240","unresolved":false,"context_lines":[{"line_number":62,"context_line":" - isolate: if the host has threads, each vCPU will be placed on a"},{"line_number":63,"context_line":"   different core and no vCPUs from other guests will be able to be"},{"line_number":64,"context_line":"   placed on the same core. ie one thread sibling is guaranteed to"},{"line_number":65,"context_line":"   always be unused,"},{"line_number":66,"context_line":" - require: the scheduler will only place the guest on a host which has"},{"line_number":67,"context_line":"   hyperthreads, and vCPUs will be placed on the thread siblings of a single"},{"line_number":68,"context_line":"   core. This implies that you cannot have more vCPUs in the guest than the"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da85f559_883002ea","line":65,"in_reply_to":"da85f559_12ef6a88","updated":"2015-11-09 20:11:34.000000000","message":"@Chris: RE: #1417723, this looks very similar to #1466780, which I fixed but Nikola has marked as incorrect. If I understand him correctly, the host topology should have no bearing on the guest topology so that bug could in fact be invalid.\n\nRE: the \u0027auto\u0027 case, I think we should retain add an \u0027auto\u0027 policy. Like the \u0027shared\u0027 option in the \u0027hw:cpu_policy\u0027 field, this should be implicitly applied when no other option is specified. I\u0027ll rework the spec per this.\n\nRE: operator input - how would I get this feedback?","commit_id":"b56d8017000bbdb73286c132917604aed53ee339"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"c95ad735e8d5b576bd409c3c0d739545e07b9b14","unresolved":false,"context_lines":[{"line_number":62,"context_line":" - isolate: if the host has threads, each vCPU will be placed on a"},{"line_number":63,"context_line":"   different core and no vCPUs from other guests will be able to be"},{"line_number":64,"context_line":"   placed on the same core. ie one thread sibling is guaranteed to"},{"line_number":65,"context_line":"   always be unused,"},{"line_number":66,"context_line":" - require: the scheduler will only place the guest on a host which has"},{"line_number":67,"context_line":"   hyperthreads, and vCPUs will be placed on the thread siblings of a single"},{"line_number":68,"context_line":"   core. This implies that you cannot have more vCPUs in the guest than the"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da85f559_badfe367","line":65,"in_reply_to":"da85f559_1ff621f9","updated":"2015-11-11 17:12:51.000000000","message":"If you consider non-HT hosts valuable and want to restrict usage, then host aggregates + flavours already provide such facility.","commit_id":"b56d8017000bbdb73286c132917604aed53ee339"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"23129279d82660d0e71a204b7a11bb10e7663adb","unresolved":false,"context_lines":[{"line_number":62,"context_line":" - isolate: if the host has threads, each vCPU will be placed on a"},{"line_number":63,"context_line":"   different core and no vCPUs from other guests will be able to be"},{"line_number":64,"context_line":"   placed on the same core. ie one thread sibling is guaranteed to"},{"line_number":65,"context_line":"   always be unused,"},{"line_number":66,"context_line":" - require: the scheduler will only place the guest on a host which has"},{"line_number":67,"context_line":"   hyperthreads, and vCPUs will be placed on the thread siblings of a single"},{"line_number":68,"context_line":"   core. This implies that you cannot have more vCPUs in the guest than the"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da85f559_c90cd996","line":65,"in_reply_to":"da85f559_49354a44","updated":"2015-11-09 16:54:46.000000000","message":"@Chris, @Sean: So do you both agree that I need to allow separate and isolate to handle both SMT and non-SMT hosts? This will be a bit of rework and a few more tests but it should be doable. I still don\u0027t agree (non-SMT should be an exceptional choice, in the literal sense of the word), but I\u0027ll go with the majority option here.\n\n@Daniel: We already support a \u0027default\u0027 action, which as you say is the existing \"psuedo-prefer\" option. This isn\u0027t explicit, though maybe it should be? However, in any case Nikola is changing this default in his fix for bug #1501358 [1]. Now I don\u0027t agree with that change, but you\u0027re saying we can\u0027t change the default. Does that mean he shouldn\u0027t be doing this?\n\n[1] https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bug/1501358,n,z","commit_id":"b56d8017000bbdb73286c132917604aed53ee339"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"c0e0d035feec745ef99d6838c2e0e20bc396abf5","unresolved":false,"context_lines":[{"line_number":62,"context_line":" - isolate: if the host has threads, each vCPU will be placed on a"},{"line_number":63,"context_line":"   different core and no vCPUs from other guests will be able to be"},{"line_number":64,"context_line":"   placed on the same core. ie one thread sibling is guaranteed to"},{"line_number":65,"context_line":"   always be unused,"},{"line_number":66,"context_line":" - require: the scheduler will only place the guest on a host which has"},{"line_number":67,"context_line":"   hyperthreads, and vCPUs will be placed on the thread siblings of a single"},{"line_number":68,"context_line":"   core. This implies that you cannot have more vCPUs in the guest than the"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da85f559_1ff621f9","line":65,"in_reply_to":"da85f559_54eb3013","updated":"2015-11-11 16:56:32.000000000","message":"What about the \"non-HT hosts are valuable\" argument? The granularity the existing proposal, per my interpretation, grants is desirable","commit_id":"b56d8017000bbdb73286c132917604aed53ee339"},{"author":{"_account_id":8768,"name":"Chris Friesen","email":"chris.friesen@windriver.com","username":"cbf123"},"change_message_id":"e55ba3756bdebc7dffef3c09febadc5ac0fa920b","unresolved":false,"context_lines":[{"line_number":62,"context_line":" - isolate: if the host has threads, each vCPU will be placed on a"},{"line_number":63,"context_line":"   different core and no vCPUs from other guests will be able to be"},{"line_number":64,"context_line":"   placed on the same core. ie one thread sibling is guaranteed to"},{"line_number":65,"context_line":"   always be unused,"},{"line_number":66,"context_line":" - require: the scheduler will only place the guest on a host which has"},{"line_number":67,"context_line":"   hyperthreads, and vCPUs will be placed on the thread siblings of a single"},{"line_number":68,"context_line":"   core. This implies that you cannot have more vCPUs in the guest than the"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da85f559_a84066a6","line":65,"in_reply_to":"da85f559_883002ea","updated":"2015-11-09 20:22:15.000000000","message":"@Stephen: In #1466780 Nikola mentioned that we shouldn\u0027t *store* the calculated topology.  But he says \"The NUMA cell CPU topology was meant to carry information about threads that we want to expose to the single cell based on how it was fitted with regards to threading on the host, so that we can expose this information to the guest OS if possible for optimal perf.\"  To me that sounds very much like fixing #1417723.  (Which is basically making the guest topology look like the actual topology on the host.)\n\nMy take is that we should be consistent with threading regardless of even/odd, but then reflect that information up to the guest.\n\n(And if #1417723 has already been fixed, so much the better.)","commit_id":"b56d8017000bbdb73286c132917604aed53ee339"},{"author":{"_account_id":8768,"name":"Chris Friesen","email":"chris.friesen@windriver.com","username":"cbf123"},"change_message_id":"6cf095a7daad30ec1524d7b2871d4d2ece31b72f","unresolved":false,"context_lines":[{"line_number":62,"context_line":" - isolate: if the host has threads, each vCPU will be placed on a"},{"line_number":63,"context_line":"   different core and no vCPUs from other guests will be able to be"},{"line_number":64,"context_line":"   placed on the same core. ie one thread sibling is guaranteed to"},{"line_number":65,"context_line":"   always be unused,"},{"line_number":66,"context_line":" - require: the scheduler will only place the guest on a host which has"},{"line_number":67,"context_line":"   hyperthreads, and vCPUs will be placed on the thread siblings of a single"},{"line_number":68,"context_line":"   core. This implies that you cannot have more vCPUs in the guest than the"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da85f559_e3102c02","line":65,"in_reply_to":"da85f559_a58b4467","updated":"2015-11-13 03:45:09.000000000","message":"I would agree that if we\u0027re using aggregates to provide true non-HT hosts then \"avoid\" isn\u0027t needed anymore.","commit_id":"b56d8017000bbdb73286c132917604aed53ee339"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"8c2484bd6d571a2ae2c58e87656c0143ca1a9f8f","unresolved":false,"context_lines":[{"line_number":62,"context_line":" - isolate: if the host has threads, each vCPU will be placed on a"},{"line_number":63,"context_line":"   different core and no vCPUs from other guests will be able to be"},{"line_number":64,"context_line":"   placed on the same core. ie one thread sibling is guaranteed to"},{"line_number":65,"context_line":"   always be unused,"},{"line_number":66,"context_line":" - require: the scheduler will only place the guest on a host which has"},{"line_number":67,"context_line":"   hyperthreads, and vCPUs will be placed on the thread siblings of a single"},{"line_number":68,"context_line":"   core. This implies that you cannot have more vCPUs in the guest than the"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da85f559_4b9ab06f","line":65,"in_reply_to":"da85f559_a84066a6","updated":"2015-11-11 15:46:30.000000000","message":"@Chris: Any feedback on the other question? I think we should add an \u0027auto\u0027 option that doesn\u0027t really do anything (like \u0027shared\u0027), but do I need to rework the \u0027isolate\u0027 and \u0027separate\u0027 case? If possible, I\u0027d really rather not do so as I\u0027d like this to actually make it in for the Mitaka release. It\u0027s unlikely we\u0027ll get operator feedback in suitable time for rework before the December 3 deadline (I\u0027m on vacation next week too, which doesn\u0027t help).","commit_id":"b56d8017000bbdb73286c132917604aed53ee339"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"1c9e3992f6c88adc0bc5ba5d3e9591fba5258dd5","unresolved":false,"context_lines":[{"line_number":62,"context_line":" - isolate: if the host has threads, each vCPU will be placed on a"},{"line_number":63,"context_line":"   different core and no vCPUs from other guests will be able to be"},{"line_number":64,"context_line":"   placed on the same core. ie one thread sibling is guaranteed to"},{"line_number":65,"context_line":"   always be unused,"},{"line_number":66,"context_line":" - require: the scheduler will only place the guest on a host which has"},{"line_number":67,"context_line":"   hyperthreads, and vCPUs will be placed on the thread siblings of a single"},{"line_number":68,"context_line":"   core. This implies that you cannot have more vCPUs in the guest than the"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da85f559_a58b4467","line":65,"in_reply_to":"da85f559_badfe367","updated":"2015-11-11 17:25:30.000000000","message":"In that case, does it make sense to combine drop \u0027avoid\u0027? Both \u0027avoid\u0027 and \u0027separate\u0027 provide non-HT performance/determinism, and the only difference was the former worked for real non-HT hosts while the latter emulates non-HT behavior on HT hosts. If we\u0027re blurring the lines for the latter, I don\u0027t think \u0027avoid\u0027 by itself makes sense. As you suggest, we can use aggregates and flavours if operators wished to use only true non-HT hosts.","commit_id":"b56d8017000bbdb73286c132917604aed53ee339"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"976b8cfd4101c58195ee8a29517951abc43628fe","unresolved":false,"context_lines":[{"line_number":62,"context_line":" - isolate: if the host has threads, each vCPU will be placed on a"},{"line_number":63,"context_line":"   different core and no vCPUs from other guests will be able to be"},{"line_number":64,"context_line":"   placed on the same core. ie one thread sibling is guaranteed to"},{"line_number":65,"context_line":"   always be unused,"},{"line_number":66,"context_line":" - require: the scheduler will only place the guest on a host which has"},{"line_number":67,"context_line":"   hyperthreads, and vCPUs will be placed on the thread siblings of a single"},{"line_number":68,"context_line":"   core. This implies that you cannot have more vCPUs in the guest than the"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da85f559_efebdd76","line":65,"in_reply_to":"da85f559_c90cd996","updated":"2015-11-09 17:35:44.000000000","message":"the auto/allow option we were discussing would maintain the current behavior.\ni am suggesting we make that an explicit default value for the threading policy.\n\nthis would require a data migration on upgrade to add the auto/allow threading policy value for instance with pinned cpus but would not change the behavior on upgrade.\n\nthe current behavior is to emulate prefer for an even number of cpus and separate for an odd number of cpus.\n\nthe nikola\u0027s bug https://bugs.launchpad.net/nova/+bug/1501358\nwill change the existing default behavior in the case of odd cpus which i thinks is incorrect.","commit_id":"b56d8017000bbdb73286c132917604aed53ee339"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"24f9ebf3351876941bdfe37f17602e647b6eff49","unresolved":false,"context_lines":[{"line_number":62,"context_line":" - isolate: if the host has threads, each vCPU will be placed on a"},{"line_number":63,"context_line":"   different core and no vCPUs from other guests will be able to be"},{"line_number":64,"context_line":"   placed on the same core. ie one thread sibling is guaranteed to"},{"line_number":65,"context_line":"   always be unused,"},{"line_number":66,"context_line":" - require: the scheduler will only place the guest on a host which has"},{"line_number":67,"context_line":"   hyperthreads, and vCPUs will be placed on the thread siblings of a single"},{"line_number":68,"context_line":"   core. This implies that you cannot have more vCPUs in the guest than the"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da85f559_49354a44","line":65,"in_reply_to":"da85f559_c9905a98","updated":"2015-11-09 16:37:37.000000000","message":"NB, we cannot change the default of what we already shipped, as that would result in a change in behaviour for people upgrading. So the currently implemented policy is implicating the default if the policy is not specified.","commit_id":"b56d8017000bbdb73286c132917604aed53ee339"},{"author":{"_account_id":19212,"name":"Antonio López","email":"antonio.lopezgracia@telefonica.com","username":"alogra"},"change_message_id":"e965e0704fcd007f986b5ed6b65258b200def5d6","unresolved":false,"context_lines":[{"line_number":62,"context_line":" - isolate: if the host has threads, each vCPU will be placed on a"},{"line_number":63,"context_line":"   different core and no vCPUs from other guests will be able to be"},{"line_number":64,"context_line":"   placed on the same core. ie one thread sibling is guaranteed to"},{"line_number":65,"context_line":"   always be unused,"},{"line_number":66,"context_line":" - require: the scheduler will only place the guest on a host which has"},{"line_number":67,"context_line":"   hyperthreads, and vCPUs will be placed on the thread siblings of a single"},{"line_number":68,"context_line":"   core. This implies that you cannot have more vCPUs in the guest than the"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da85f559_51759961","line":65,"in_reply_to":"da85f559_efebdd76","updated":"2015-11-11 08:42:51.000000000","message":"@Sean, what is the logic behind having different default behaviour, emulate prefer for an even number of cpus and separate of an odd number? Why do you think it is incorrect to change the existing default behaviour in the case of odd cpus?","commit_id":"b56d8017000bbdb73286c132917604aed53ee339"},{"author":{"_account_id":8768,"name":"Chris Friesen","email":"chris.friesen@windriver.com","username":"cbf123"},"change_message_id":"ff5ae86284faf16d911d2b99538fb8a7cef41d14","unresolved":false,"context_lines":[{"line_number":62,"context_line":" - isolate: if the host has threads, each vCPU will be placed on a"},{"line_number":63,"context_line":"   different core and no vCPUs from other guests will be able to be"},{"line_number":64,"context_line":"   placed on the same core. ie one thread sibling is guaranteed to"},{"line_number":65,"context_line":"   always be unused,"},{"line_number":66,"context_line":" - require: the scheduler will only place the guest on a host which has"},{"line_number":67,"context_line":"   hyperthreads, and vCPUs will be placed on the thread siblings of a single"},{"line_number":68,"context_line":"   core. This implies that you cannot have more vCPUs in the guest than the"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da85f559_12ef6a88","line":65,"in_reply_to":"da85f559_efebdd76","updated":"2015-11-09 18:02:39.000000000","message":"@Stephen Ideally I\u0027d like to see operator input on the semantics for \"separate\" and \"isolate\".  If they would prefer to fail to schedule rather than place such an instance on a non-HT host then I\u0027m okay with that.  (From a \"does it meet the requirements of the flavor\" I think it would make sense though.)\n\nI agree that a thread policy of \"auto\" or \"allow\" should be added, and should map to the behaviour when no thread policy is specified.  I don\u0027t think we should cause any extra specs to be enabled by default, special-casing the not-specified case is trivial.\n\nAs to the default behaviour...Daniel\u0027s argument was that we shouldn\u0027t prefer thread siblings and then tell the host we\u0027re using separate sockets, which I think is reasonable.  So I think the best option is to submit Nikola\u0027s fix to make even/odd behaviour consistent, but then also make the guest topology match the host topology.  (Thus fixing https://bugs.launchpad.net/nova/+bug/1417723 )","commit_id":"b56d8017000bbdb73286c132917604aed53ee339"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"b41139e8672712b6979788dfdc9dbae80a1b6052","unresolved":false,"context_lines":[{"line_number":62,"context_line":" - isolate: if the host has threads, each vCPU will be placed on a"},{"line_number":63,"context_line":"   different core and no vCPUs from other guests will be able to be"},{"line_number":64,"context_line":"   placed on the same core. ie one thread sibling is guaranteed to"},{"line_number":65,"context_line":"   always be unused,"},{"line_number":66,"context_line":" - require: the scheduler will only place the guest on a host which has"},{"line_number":67,"context_line":"   hyperthreads, and vCPUs will be placed on the thread siblings of a single"},{"line_number":68,"context_line":"   core. This implies that you cannot have more vCPUs in the guest than the"}],"source_content_type":"text/x-rst","patch_set":1,"id":"fa80f949_7510fe01","line":65,"in_reply_to":"fa80f949_3845a3b2","updated":"2015-11-05 12:10:21.000000000","message":"if you compare single thread cpu performance with hyperthreading (HT) enabled with nothing using the HT siblings as in the isolate case to the same host with HT disabled (the avoid case) you will have better performance in the avoid case.\n\nthe reason for this is at the hardware level, when HT is enabled the l1 cache and cpu registers of a core are split between the primary thread and the hyper thread sibling. when hyper threading is disabled the l1 cache and cpu registers are fully available to primary thread.\n\nin the separate case the user has not expressed any preferences for a system with our with out HT enabled.\n\nin the avoid case they have explicitly requested a system with HT disabled(perhaps they have a cache sensitive workload)\n\nwith require and isolate they have semantics that required HT to exist and specify how it should behave in that case. \n\non a system with HT disabled separate and isolate function identically.\n\ni think all 4 policies should be kept avoid|separate|isolate|require \n\nas for \"allow\" i think a better approach would be to have a config value to specify the default threading policy.\n\nthe current threading policy is fundamentally broken emulating the behavior of prefer for even number of cpus and separate for odd numbers of vcpus.\n\nif a new type was added to represent this behavior i think \"auto\" would be a better name then \"allow\" however i think in the long run having the operator choose a default in the config will have more consistent results and easier documentation.","commit_id":"b56d8017000bbdb73286c132917604aed53ee339"},{"author":{"_account_id":8768,"name":"Chris Friesen","email":"chris.friesen@windriver.com","username":"cbf123"},"change_message_id":"7f3c13d09350c65d7652115d9f8966bd8f4bab6c","unresolved":false,"context_lines":[{"line_number":62,"context_line":" - isolate: if the host has threads, each vCPU will be placed on a"},{"line_number":63,"context_line":"   different core and no vCPUs from other guests will be able to be"},{"line_number":64,"context_line":"   placed on the same core. ie one thread sibling is guaranteed to"},{"line_number":65,"context_line":"   always be unused,"},{"line_number":66,"context_line":" - require: the scheduler will only place the guest on a host which has"},{"line_number":67,"context_line":"   hyperthreads, and vCPUs will be placed on the thread siblings of a single"},{"line_number":68,"context_line":"   core. This implies that you cannot have more vCPUs in the guest than the"}],"source_content_type":"text/x-rst","patch_set":1,"id":"fa80f949_d791a79a","line":65,"in_reply_to":"fa80f949_7510fe01","updated":"2015-11-05 22:26:38.000000000","message":"From what I\u0027ve read, the L1 cache is actually shared between threads, not partitioned.  But you are correct that there are some resources that are partitioned, which could conceivably lead to a performance hit.\n\nI don\u0027t see \"isolate\" and \"separate\" as requiring a host with SMT, but rather imposing requirements if the host has SMT.\n\nI see separate/isolate/avoid as a continuum.  \"separate\" means \"I want each of my vCPUs to run on a separate host core.\"  \"isolate\" takes that further and adds \"and I don\u0027t want anyone else to run on those host cores\".  \"avoid\" takes it one step further and says \"and I don\u0027t want any other threads to exist on those host cores.\"  Theoretically all of those requirements are met by a non-SMT host.\n\nAs for the proposed \"allow\" (or \"auto\", I\u0027m not picky), I just think that if we\u0027re specifying flavor extra specs it would be good to have one option represent the behaviour that you get if you don\u0027t specify the extra spec at all...otherwise end-users would have to special-case that code path by removing the option entirely.  When we specify the CPU policy, for example, we can indicate \"dedicated\" or \"shared\", where the latter is what you get without specifying the policy at all.","commit_id":"b56d8017000bbdb73286c132917604aed53ee339"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"28b5213eca6d5ca6eb97dfa589ea71689d601a59","unresolved":false,"context_lines":[{"line_number":62,"context_line":" - isolate: if the host has threads, each vCPU will be placed on a"},{"line_number":63,"context_line":"   different core and no vCPUs from other guests will be able to be"},{"line_number":64,"context_line":"   placed on the same core. ie one thread sibling is guaranteed to"},{"line_number":65,"context_line":"   always be unused,"},{"line_number":66,"context_line":" - require: the scheduler will only place the guest on a host which has"},{"line_number":67,"context_line":"   hyperthreads, and vCPUs will be placed on the thread siblings of a single"},{"line_number":68,"context_line":"   core. This implies that you cannot have more vCPUs in the guest than the"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da85f559_c9905a98","line":65,"in_reply_to":"fa80f949_d791a79a","updated":"2015-11-09 16:34:49.000000000","message":"hi yes you are correct.\n\nthe l1 data and instuction cache on each physical core is shared with the primary thread/logical core and the hyper thread. when hyper threading is disabled the the primary thread has exclusive access to the l1 data and instruction cache.\ni cannot recall if l1 cache it is explicitly partition or if the cache is freely between the primary and hyper thread sibling.\n\nfor isolate and separate i guess i would have no real objection to decoupling it from whether or not the  host is SMT or Non-SMT.\n\ni agree that if you look at separate/isolate/avoid as a continuum then the decoupling makes sense. in all 3 cases each vCPU is pinned to a different logical core and each one progressivly is more strict as to how much of the phyical cpu resouces it is requesting to be detecated\n\nseparate -\u003e dedicated logical core.\nisolate -\u003e dedicated logical core and reserved/free lcore/hyperthred sibling\navoid -\u003e dedicated physical core.\n\nif we introduce auto (which i also don\u0027t disagree with) i think it would be better to go one step further and make the threading policy a required field with a default value of auto. that way there is always a threading policy specified when a dedicated cpu is requested but existing client don\u0027t need to be updated as the api would use the default value of auto if not supplied.\n\ndoes that seam reasonable?\ni would like to avoid and special db logic to handle null vs auto for the threading policy.","commit_id":"b56d8017000bbdb73286c132917604aed53ee339"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"67af50cfccf053ff5290c0a06cb87c48b24f0d67","unresolved":false,"context_lines":[{"line_number":62,"context_line":" - isolate: if the host has threads, each vCPU will be placed on a"},{"line_number":63,"context_line":"   different core and no vCPUs from other guests will be able to be"},{"line_number":64,"context_line":"   placed on the same core. ie one thread sibling is guaranteed to"},{"line_number":65,"context_line":"   always be unused,"},{"line_number":66,"context_line":" - require: the scheduler will only place the guest on a host which has"},{"line_number":67,"context_line":"   hyperthreads, and vCPUs will be placed on the thread siblings of a single"},{"line_number":68,"context_line":"   core. This implies that you cannot have more vCPUs in the guest than the"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da85f559_54eb3013","line":65,"in_reply_to":"fa80f949_e5f535cb","updated":"2015-11-11 16:41:25.000000000","message":"In response to Stephens very first comment, I want to say it was really a mistake on my part to use the language \"if the host has threads\". There was no intention that the code actually special case hosts with threads vs hosts without threads. When calculating placement all hosts are considered to have hyperthreads, it is simply that sometimes they don\u0027t have any siblings (ie threads-per-core is 1).\n\nIOW, when the host has threads-per-core \u003d\u003d 1, the isolate + separate options are functionally equivalent, as in both cases we (trivially) ensure no vCPUs share a hyperthread siblings because there are no hyperthread siblings.\n\nWe *should* still consider hosts with thread-per-core\u003d\u003d1 as valid targets.","commit_id":"b56d8017000bbdb73286c132917604aed53ee339"},{"author":{"_account_id":8768,"name":"Chris Friesen","email":"chris.friesen@windriver.com","username":"cbf123"},"change_message_id":"3eaca33710beae3c2dccf2ea68a58fc13a5cebbd","unresolved":false,"context_lines":[{"line_number":62,"context_line":" - isolate: if the host has threads, each vCPU will be placed on a"},{"line_number":63,"context_line":"   different core and no vCPUs from other guests will be able to be"},{"line_number":64,"context_line":"   placed on the same core. ie one thread sibling is guaranteed to"},{"line_number":65,"context_line":"   always be unused,"},{"line_number":66,"context_line":" - require: the scheduler will only place the guest on a host which has"},{"line_number":67,"context_line":"   hyperthreads, and vCPUs will be placed on the thread siblings of a single"},{"line_number":68,"context_line":"   core. This implies that you cannot have more vCPUs in the guest than the"}],"source_content_type":"text/x-rst","patch_set":1,"id":"fa80f949_3845a3b2","line":65,"in_reply_to":"fa80f949_e5f535cb","updated":"2015-11-04 22:57:20.000000000","message":"Responding to your points:\n1) To me, \u0027separate\u0027 and \u0027isolate\u0027 could be understood as: \"each host core should map to a single vCPU\", and \u0027isolate\u0027 extends this with \"and nobody else is allowed to share my host cores\".  Both of these are true by definition for a host that doesn\u0027t have SMT, so I would consider the policies to be satisfied when running on a non-SMT host.\n\n2) As an operator, if a thread policy of \"isolate\" is available, does \"avoid\" really buy you enough to set aside hosts for it?  Presumably the operator would already charge more for the \u0027isolate\u0027 option.  I wonder if it might not make sense to remove the \u0027avoid\u0027 option altogether, and allow \u0027isolate\u0027 to run on non-SMT hosts.  Operators could then charge extra for the use of \u0027isolate\u0027, and each vCPU in the instance gets a whole core, whether SMT is enabled or not.\n\n\u0027separate\u0027 is a bit of an odd duck.  Realistically I\u0027m not sure who would ever use it.  Logically it would be satisfied if running on a host with no SMT, but I don\u0027t know if it makes sense to charge more for it when on an SMT host.\n\nLastly, I think we should add an \u0027allow\u0027 policy, which basically represents the behaviour you get if no policy at all is specified.","commit_id":"b56d8017000bbdb73286c132917604aed53ee339"},{"author":{"_account_id":19212,"name":"Antonio López","email":"antonio.lopezgracia@telefonica.com","username":"alogra"},"change_message_id":"99189f42816cfef389c48e27b6e1e86c44e2c75c","unresolved":false,"context_lines":[{"line_number":67,"context_line":"   hyperthreads, and vCPUs will be placed on the thread siblings of a single"},{"line_number":68,"context_line":"   core. This implies that you cannot have more vCPUs in the guest than the"},{"line_number":69,"context_line":"   number of thread siblings on a single core on the host."},{"line_number":70,"context_line":""},{"line_number":71,"context_line":"The image metadata properties will also allow specification of the"},{"line_number":72,"context_line":"threads policy:"},{"line_number":73,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"da85f559_901ebb61","line":70,"updated":"2015-11-10 16:43:22.000000000","message":"[TEF] We agree with the current specification of \"require\". This was called previously \"prefer\", right? I understand \"require\" is more restrictive and clear: if it is not possible to pin the vCPUs to hyperthreads the guest will not be scheduled.","commit_id":"b56d8017000bbdb73286c132917604aed53ee339"},{"author":{"_account_id":19212,"name":"Antonio López","email":"antonio.lopezgracia@telefonica.com","username":"alogra"},"change_message_id":"cf0ab748ab349066882f653393286544bf322fc1","unresolved":false,"context_lines":[{"line_number":67,"context_line":"   hyperthreads, and vCPUs will be placed on the thread siblings of a single"},{"line_number":68,"context_line":"   core. This implies that you cannot have more vCPUs in the guest than the"},{"line_number":69,"context_line":"   number of thread siblings on a single core on the host."},{"line_number":70,"context_line":""},{"line_number":71,"context_line":"The image metadata properties will also allow specification of the"},{"line_number":72,"context_line":"threads policy:"},{"line_number":73,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"da85f559_f14a7625","line":70,"in_reply_to":"da85f559_50010105","updated":"2015-11-11 16:29:20.000000000","message":"I think isolate and separate should both work on non-HT hosts, with a slight difference: isolate should search first for non-HT hosts to minimize the chance of wasting -without using them- threads that could be used by other guests; separate should search first for HT hosts to maximize the possibility of having threads available for other guests with the same policy. We do not identify separate as a required policy for NFV workloads, but it should be kept for completeness of the model and for Cloud workloads. \n\nBy using [TEF] I wanted to mean that this was an opinion agreed with my colleagues in Telefonica, but I think it is confusing in this context and I am not going to use it anymore.","commit_id":"b56d8017000bbdb73286c132917604aed53ee339"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"2f599bab190819092b1c2101deb5039240c2c8a2","unresolved":false,"context_lines":[{"line_number":67,"context_line":"   hyperthreads, and vCPUs will be placed on the thread siblings of a single"},{"line_number":68,"context_line":"   core. This implies that you cannot have more vCPUs in the guest than the"},{"line_number":69,"context_line":"   number of thread siblings on a single core on the host."},{"line_number":70,"context_line":""},{"line_number":71,"context_line":"The image metadata properties will also allow specification of the"},{"line_number":72,"context_line":"threads policy:"},{"line_number":73,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"da85f559_50010105","line":70,"in_reply_to":"da85f559_901ebb61","updated":"2015-11-11 15:37:25.000000000","message":"Yes, exactly. \u0027require\u0027 was a good counterpoint to \u0027avoid\u0027. The discussion now is on the \u0027isolate\u0027 and \u0027separate\u0027 case, and whether they should also require hyperthreads. Have you any input here?\n\nAlso, what do you mean by \u0027[TEF]\u0027?","commit_id":"b56d8017000bbdb73286c132917604aed53ee339"},{"author":{"_account_id":8768,"name":"Chris Friesen","email":"chris.friesen@windriver.com","username":"cbf123"},"change_message_id":"6cf095a7daad30ec1524d7b2871d4d2ece31b72f","unresolved":false,"context_lines":[{"line_number":67,"context_line":"   hyperthreads, and vCPUs will be placed on the thread siblings of a single"},{"line_number":68,"context_line":"   core. This implies that you cannot have more vCPUs in the guest than the"},{"line_number":69,"context_line":"   number of thread siblings on a single core on the host."},{"line_number":70,"context_line":""},{"line_number":71,"context_line":"The image metadata properties will also allow specification of the"},{"line_number":72,"context_line":"threads policy:"},{"line_number":73,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"da85f559_c3a96891","line":70,"in_reply_to":"da85f559_f14a7625","updated":"2015-11-13 03:45:09.000000000","message":"The problem with any \"X should search first for Y\" is that the scheduler is currently binary.  Either a host passes or it doesn\u0027t.\n\nThe weights are where we express a preference between all the valid hosts, and currently we don\u0027t support any complicated weighing algorithms.","commit_id":"b56d8017000bbdb73286c132917604aed53ee339"}]}