)]}'
{"specs/cpuset-container.rst":[{"author":{"_account_id":11536,"name":"hongbin","email":"hongbin034@gmail.com","username":"hongbin"},"change_message_id":"2afbabe61ee4e4e85c134f4ebdb5ec76c436f961","unresolved":false,"context_lines":[{"line_number":59,"context_line":""},{"line_number":60,"context_line":"6. The cpu-policy parameter is specified in conjuction with the vcpus field"},{"line_number":61,"context_line":"for container creation. The number of vcpus shall determine the number of"},{"line_number":62,"context_line":"cpusets requested for dedicated usage."},{"line_number":63,"context_line":""},{"line_number":64,"context_line":"For example:"},{"line_number":65,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"3a461143_4243ff12","line":62,"updated":"2017-01-31 21:40:56.000000000","message":"I guess we also need to extend the scheduler to be host capabilities aware?","commit_id":"1f7e848c43058b92bfd2bee6263bdd5f1c263181"},{"author":{"_account_id":9775,"name":"Sudipta Biswas","email":"sbiswas7@in.ibm.com","username":"sudswas"},"change_message_id":"a26927f3b2a8ad960fd265b90cc84be5f939c29f","unresolved":false,"context_lines":[{"line_number":59,"context_line":""},{"line_number":60,"context_line":"6. The cpu-policy parameter is specified in conjuction with the vcpus field"},{"line_number":61,"context_line":"for container creation. The number of vcpus shall determine the number of"},{"line_number":62,"context_line":"cpusets requested for dedicated usage."},{"line_number":63,"context_line":""},{"line_number":64,"context_line":"For example:"},{"line_number":65,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"3a461143_13b3000b","line":62,"in_reply_to":"3a461143_4243ff12","updated":"2017-02-01 04:38:39.000000000","message":"Updated, if this feature is being used without the scheduler then the API code could directly talk to the host_compute table and decide on dedicated and shared. But given we will have a scheduler in place, we should be making the decision via the scheduler.","commit_id":"1f7e848c43058b92bfd2bee6263bdd5f1c263181"},{"author":{"_account_id":25026,"name":"Bin Zhou","email":"lakerzhou@gmail.com","username":"lakerzhou"},"change_message_id":"c21da4fa13abd9dbcc66a1ad4d5279642641e09f","unresolved":false,"context_lines":[{"line_number":27,"context_line":"This is because, exposing such parameters to the end user breaks"},{"line_number":28,"context_line":"the cloudy model of doing things."},{"line_number":29,"context_line":""},{"line_number":30,"context_line":"Exposing cpusets can be broadly thought of as combination of user policies"},{"line_number":31,"context_line":"and host capabilities."},{"line_number":32,"context_line":""},{"line_number":33,"context_line":"The user policies are applied against compute host capabilities and if it"},{"line_number":34,"context_line":"matches, the user is allowed to perform the CRUD operations on a container."}],"source_content_type":"text/x-rst","patch_set":3,"id":"3a461143_2f34e848","line":31,"range":{"start_line":30,"start_character":0,"end_line":31,"end_character":22},"updated":"2017-02-02 04:04:14.000000000","message":"It is more natural to model this as resource provider and resource consumer. From Ocata release, Nova scheduler will start use placement service for filtering CPU, memory and so on. The placement service is really based on resource provider/consumer model: http://docs.openstack.org/developer/nova/placement.html. \nIMHO, we should follow the new NOVA design.","commit_id":"295979cb41a6d8b495a441c1c0c4dd23476579c3"},{"author":{"_account_id":9775,"name":"Sudipta Biswas","email":"sbiswas7@in.ibm.com","username":"sudswas"},"change_message_id":"ab3f63a7ad35cb8f0ae67b028136a2ed6625cc56","unresolved":false,"context_lines":[{"line_number":27,"context_line":"This is because, exposing such parameters to the end user breaks"},{"line_number":28,"context_line":"the cloudy model of doing things."},{"line_number":29,"context_line":""},{"line_number":30,"context_line":"Exposing cpusets can be broadly thought of as combination of user policies"},{"line_number":31,"context_line":"and host capabilities."},{"line_number":32,"context_line":""},{"line_number":33,"context_line":"The user policies are applied against compute host capabilities and if it"},{"line_number":34,"context_line":"matches, the user is allowed to perform the CRUD operations on a container."}],"source_content_type":"text/x-rst","patch_set":3,"id":"3a461143_622b4934","line":31,"range":{"start_line":30,"start_character":0,"end_line":31,"end_character":22},"in_reply_to":"3a461143_2f34e848","updated":"2017-02-02 17:47:41.000000000","message":"I went through the resource provider/allocations model, while it\u0027s the right way to go about it, we should not be duplicating that logic.\n\nHowever, if you notice below, I have mentioned that we won\u0027t be using the resource tracker in zun, since that gives a wrong view of the system resources.\n\nIt might be possible to design a generic way of representing resources in zun for the time being. But I think our long term goal will entail us to move to the common scheduler framework. So while writing the code, maybe we can be as compatible to that design as possible.\n\nThanks for bringing this up though! It\u0027s a great thought.","commit_id":"295979cb41a6d8b495a441c1c0c4dd23476579c3"},{"author":{"_account_id":25026,"name":"Bin Zhou","email":"lakerzhou@gmail.com","username":"lakerzhou"},"change_message_id":"c21da4fa13abd9dbcc66a1ad4d5279642641e09f","unresolved":false,"context_lines":[{"line_number":49,"context_line":"can be used to obtain the required information. Implement a periodic task"},{"line_number":50,"context_line":"that inventories the host at regular intervals."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"3. Define 2 user cpu-policies called \"dedicated\" and \"shared\". The first"},{"line_number":53,"context_line":"policy signifies that the user wants to use dedicated cpusets for their"},{"line_number":54,"context_line":"workloads. The shared mode is very similar to the default behavior. If unless"},{"line_number":55,"context_line":"specified, the behavior will be defaulted to \"shared\"."},{"line_number":56,"context_line":""},{"line_number":57,"context_line":"4. Write driver methods to provision containers with dedicated cpusets."},{"line_number":58,"context_line":"The logic of \u0027what\u0027 cpusets should be picked up for a given requests lies"}],"source_content_type":"text/x-rst","patch_set":3,"id":"3a461143_2f6ce807","line":55,"range":{"start_line":52,"start_character":3,"end_line":55,"end_character":54},"updated":"2017-02-02 04:04:14.000000000","message":"For CPU pinning, nova is going to model this as nested resource provider: https://specs.openstack.org/openstack/nova-specs/specs/ocata/approved/nested-resource-providers.html.\nIt is actually a great fit to our container context too.","commit_id":"295979cb41a6d8b495a441c1c0c4dd23476579c3"},{"author":{"_account_id":11536,"name":"hongbin","email":"hongbin034@gmail.com","username":"hongbin"},"change_message_id":"893eec1005b0b5ddbcf9b3e7bc44b2cdf0c0f8aa","unresolved":false,"context_lines":[{"line_number":49,"context_line":"can be used to obtain the required information. Implement a periodic task"},{"line_number":50,"context_line":"that inventories the host at regular intervals."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"3. Define 2 user cpu-policies called \"dedicated\" and \"shared\". The first"},{"line_number":53,"context_line":"policy signifies that the user wants to use dedicated cpusets for their"},{"line_number":54,"context_line":"workloads. The shared mode is very similar to the default behavior. If unless"},{"line_number":55,"context_line":"specified, the behavior will be defaulted to \"shared\"."},{"line_number":56,"context_line":""},{"line_number":57,"context_line":"4. Write driver methods to provision containers with dedicated cpusets."},{"line_number":58,"context_line":"The logic of \u0027what\u0027 cpusets should be picked up for a given requests lies"}],"source_content_type":"text/x-rst","patch_set":3,"id":"3a461143_3706e420","line":55,"range":{"start_line":52,"start_character":3,"end_line":55,"end_character":54},"in_reply_to":"3a461143_2f6ce807","updated":"2017-02-02 16:38:20.000000000","message":"@Zhou Bin,\n\nIt seems what #3 proposed is a way to expose this feature at REST API level (i.e. expose it as cpu_policy that can be specified by end-users). On the other side, what you suggested seems to be an internal implementation (not at REST API level).\n\nI think it is possible to have both. That is:\n* Expose cpu_policy for end-users to specify if they want dedicated resources.\n* Implement cpu_policy by using nested resource provider model (possibly after the placement API is available). If cpu_policy is \u0027shared\u0027, schedule the container to the resource provider (i.e. compute host) that supports the \u0027shared\u0027 policy. Similarly, if cpu_policy is \u0027dedicated\u0027, schedule it to host that supports \u0027dedicated\u0027 cpu pinning.\n\nWhat do you think?","commit_id":"295979cb41a6d8b495a441c1c0c4dd23476579c3"},{"author":{"_account_id":9775,"name":"Sudipta Biswas","email":"sbiswas7@in.ibm.com","username":"sudswas"},"change_message_id":"ab3f63a7ad35cb8f0ae67b028136a2ed6625cc56","unresolved":false,"context_lines":[{"line_number":49,"context_line":"can be used to obtain the required information. Implement a periodic task"},{"line_number":50,"context_line":"that inventories the host at regular intervals."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"3. Define 2 user cpu-policies called \"dedicated\" and \"shared\". The first"},{"line_number":53,"context_line":"policy signifies that the user wants to use dedicated cpusets for their"},{"line_number":54,"context_line":"workloads. The shared mode is very similar to the default behavior. If unless"},{"line_number":55,"context_line":"specified, the behavior will be defaulted to \"shared\"."},{"line_number":56,"context_line":""},{"line_number":57,"context_line":"4. Write driver methods to provision containers with dedicated cpusets."},{"line_number":58,"context_line":"The logic of \u0027what\u0027 cpusets should be picked up for a given requests lies"}],"source_content_type":"text/x-rst","patch_set":3,"id":"3a461143_629a69c7","line":55,"range":{"start_line":52,"start_character":3,"end_line":55,"end_character":54},"in_reply_to":"3a461143_3706e420","updated":"2017-02-02 17:47:41.000000000","message":"+1\nThe cpu_policy does not \u0027affect\u0027 the internal implementation of how we want to achieve it.","commit_id":"295979cb41a6d8b495a441c1c0c4dd23476579c3"},{"author":{"_account_id":25026,"name":"Bin Zhou","email":"lakerzhou@gmail.com","username":"lakerzhou"},"change_message_id":"a4f8537b150f4901aa7c6b509214cd504087c5db","unresolved":false,"context_lines":[{"line_number":49,"context_line":"can be used to obtain the required information. Implement a periodic task"},{"line_number":50,"context_line":"that inventories the host at regular intervals."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"3. Define 2 user cpu-policies called \"dedicated\" and \"shared\". The first"},{"line_number":53,"context_line":"policy signifies that the user wants to use dedicated cpusets for their"},{"line_number":54,"context_line":"workloads. The shared mode is very similar to the default behavior. If unless"},{"line_number":55,"context_line":"specified, the behavior will be defaulted to \"shared\"."},{"line_number":56,"context_line":""},{"line_number":57,"context_line":"4. Write driver methods to provision containers with dedicated cpusets."},{"line_number":58,"context_line":"The logic of \u0027what\u0027 cpusets should be picked up for a given requests lies"}],"source_content_type":"text/x-rst","patch_set":3,"id":"3a461143_988184fe","line":55,"range":{"start_line":52,"start_character":3,"end_line":55,"end_character":54},"in_reply_to":"3a461143_629a69c7","updated":"2017-02-02 19:00:33.000000000","message":"I agree. \nIt depending on how we implement the host capabilities. Child host should inherit some host capabilities of its parents.","commit_id":"295979cb41a6d8b495a441c1c0c4dd23476579c3"},{"author":{"_account_id":9775,"name":"Sudipta Biswas","email":"sbiswas7@in.ibm.com","username":"sudswas"},"change_message_id":"e6ba378847f80826fafb8603e7f486854f83a4d8","unresolved":false,"context_lines":[{"line_number":49,"context_line":"can be used to obtain the required information. Implement a periodic task"},{"line_number":50,"context_line":"that inventories the host at regular intervals."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"3. Define 2 user cpu-policies called \"dedicated\" and \"shared\". The first"},{"line_number":53,"context_line":"policy signifies that the user wants to use dedicated cpusets for their"},{"line_number":54,"context_line":"workloads. The shared mode is very similar to the default behavior. If unless"},{"line_number":55,"context_line":"specified, the behavior will be defaulted to \"shared\"."},{"line_number":56,"context_line":""},{"line_number":57,"context_line":"4. Write driver methods to provision containers with dedicated cpusets."},{"line_number":58,"context_line":"The logic of \u0027what\u0027 cpusets should be picked up for a given requests lies"}],"source_content_type":"text/x-rst","patch_set":3,"id":"1a430d35_5518ce1d","line":55,"range":{"start_line":52,"start_character":3,"end_line":55,"end_character":54},"in_reply_to":"3a461143_988184fe","updated":"2017-02-04 03:52:57.000000000","message":"Yeah in the new model, I do have the notion of the parent child relationship.","commit_id":"295979cb41a6d8b495a441c1c0c4dd23476579c3"},{"author":{"_account_id":25026,"name":"Bin Zhou","email":"lakerzhou@gmail.com","username":"lakerzhou"},"change_message_id":"a4f8537b150f4901aa7c6b509214cd504087c5db","unresolved":false,"context_lines":[{"line_number":54,"context_line":"workloads. The shared mode is very similar to the default behavior. If unless"},{"line_number":55,"context_line":"specified, the behavior will be defaulted to \"shared\"."},{"line_number":56,"context_line":""},{"line_number":57,"context_line":"4. Write driver methods to provision containers with dedicated cpusets."},{"line_number":58,"context_line":"The logic of \u0027what\u0027 cpusets should be picked up for a given requests lies"},{"line_number":59,"context_line":"in the control of the zun code and is not exposed to the user."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"5. The cpu-policy parameter is specified in conjuction with the vcpus field"},{"line_number":62,"context_line":"for container creation. The number of vcpus shall determine the number of"}],"source_content_type":"text/x-rst","patch_set":3,"id":"3a461143_93f24505","line":59,"range":{"start_line":57,"start_character":0,"end_line":59,"end_character":62},"updated":"2017-02-02 19:00:33.000000000","message":"If my understanding is correct, you plan to hide the mapping to physical core from user. But it doesn\u0027t work in some real world scenarios. As a operator of an VNF, I want to pin a VNF to exact CPU cores (or socket). For example, I know my NIC connect only to cpu socket 1 (as of the physical setup of host server). My VNF should run on cores in cpu socket 1, otherwise I will get performance degrade. Let\u0027s take the following example: CPU socket 1 has core 1-4; CPU socket 2 has core 5-8. If VNF requires 2 cores, I want to allocate them from core 1-4 instead of core 5-8. With Numa, it can be more complicated.","commit_id":"295979cb41a6d8b495a441c1c0c4dd23476579c3"},{"author":{"_account_id":9775,"name":"Sudipta Biswas","email":"sbiswas7@in.ibm.com","username":"sudswas"},"change_message_id":"e6ba378847f80826fafb8603e7f486854f83a4d8","unresolved":false,"context_lines":[{"line_number":54,"context_line":"workloads. The shared mode is very similar to the default behavior. If unless"},{"line_number":55,"context_line":"specified, the behavior will be defaulted to \"shared\"."},{"line_number":56,"context_line":""},{"line_number":57,"context_line":"4. Write driver methods to provision containers with dedicated cpusets."},{"line_number":58,"context_line":"The logic of \u0027what\u0027 cpusets should be picked up for a given requests lies"},{"line_number":59,"context_line":"in the control of the zun code and is not exposed to the user."},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"5. The cpu-policy parameter is specified in conjuction with the vcpus field"},{"line_number":62,"context_line":"for container creation. The number of vcpus shall determine the number of"}],"source_content_type":"text/x-rst","patch_set":3,"id":"1a430d35_f520da4b","line":59,"range":{"start_line":57,"start_character":0,"end_line":59,"end_character":62},"in_reply_to":"3a461143_93f24505","updated":"2017-02-04 03:52:57.000000000","message":"The NUMA use cases might be simpler with containers. Since there may not be a case like NUMA on the guest should match that of the host\u0027s etc.\n\nHowever, If you are looking at pinning only particular cpusets. How do you do that via NOVA right now? Using the vcpu_pin_set seems like a old way of trying it. I may not be aware of the new way.","commit_id":"295979cb41a6d8b495a441c1c0c4dd23476579c3"},{"author":{"_account_id":25026,"name":"Bin Zhou","email":"lakerzhou@gmail.com","username":"lakerzhou"},"change_message_id":"a4f8537b150f4901aa7c6b509214cd504087c5db","unresolved":false,"context_lines":[{"line_number":67,"context_line":""},{"line_number":68,"context_line":"For example:"},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"zun run -i -t --name test --cpu 4 --cpu-policy dedicated"},{"line_number":71,"context_line":""},{"line_number":72,"context_line":""},{"line_number":73,"context_line":"Design Principles"}],"source_content_type":"text/x-rst","patch_set":3,"id":"3a461143_938245c7","line":70,"range":{"start_line":70,"start_character":0,"end_line":70,"end_character":56},"updated":"2017-02-02 19:00:33.000000000","message":"Is \"4\" the number of vcpus? vcpus are not really identical to user because of NUMA.","commit_id":"295979cb41a6d8b495a441c1c0c4dd23476579c3"},{"author":{"_account_id":11536,"name":"hongbin","email":"hongbin034@gmail.com","username":"hongbin"},"change_message_id":"9ed3d3620ac99fd57e2d0ed900cd540961a0b499","unresolved":false,"context_lines":[{"line_number":23,"context_line":"Problem description"},{"line_number":24,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":25,"context_line":""},{"line_number":26,"context_line":"Exposing cpusets to the cloud users cannot be done in it\u0027s raw form."},{"line_number":27,"context_line":"This is because, exposing such parameters to the end user breaks"},{"line_number":28,"context_line":"the cloudy model of doing things."},{"line_number":29,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a430d35_fee529a8","line":26,"range":{"start_line":26,"start_character":54,"end_line":26,"end_character":58},"updated":"2017-02-03 20:37:55.000000000","message":"its","commit_id":"4eaae8a78007726dc34937d8225ea09e14e3c0c4"},{"author":{"_account_id":11536,"name":"hongbin","email":"hongbin034@gmail.com","username":"hongbin"},"change_message_id":"9ed3d3620ac99fd57e2d0ed900cd540961a0b499","unresolved":false,"context_lines":[{"line_number":36,"context_line":"Proposed change"},{"line_number":37,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":38,"context_line":"1. Piggy back on the work done for host capabilities. Keeping the placement"},{"line_number":39,"context_line":"scheduler work in Nova, zun design it\u0027s data model to be easier to migrate"},{"line_number":40,"context_line":"to the common scheduler framework eventually. This could be achieved by creating"},{"line_number":41,"context_line":"a data model where the compute host capabilities are tracked under a resource"},{"line_number":42,"context_line":"provider called the \"Compute node\" and it\u0027s resource classes along with the"}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a430d35_b893b4a2","line":39,"range":{"start_line":39,"start_character":35,"end_line":39,"end_character":39},"updated":"2017-02-03 20:37:55.000000000","message":"its","commit_id":"4eaae8a78007726dc34937d8225ea09e14e3c0c4"},{"author":{"_account_id":11536,"name":"hongbin","email":"hongbin034@gmail.com","username":"hongbin"},"change_message_id":"9ed3d3620ac99fd57e2d0ed900cd540961a0b499","unresolved":false,"context_lines":[{"line_number":39,"context_line":"scheduler work in Nova, zun design it\u0027s data model to be easier to migrate"},{"line_number":40,"context_line":"to the common scheduler framework eventually. This could be achieved by creating"},{"line_number":41,"context_line":"a data model where the compute host capabilities are tracked under a resource"},{"line_number":42,"context_line":"provider called the \"Compute node\" and it\u0027s resource classes along with the"},{"line_number":43,"context_line":"inventory and the allocations table."},{"line_number":44,"context_line":""},{"line_number":45,"context_line":"More details of this work would be covered on a seperate blueprint:"}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a430d35_d8bec0fe","line":42,"range":{"start_line":42,"start_character":39,"end_line":42,"end_character":43},"updated":"2017-02-03 20:37:55.000000000","message":"its","commit_id":"4eaae8a78007726dc34937d8225ea09e14e3c0c4"},{"author":{"_account_id":11536,"name":"hongbin","email":"hongbin034@gmail.com","username":"hongbin"},"change_message_id":"9ed3d3620ac99fd57e2d0ed900cd540961a0b499","unresolved":false,"context_lines":[{"line_number":96,"context_line":"   - name"},{"line_number":97,"context_line":"   - root_provider"},{"line_number":98,"context_line":"   - parent_provider"},{"line_number":99,"context_line":"   - can_host"},{"line_number":100,"context_line":"   - can_use"},{"line_number":101,"context_line":""},{"line_number":102,"context_line":"- Create a inventories table that consists of the following fields:"}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a430d35_381d0474","line":99,"range":{"start_line":99,"start_character":5,"end_line":99,"end_character":13},"updated":"2017-02-03 20:37:55.000000000","message":"Perhaps it worth to explain what is \u0027can_host\u0027 and \u0027can_use\u0027.","commit_id":"4eaae8a78007726dc34937d8225ea09e14e3c0c4"},{"author":{"_account_id":25026,"name":"Bin Zhou","email":"lakerzhou@gmail.com","username":"lakerzhou"},"change_message_id":"03d34709f7fec246c067f939dcfc4f9b0fee93f0","unresolved":false,"context_lines":[{"line_number":99,"context_line":"   - can_host"},{"line_number":100,"context_line":"   - can_use"},{"line_number":101,"context_line":""},{"line_number":102,"context_line":"- Create a inventories table that consists of the following fields:"},{"line_number":103,"context_line":"   - id"},{"line_number":104,"context_line":"   - resource_provider_id"},{"line_number":105,"context_line":"   - resource_class_id"},{"line_number":106,"context_line":"   - total"},{"line_number":107,"context_line":"   - reserved"},{"line_number":108,"context_line":"   - max_size"},{"line_number":109,"context_line":"   - min_size"},{"line_number":110,"context_line":"   - step_size"},{"line_number":111,"context_line":"   - is_nested"},{"line_number":112,"context_line":"   - blob"},{"line_number":113,"context_line":""},{"line_number":114,"context_line":"For a flat inventory of resources like Total CPUs of a system, the"},{"line_number":115,"context_line":"value for is_nested shall be False. However for a resource which is"}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a430d35_2284d3d2","line":112,"range":{"start_line":102,"start_character":2,"end_line":112,"end_character":9},"updated":"2017-02-06 05:17:27.000000000","message":"Is the inventory for CPU only? A more generic resource inventory class will be certainly helpful for future resources such as SR-IOV. If it is not possible to abstract this, there will be parallel tables like SR-IOV inventory, GPU inventory and so on. I am not recommend to do the common inventory class at all, but please keep this in mind.","commit_id":"4eaae8a78007726dc34937d8225ea09e14e3c0c4"},{"author":{"_account_id":9775,"name":"Sudipta Biswas","email":"sbiswas7@in.ibm.com","username":"sudswas"},"change_message_id":"0680d5c4bf0dc80535668602ea85a668f3e8ecd1","unresolved":false,"context_lines":[{"line_number":99,"context_line":"   - can_host"},{"line_number":100,"context_line":"   - can_use"},{"line_number":101,"context_line":""},{"line_number":102,"context_line":"- Create a inventories table that consists of the following fields:"},{"line_number":103,"context_line":"   - id"},{"line_number":104,"context_line":"   - resource_provider_id"},{"line_number":105,"context_line":"   - resource_class_id"},{"line_number":106,"context_line":"   - total"},{"line_number":107,"context_line":"   - reserved"},{"line_number":108,"context_line":"   - max_size"},{"line_number":109,"context_line":"   - min_size"},{"line_number":110,"context_line":"   - step_size"},{"line_number":111,"context_line":"   - is_nested"},{"line_number":112,"context_line":"   - blob"},{"line_number":113,"context_line":""},{"line_number":114,"context_line":"For a flat inventory of resources like Total CPUs of a system, the"},{"line_number":115,"context_line":"value for is_nested shall be False. However for a resource which is"}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a430d35_dbe3f631","line":112,"range":{"start_line":102,"start_character":2,"end_line":112,"end_character":9},"in_reply_to":"1a430d35_2284d3d2","updated":"2017-02-06 07:14:48.000000000","message":"I didn\u0027t think of only cpusets while writing this. Do you see a reason why the SR-IOV cannot be inventoried with the above fields?","commit_id":"4eaae8a78007726dc34937d8225ea09e14e3c0c4"},{"author":{"_account_id":25026,"name":"Bin Zhou","email":"lakerzhou@gmail.com","username":"lakerzhou"},"change_message_id":"3c407fb7c2ed59b58113cb4278aa882b880ca913","unresolved":false,"context_lines":[{"line_number":99,"context_line":"   - can_host"},{"line_number":100,"context_line":"   - can_use"},{"line_number":101,"context_line":""},{"line_number":102,"context_line":"- Create a inventories table that consists of the following fields:"},{"line_number":103,"context_line":"   - id"},{"line_number":104,"context_line":"   - resource_provider_id"},{"line_number":105,"context_line":"   - resource_class_id"},{"line_number":106,"context_line":"   - total"},{"line_number":107,"context_line":"   - reserved"},{"line_number":108,"context_line":"   - max_size"},{"line_number":109,"context_line":"   - min_size"},{"line_number":110,"context_line":"   - step_size"},{"line_number":111,"context_line":"   - is_nested"},{"line_number":112,"context_line":"   - blob"},{"line_number":113,"context_line":""},{"line_number":114,"context_line":"For a flat inventory of resources like Total CPUs of a system, the"},{"line_number":115,"context_line":"value for is_nested shall be False. However for a resource which is"}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a430d35_bb241860","line":112,"range":{"start_line":102,"start_character":2,"end_line":112,"end_character":9},"in_reply_to":"1a430d35_dbe3f631","updated":"2017-02-08 19:58:43.000000000","message":"Thanks for the confirmation. SR-IOV is simpler than CPU, given VFs cannot be shared.\nThere are more and more interests on supporting GPUs and FPGA passthrough to guest. There are still lots of unknowns, but it is great to know you already trying to make it generic.","commit_id":"4eaae8a78007726dc34937d8225ea09e14e3c0c4"},{"author":{"_account_id":11536,"name":"hongbin","email":"hongbin034@gmail.com","username":"hongbin"},"change_message_id":"9ed3d3620ac99fd57e2d0ed900cd540961a0b499","unresolved":false,"context_lines":[{"line_number":116,"context_line":"nested, like cpuset is a part of a NUMA node and cpuset is a set"},{"line_number":117,"context_line":"instead of an integer, the is_nested field is used."},{"line_number":118,"context_line":""},{"line_number":119,"context_line":"If the is_nested field is used, the value of the nested field is"},{"line_number":120,"context_line":"read from the blob while total represents a len(blob) in majority"},{"line_number":121,"context_line":"of the cases."},{"line_number":122,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a430d35_b88654b7","line":119,"range":{"start_line":119,"start_character":49,"end_line":119,"end_character":55},"updated":"2017-02-03 20:37:55.000000000","message":"It looks there is no \u0027nested\u0027 field. Need a clarification?","commit_id":"4eaae8a78007726dc34937d8225ea09e14e3c0c4"},{"author":{"_account_id":9775,"name":"Sudipta Biswas","email":"sbiswas7@in.ibm.com","username":"sudswas"},"change_message_id":"0680d5c4bf0dc80535668602ea85a668f3e8ecd1","unresolved":false,"context_lines":[{"line_number":116,"context_line":"nested, like cpuset is a part of a NUMA node and cpuset is a set"},{"line_number":117,"context_line":"instead of an integer, the is_nested field is used."},{"line_number":118,"context_line":""},{"line_number":119,"context_line":"If the is_nested field is used, the value of the nested field is"},{"line_number":120,"context_line":"read from the blob while total represents a len(blob) in majority"},{"line_number":121,"context_line":"of the cases."},{"line_number":122,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a430d35_9bc08e7e","line":119,"range":{"start_line":119,"start_character":49,"end_line":119,"end_character":55},"in_reply_to":"1a430d35_b88654b7","updated":"2017-02-06 07:14:48.000000000","message":"Yeah will correct this in the new version.","commit_id":"4eaae8a78007726dc34937d8225ea09e14e3c0c4"},{"author":{"_account_id":25026,"name":"Bin Zhou","email":"lakerzhou@gmail.com","username":"lakerzhou"},"change_message_id":"03d34709f7fec246c067f939dcfc4f9b0fee93f0","unresolved":false,"context_lines":[{"line_number":133,"context_line":" is_nested \u003d True"},{"line_number":134,"context_line":" blob \u003d [1, 2, 5]"},{"line_number":135,"context_line":""},{"line_number":136,"context_line":"The reason why we have one entry less in the blob is because the cpuset"},{"line_number":137,"context_line":"that is reserved, is removed from the blob."},{"line_number":138,"context_line":""},{"line_number":139,"context_line":"The relationship between the inventory data and the resource provider is"},{"line_number":140,"context_line":"then established by looking at the resource_provider table:"}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a430d35_e2b64bdb","line":137,"range":{"start_line":136,"start_character":0,"end_line":137,"end_character":43},"updated":"2017-02-06 05:17:27.000000000","message":"Does the blob field indicate the reserved CPU sets? If so, the name of cpusets_reserved seems better. Please refer to a recent change in NOVA: https://github.com/openstack/nova/commit/af08620e96907b00e2890b132c8de0ebf61cc0cf","commit_id":"4eaae8a78007726dc34937d8225ea09e14e3c0c4"},{"author":{"_account_id":25026,"name":"Bin Zhou","email":"lakerzhou@gmail.com","username":"lakerzhou"},"change_message_id":"3c407fb7c2ed59b58113cb4278aa882b880ca913","unresolved":false,"context_lines":[{"line_number":133,"context_line":" is_nested \u003d True"},{"line_number":134,"context_line":" blob \u003d [1, 2, 5]"},{"line_number":135,"context_line":""},{"line_number":136,"context_line":"The reason why we have one entry less in the blob is because the cpuset"},{"line_number":137,"context_line":"that is reserved, is removed from the blob."},{"line_number":138,"context_line":""},{"line_number":139,"context_line":"The relationship between the inventory data and the resource provider is"},{"line_number":140,"context_line":"then established by looking at the resource_provider table:"}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a430d35_1be7ec11","line":137,"range":{"start_line":136,"start_character":0,"end_line":137,"end_character":43},"in_reply_to":"1a430d35_bb562ad0","updated":"2017-02-08 19:58:43.000000000","message":"Right. cpusets_reserved will be too specific.","commit_id":"4eaae8a78007726dc34937d8225ea09e14e3c0c4"},{"author":{"_account_id":9775,"name":"Sudipta Biswas","email":"sbiswas7@in.ibm.com","username":"sudswas"},"change_message_id":"0680d5c4bf0dc80535668602ea85a668f3e8ecd1","unresolved":false,"context_lines":[{"line_number":133,"context_line":" is_nested \u003d True"},{"line_number":134,"context_line":" blob \u003d [1, 2, 5]"},{"line_number":135,"context_line":""},{"line_number":136,"context_line":"The reason why we have one entry less in the blob is because the cpuset"},{"line_number":137,"context_line":"that is reserved, is removed from the blob."},{"line_number":138,"context_line":""},{"line_number":139,"context_line":"The relationship between the inventory data and the resource provider is"},{"line_number":140,"context_line":"then established by looking at the resource_provider table:"}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a430d35_bb562ad0","line":137,"range":{"start_line":136,"start_character":0,"end_line":137,"end_character":43},"in_reply_to":"1a430d35_e2b64bdb","updated":"2017-02-06 07:14:48.000000000","message":"The cpusets_reserved would make it specific for the cpu case right? What if there are other resources like VFs that need reservation?","commit_id":"4eaae8a78007726dc34937d8225ea09e14e3c0c4"},{"author":{"_account_id":25026,"name":"Bin Zhou","email":"lakerzhou@gmail.com","username":"lakerzhou"},"change_message_id":"03d34709f7fec246c067f939dcfc4f9b0fee93f0","unresolved":false,"context_lines":[{"line_number":198,"context_line":"   select parent_provider_id from resource_provider where"},{"line_number":199,"context_line":"   resource_provider_id\u003dyyy"},{"line_number":200,"context_line":""},{"line_number":201,"context_line":"4. The above would return the numa node, if can_host is set to False"},{"line_number":202,"context_line":"   keep going back wards till you reach a parent_provider_id that can"},{"line_number":203,"context_line":"   host and update the allocations table against each resource_provider."},{"line_number":204,"context_line":""},{"line_number":205,"context_line":"There has to be a certain set of validations to assert whether an"},{"line_number":206,"context_line":"allocation should update multiple resource providers."}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a430d35_62b17b8f","line":203,"range":{"start_line":201,"start_character":0,"end_line":203,"end_character":72},"updated":"2017-02-06 05:17:27.000000000","message":"If the required cpu-policy is dedicated, then the cpu-set should be removed from all the parents\u0027 available resources. In the case of \"shared\", is there a configurable CPU over-subscribe rate for container also? How do you update the parents?","commit_id":"4eaae8a78007726dc34937d8225ea09e14e3c0c4"},{"author":{"_account_id":9775,"name":"Sudipta Biswas","email":"sbiswas7@in.ibm.com","username":"sudswas"},"change_message_id":"0680d5c4bf0dc80535668602ea85a668f3e8ecd1","unresolved":false,"context_lines":[{"line_number":198,"context_line":"   select parent_provider_id from resource_provider where"},{"line_number":199,"context_line":"   resource_provider_id\u003dyyy"},{"line_number":200,"context_line":""},{"line_number":201,"context_line":"4. The above would return the numa node, if can_host is set to False"},{"line_number":202,"context_line":"   keep going back wards till you reach a parent_provider_id that can"},{"line_number":203,"context_line":"   host and update the allocations table against each resource_provider."},{"line_number":204,"context_line":""},{"line_number":205,"context_line":"There has to be a certain set of validations to assert whether an"},{"line_number":206,"context_line":"allocation should update multiple resource providers."}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a430d35_fbd35236","line":203,"range":{"start_line":201,"start_character":0,"end_line":203,"end_character":72},"in_reply_to":"1a430d35_62b17b8f","updated":"2017-02-06 07:14:48.000000000","message":"Very good question.  I think we have to include a allocation_ratio field above against the inventory table to allow the admins to decide on how much cpu over-commiting is allowed.\n\nI will clarify the point on how the updation should happen when a resource is completely consumed under a given parent in the next patchset.","commit_id":"4eaae8a78007726dc34937d8225ea09e14e3c0c4"},{"author":{"_account_id":25026,"name":"Bin Zhou","email":"lakerzhou@gmail.com","username":"lakerzhou"},"change_message_id":"c7d17e6d244e3c8e1e8a76333651e90ed015daf5","unresolved":false,"context_lines":[{"line_number":97,"context_line":""},{"line_number":98,"context_line":"How does the scheduling happen in the backend?"},{"line_number":99,"context_line":""},{"line_number":100,"context_line":"To begin with, ZUN won\u0027t support a compute host to have both dedicated"},{"line_number":101,"context_line":"and a shared policy at the same time applicable to it. If a host supports"},{"line_number":102,"context_line":"the dedicated policy, it won\u0027t be able to host containers with shared policy."},{"line_number":103,"context_line":""},{"line_number":104,"context_line":"We need to find a way to determine if a given compute node qualifies to host"},{"line_number":105,"context_line":"only dedicated cpusets."}],"source_content_type":"text/x-rst","patch_set":6,"id":"1a430d35_4cb1e0e0","line":102,"range":{"start_line":100,"start_character":0,"end_line":102,"end_character":77},"updated":"2017-02-07 04:20:10.000000000","message":"This is new behavior than NOVA\u0027s. Is there any particular reason to make such assumption? My experience with NOVA was the default scheduling policy is shared. I understand the new NOVA nest-resource-provider proposed this change. However, it will greatly limited the flexibility of using CPU resource. In the following, I am trying to describe how it can work in my mind. Please let me know if I am wrong.\nAssuming a server has 16 cores (pCPU 0-15):\ntwo configuration:\n1) on host isolcpu\u003d0-7 (this is reserved pCPUs for pinning)\n2) nova.conf: vcpu_pin_set \u003d \"4-12,^8,15\", meaning cpuset \"4-12,^8,15\" will be added to all the XML configuration file for libvirt. Should vcpu_pin_set always be a subset of isolcpu? I think it is not. But I don\u0027t have concrete proof.\nthen:\nPinning CPU pool on host: (0-3)\nFlooting CPU on host: (8-15)\nPinning CPU pool on guests: (4-7)\nFloating CPU pool on guests: (9-12, 15)","commit_id":"34fb0affdb44010d75d3ca5ee6e4454425bec0dc"},{"author":{"_account_id":25026,"name":"Bin Zhou","email":"lakerzhou@gmail.com","username":"lakerzhou"},"change_message_id":"71661a7d6b420282114cd5d361adddc274ea4b75","unresolved":false,"context_lines":[{"line_number":97,"context_line":""},{"line_number":98,"context_line":"How does the scheduling happen in the backend?"},{"line_number":99,"context_line":""},{"line_number":100,"context_line":"To begin with, ZUN won\u0027t support a compute host to have both dedicated"},{"line_number":101,"context_line":"and a shared policy at the same time applicable to it. If a host supports"},{"line_number":102,"context_line":"the dedicated policy, it won\u0027t be able to host containers with shared policy."},{"line_number":103,"context_line":""},{"line_number":104,"context_line":"We need to find a way to determine if a given compute node qualifies to host"},{"line_number":105,"context_line":"only dedicated cpusets."}],"source_content_type":"text/x-rst","patch_set":6,"id":"1a430d35_c039f5f3","line":102,"range":{"start_line":100,"start_character":0,"end_line":102,"end_character":77},"in_reply_to":"1a430d35_0fb50530","updated":"2017-02-08 19:58:22.000000000","message":"\"CPU_PIN_SET\" is in current NOVA. For Zun, I propose to use a configuration parameter such as CPU_POOL on each compute host. This parameter specifies which pCPU can be used for containers. Some of the pCPUs are floating, the others are isolated from scheduler (isolcpu).\nif a pCPU in both CPU_POOL and isolcpu, container can pin to this pCPU;\nif a pCPU in CPU_POOL but not isolcpu, container can use this pCPU in shared mode;\nif a pCPU is not in CPU_POOL, container cannot use the pCPU.\nDoes this make sense to you?","commit_id":"34fb0affdb44010d75d3ca5ee6e4454425bec0dc"},{"author":{"_account_id":9775,"name":"Sudipta Biswas","email":"sbiswas7@in.ibm.com","username":"sudswas"},"change_message_id":"e7ca917887b384491d5297e705740074f93341cc","unresolved":false,"context_lines":[{"line_number":97,"context_line":""},{"line_number":98,"context_line":"How does the scheduling happen in the backend?"},{"line_number":99,"context_line":""},{"line_number":100,"context_line":"To begin with, ZUN won\u0027t support a compute host to have both dedicated"},{"line_number":101,"context_line":"and a shared policy at the same time applicable to it. If a host supports"},{"line_number":102,"context_line":"the dedicated policy, it won\u0027t be able to host containers with shared policy."},{"line_number":103,"context_line":""},{"line_number":104,"context_line":"We need to find a way to determine if a given compute node qualifies to host"},{"line_number":105,"context_line":"only dedicated cpusets."}],"source_content_type":"text/x-rst","patch_set":6,"id":"1a430d35_1880e48a","line":102,"range":{"start_line":100,"start_character":0,"end_line":102,"end_character":77},"in_reply_to":"1a430d35_0fb50530","updated":"2017-02-09 17:07:54.000000000","message":"I missed a point while replying on the above comment.\nIf you have some cpus which are floating in nature and some which are pinned, you need to ensure somehow that the linux kernel does not schedule processes on the pinned CPUs.\n\nIn such a case, a shared policy has to always start scheduling containers on a given set of cpusets which are meant for floating across processes.\nSomewhat restricting them to only the floating set.\n\nCan you think of a better way to deal with it?","commit_id":"34fb0affdb44010d75d3ca5ee6e4454425bec0dc"},{"author":{"_account_id":9775,"name":"Sudipta Biswas","email":"sbiswas7@in.ibm.com","username":"sudswas"},"change_message_id":"1bfe73f84c4ae21851012d3f8ec89cb19f8fed8c","unresolved":false,"context_lines":[{"line_number":97,"context_line":""},{"line_number":98,"context_line":"How does the scheduling happen in the backend?"},{"line_number":99,"context_line":""},{"line_number":100,"context_line":"To begin with, ZUN won\u0027t support a compute host to have both dedicated"},{"line_number":101,"context_line":"and a shared policy at the same time applicable to it. If a host supports"},{"line_number":102,"context_line":"the dedicated policy, it won\u0027t be able to host containers with shared policy."},{"line_number":103,"context_line":""},{"line_number":104,"context_line":"We need to find a way to determine if a given compute node qualifies to host"},{"line_number":105,"context_line":"only dedicated cpusets."}],"source_content_type":"text/x-rst","patch_set":6,"id":"1a430d35_0fb50530","line":102,"range":{"start_line":100,"start_character":0,"end_line":102,"end_character":77},"in_reply_to":"1a430d35_4cb1e0e0","updated":"2017-02-07 07:10:21.000000000","message":"You are talking about the old NOVA behavior.\nI remember fixing a bug around few releases back:\n\nhttps://review.openstack.org/#/c/173187/\n\nIf you are recommending a similar implementation, it could be achieved like this:\n\n1. Create a resource_class called CPU_PIN_SET and define the cpus that are dedicated for the cpuset pinning.\n2. Map the user policy for dedicated to always pick cpusets from the CPU_PIN_SET and not from the normal CPUSET resource class.\n3. What constitutes the CPU_PIN_SET for a given host?\n\nFor the first draft we could make it a config parameter in zun.conf and have the compute agent call the API to create a inventory against this.\n\nThoughts?","commit_id":"34fb0affdb44010d75d3ca5ee6e4454425bec0dc"},{"author":{"_account_id":9775,"name":"Sudipta Biswas","email":"sbiswas7@in.ibm.com","username":"sudswas"},"change_message_id":"22fcaa6c90263f14de662fce227506ad64690387","unresolved":false,"context_lines":[{"line_number":97,"context_line":""},{"line_number":98,"context_line":"How does the scheduling happen in the backend?"},{"line_number":99,"context_line":""},{"line_number":100,"context_line":"To begin with, ZUN won\u0027t support a compute host to have both dedicated"},{"line_number":101,"context_line":"and a shared policy at the same time applicable to it. If a host supports"},{"line_number":102,"context_line":"the dedicated policy, it won\u0027t be able to host containers with shared policy."},{"line_number":103,"context_line":""},{"line_number":104,"context_line":"We need to find a way to determine if a given compute node qualifies to host"},{"line_number":105,"context_line":"only dedicated cpusets."}],"source_content_type":"text/x-rst","patch_set":6,"id":"1a430d35_b77203af","line":102,"range":{"start_line":100,"start_character":0,"end_line":102,"end_character":77},"in_reply_to":"1a430d35_c039f5f3","updated":"2017-02-09 17:11:48.000000000","message":"The comment i have made below was done yesterday and I too forgot to post it.\n\nI am not sure how your above proposal will work simply because, if you have isolcpus, they are isolated from the kernel to schedule anything on.\n\nHowever, if you are referring to isolcpus w.r.t Zun\u0027s scheduler - then please read my comment below :-)\n\nAlso, if you check the latest spec # 7 - you would see that i have proposed a mechanism to do this work. Just that we will have to always using the cpusets parameter during  any container run.","commit_id":"34fb0affdb44010d75d3ca5ee6e4454425bec0dc"},{"author":{"_account_id":25026,"name":"Bin Zhou","email":"lakerzhou@gmail.com","username":"lakerzhou"},"change_message_id":"71661a7d6b420282114cd5d361adddc274ea4b75","unresolved":false,"context_lines":[{"line_number":156,"context_line":"   - resource_class_id"},{"line_number":157,"context_line":"   - total"},{"line_number":158,"context_line":"   - reserved"},{"line_number":159,"context_line":"   - allocation_ratio"},{"line_number":160,"context_line":"   - max_size"},{"line_number":161,"context_line":"   - min_size"},{"line_number":162,"context_line":"   - step_size"}],"source_content_type":"text/x-rst","patch_set":6,"id":"1a430d35_4b589856","line":159,"range":{"start_line":159,"start_character":5,"end_line":159,"end_character":21},"updated":"2017-02-08 19:58:22.000000000","message":"is this allocation_ratio a global constant cross the whole cloud, or per compute node? Can you please explain it a bit more.","commit_id":"34fb0affdb44010d75d3ca5ee6e4454425bec0dc"},{"author":{"_account_id":11536,"name":"hongbin","email":"hongbin034@gmail.com","username":"hongbin"},"change_message_id":"998b7d87626fbb21b7d038275b47e53cb5bd8f35","unresolved":false,"context_lines":[{"line_number":208,"context_line":"of cpus as `4`. Following operation should happen from the zun scheduler"},{"line_number":209,"context_line":"code:"},{"line_number":210,"context_line":""},{"line_number":211,"context_line":"1. Obtain the resource_class_id for cpusets. Let this be \u0027xxx\u0027."},{"line_number":212,"context_line":""},{"line_number":213,"context_line":"2. Select resource_provider_id from inventory where resource_class_id\u003dxxx"},{"line_number":214,"context_line":"   and total \u003e\u003d 4. Subtract this value with any corresponding entry for"}],"source_content_type":"text/x-rst","patch_set":6,"id":"1a430d35_427a96f2","line":211,"range":{"start_line":211,"start_character":3,"end_line":211,"end_character":9},"updated":"2017-02-06 21:01:52.000000000","message":"Just want to confirm how. Obtain it from a DB table or a hard-coded constant?","commit_id":"34fb0affdb44010d75d3ca5ee6e4454425bec0dc"},{"author":{"_account_id":25026,"name":"Bin Zhou","email":"lakerzhou@gmail.com","username":"lakerzhou"},"change_message_id":"31f24ee3dfc0d2214e9c79eb3ef6e049443a8837","unresolved":false,"context_lines":[{"line_number":116,"context_line":"In order to support both the cpu policies simultaneously on a given host the"},{"line_number":117,"context_line":"following actions have to be performed:"},{"line_number":118,"context_line":""},{"line_number":119,"context_line":"1. Create a resource_class called CPUSETS_PIN."},{"line_number":120,"context_line":"2. Allow users to specify which pcpus should be kept aside for pinning"},{"line_number":121,"context_line":"on a given host. (Numbers of cpus would be sufficient)."},{"line_number":122,"context_line":"3. While the inventory records are being populated, read the value of"}],"source_content_type":"text/x-rst","patch_set":7,"id":"1a430d35_7d6c2808","line":119,"range":{"start_line":119,"start_character":34,"end_line":119,"end_character":45},"updated":"2017-02-08 20:41:21.000000000","message":"Please check my comments on patch 6. Sorry I did not post it in time. I think we need a configuration input, e.g. something like \"ALLOWED_CPUS\" or \"CPU_POOL\"  from user to specify which CPUs are allowed for containers to use. Some of these CPUs can be shared among containers.  \nAFAIK, there are two shell commands to check the cpu setting:\n\"lscpu\" to check all the system CPUs;\n\"cat /proc/$$/status|grep Cpus_allowed_list\"  to check all shared CPUs (e.g. excluding lsolcpu).\nTogether with the new configuration input, you should be figure out which CPU can be used for shared or for dedicated usage.","commit_id":"12e0aa975a0f3c25a801e2096da4f8504c1166ca"},{"author":{"_account_id":25026,"name":"Bin Zhou","email":"lakerzhou@gmail.com","username":"lakerzhou"},"change_message_id":"4a519ebfd064ae44c1e53bb8174356638454ece2","unresolved":false,"context_lines":[{"line_number":228,"context_line":"   allocations table. Qualify a given resource_provider if the value after"},{"line_number":229,"context_line":"   subtraction is \u003e\u003d4."},{"line_number":230,"context_line":""},{"line_number":231,"context_line":"3. Choose a cpuset resource from the above returned list. The resource"},{"line_number":232,"context_line":"   provider of this cpuset will be the NUMA node. Let this be \u0027yyy\u0027."},{"line_number":233,"context_line":""},{"line_number":234,"context_line":"   Query the parent COMPUTE node of this cpuset provider."},{"line_number":235,"context_line":""}],"source_content_type":"text/x-rst","patch_set":7,"id":"fa31d9ce_50064421","line":232,"range":{"start_line":231,"start_character":58,"end_line":232,"end_character":50},"updated":"2017-02-09 21:56:40.000000000","message":"I missed the NUMA part. Will all cpus be allocated from same NUMA node by this design? In same cases, an application has specific NUMA requirement, especially if not all requested CPU can be fitted into same NUMA node. I think it is ok to leave NUMA out of this design, and have future blueprint take care of it. What do you think?","commit_id":"12e0aa975a0f3c25a801e2096da4f8504c1166ca"},{"author":{"_account_id":9775,"name":"Sudipta Biswas","email":"sbiswas7@in.ibm.com","username":"sudswas"},"change_message_id":"aa5b99b1a4b49656a6a10fc46db70b989b89b59b","unresolved":false,"context_lines":[{"line_number":228,"context_line":"   allocations table. Qualify a given resource_provider if the value after"},{"line_number":229,"context_line":"   subtraction is \u003e\u003d4."},{"line_number":230,"context_line":""},{"line_number":231,"context_line":"3. Choose a cpuset resource from the above returned list. The resource"},{"line_number":232,"context_line":"   provider of this cpuset will be the NUMA node. Let this be \u0027yyy\u0027."},{"line_number":233,"context_line":""},{"line_number":234,"context_line":"   Query the parent COMPUTE node of this cpuset provider."},{"line_number":235,"context_line":""}],"source_content_type":"text/x-rst","patch_set":7,"id":"fa31d9ce_4af5a881","line":232,"range":{"start_line":231,"start_character":58,"end_line":232,"end_character":50},"in_reply_to":"fa31d9ce_50064421","updated":"2017-02-10 03:09:15.000000000","message":"Hi Bin,\n\nI kind of differ with that point here. The reason why we SHOULD do NUMA to start with is because, allocating CPUs across NUMA nodes isn\u0027t quite a great thing from performance per say  - given you would be doing cross socket memory access and what not.\n\nWhile I agree with you that there is a possibility that some requests may not get served because of fragmentation of cpusets across NUMA nodes, esp in the floating case. \n\nHence let\u0027s do this instead, in case of \u0027shared\u0027 cpu-policy - there will not be any NUMA node restriction. The cpus can be chosen across NUMA nodes for these floating CPUs. The NUMA case only applies in case of dedicated. \n\nThoughts?","commit_id":"12e0aa975a0f3c25a801e2096da4f8504c1166ca"},{"author":{"_account_id":12175,"name":"Eli Qiao","email":"qiaoliyong@gmail.com","username":"Eli"},"change_message_id":"4003adb8df1bfa9f305e81bf2fcc44ef090669ed","unresolved":false,"context_lines":[{"line_number":41,"context_line":"https://blueprints.launchpad.net/zun/+spec/expose-host-capabilities"},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"2. Hydrate the schema with information obtained via calling driver specific"},{"line_number":44,"context_line":"methods that obtain the details of a host inventory. For cpusets, lscpu"},{"line_number":45,"context_line":"can be used to obtain the required information. Implement a periodic task"},{"line_number":46,"context_line":"that inventories the host at regular intervals."},{"line_number":47,"context_line":""}],"source_content_type":"text/x-rst","patch_set":8,"id":"da36d5c6_c55a5650","line":44,"range":{"start_line":44,"start_character":66,"end_line":44,"end_character":71},"updated":"2017-02-28 07:06:27.000000000","message":"do you mean this?\n`lscpu -p`","commit_id":"4aad6ba3bb7ee3275da2844f3bfec8f1e6624b04"},{"author":{"_account_id":9775,"name":"Sudipta Biswas","email":"sbiswas7@in.ibm.com","username":"sudswas"},"change_message_id":"b7b71931e79d915ee7c40a146a040d2e3e062f0c","unresolved":false,"context_lines":[{"line_number":41,"context_line":"https://blueprints.launchpad.net/zun/+spec/expose-host-capabilities"},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"2. Hydrate the schema with information obtained via calling driver specific"},{"line_number":44,"context_line":"methods that obtain the details of a host inventory. For cpusets, lscpu"},{"line_number":45,"context_line":"can be used to obtain the required information. Implement a periodic task"},{"line_number":46,"context_line":"that inventories the host at regular intervals."},{"line_number":47,"context_line":""}],"source_content_type":"text/x-rst","patch_set":8,"id":"ba2be162_6b74079c","line":44,"range":{"start_line":44,"start_character":66,"end_line":44,"end_character":71},"in_reply_to":"da36d5c6_c55a5650","updated":"2017-04-05 08:47:49.000000000","message":"Yeah. Will correct.","commit_id":"4aad6ba3bb7ee3275da2844f3bfec8f1e6624b04"},{"author":{"_account_id":12175,"name":"Eli Qiao","email":"qiaoliyong@gmail.com","username":"Eli"},"change_message_id":"4003adb8df1bfa9f305e81bf2fcc44ef090669ed","unresolved":false,"context_lines":[{"line_number":42,"context_line":""},{"line_number":43,"context_line":"2. Hydrate the schema with information obtained via calling driver specific"},{"line_number":44,"context_line":"methods that obtain the details of a host inventory. For cpusets, lscpu"},{"line_number":45,"context_line":"can be used to obtain the required information. Implement a periodic task"},{"line_number":46,"context_line":"that inventories the host at regular intervals."},{"line_number":47,"context_line":""},{"line_number":48,"context_line":"3. Define 2 user cpu-policies called \"dedicated\" and \"shared\". The first"}],"source_content_type":"text/x-rst","patch_set":8,"id":"da36d5c6_6511aa5d","line":45,"range":{"start_line":45,"start_character":58,"end_line":45,"end_character":73},"updated":"2017-02-28 07:06:27.000000000","message":"hmm.. I don\u0027t understand what does the periodic task do?\n\nwe can maintainer cpu set usage in zun\u0027s compute node object.","commit_id":"4aad6ba3bb7ee3275da2844f3bfec8f1e6624b04"},{"author":{"_account_id":9775,"name":"Sudipta Biswas","email":"sbiswas7@in.ibm.com","username":"sudswas"},"change_message_id":"b7b71931e79d915ee7c40a146a040d2e3e062f0c","unresolved":false,"context_lines":[{"line_number":42,"context_line":""},{"line_number":43,"context_line":"2. Hydrate the schema with information obtained via calling driver specific"},{"line_number":44,"context_line":"methods that obtain the details of a host inventory. For cpusets, lscpu"},{"line_number":45,"context_line":"can be used to obtain the required information. Implement a periodic task"},{"line_number":46,"context_line":"that inventories the host at regular intervals."},{"line_number":47,"context_line":""},{"line_number":48,"context_line":"3. Define 2 user cpu-policies called \"dedicated\" and \"shared\". The first"}],"source_content_type":"text/x-rst","patch_set":8,"id":"ba2be162_8bbf7bcf","line":45,"range":{"start_line":45,"start_character":58,"end_line":45,"end_character":73},"in_reply_to":"da36d5c6_6511aa5d","updated":"2017-04-05 08:47:49.000000000","message":"Actually that\u0027s a good point. There isn\u0027t a need to continuously run lscpu. However a user can make a cpu offline without restarting zun - in such a case the inventory data may be out of sync. Thoughts?","commit_id":"4aad6ba3bb7ee3275da2844f3bfec8f1e6624b04"},{"author":{"_account_id":12175,"name":"Eli Qiao","email":"qiaoliyong@gmail.com","username":"Eli"},"change_message_id":"4003adb8df1bfa9f305e81bf2fcc44ef090669ed","unresolved":false,"context_lines":[{"line_number":63,"context_line":""},{"line_number":64,"context_line":"For example:"},{"line_number":65,"context_line":""},{"line_number":66,"context_line":"zun run -i -t --name test --cpu 4 --cpu-policy dedicated"},{"line_number":67,"context_line":""},{"line_number":68,"context_line":"We would try to support scheduling using both of these policies on the same"},{"line_number":69,"context_line":"host."}],"source_content_type":"text/x-rst","patch_set":8,"id":"da36d5c6_25bb3277","line":66,"range":{"start_line":66,"start_character":0,"end_line":66,"end_character":56},"updated":"2017-02-28 07:06:27.000000000","message":"If I understand this correctly, the container will be created and given 4 cpu and each cpu will be pined to a physical cpu on the host.\n\nwhat if host have numa setting eg:\n\nhost have 2 numa cells.\n\nwhat will this result?\n1. 4 cpus pined on cpus of cell0?\n2. 4 cpus pined on cpus of cell1?\n3. 2 cpus pinged on cpus of cell 0, and another 2 on cell1\n\nbla...\n\nwhat if a user want to specified case 3? \n\nplease talk more detail about this. If you would like to support it later, please write it to spec.","commit_id":"4aad6ba3bb7ee3275da2844f3bfec8f1e6624b04"},{"author":{"_account_id":9775,"name":"Sudipta Biswas","email":"sbiswas7@in.ibm.com","username":"sudswas"},"change_message_id":"b7b71931e79d915ee7c40a146a040d2e3e062f0c","unresolved":false,"context_lines":[{"line_number":63,"context_line":""},{"line_number":64,"context_line":"For example:"},{"line_number":65,"context_line":""},{"line_number":66,"context_line":"zun run -i -t --name test --cpu 4 --cpu-policy dedicated"},{"line_number":67,"context_line":""},{"line_number":68,"context_line":"We would try to support scheduling using both of these policies on the same"},{"line_number":69,"context_line":"host."}],"source_content_type":"text/x-rst","patch_set":8,"id":"ba2be162_cb5fb370","line":66,"range":{"start_line":66,"start_character":0,"end_line":66,"end_character":56},"in_reply_to":"da36d5c6_25bb3277","updated":"2017-04-05 08:47:49.000000000","message":"Basically you are looking at a specification from the user where they specify how many NUMA nodes they want to split their CPUs? \n\nI am not sure if case 3 is really useful in case of a container process (as opposed to a VM). So my assumption was - it would always by default be 1 NUMA node or we will say host-not-found. But let\u0027s say - if we have fragmented cpusets - in such a case, binding to cpusets across various NUMA nodes will defeat the purpose right? I look forward to your suggestions here.\n\nI think we can further improve this spec by allowing numa-node-mem-set binding. Basically when we select a given set of cpusets from a socket, we should restrict it\u0027s memory access also to that node itself. Docker has a support for cpuset memsets now. So we can use that.\n\nEither way - allowing the user to specify the number of NUMA nodes can be amended to this feature in the next iteration.","commit_id":"4aad6ba3bb7ee3275da2844f3bfec8f1e6624b04"},{"author":{"_account_id":25026,"name":"Bin Zhou","email":"lakerzhou@gmail.com","username":"lakerzhou"},"change_message_id":"9558b860f58ef484bdce45b8c0b8e3041b30be25","unresolved":false,"context_lines":[{"line_number":138,"context_line":"1. Create the new schema."},{"line_number":139,"context_line":"2. Add cpu_policy field in the REST APIs and zun client."},{"line_number":140,"context_line":"3. Write logic to hydrate the inventory tables."},{"line_number":141,"context_line":"4. Implement a periodic task that inventories the host."},{"line_number":142,"context_line":"5. Write logic to check the cpusets of a given host."},{"line_number":143,"context_line":"5. Implement unit/integration test."},{"line_number":144,"context_line":""}],"source_content_type":"text/x-rst","patch_set":8,"id":"da36d5c6_40f742f5","line":141,"range":{"start_line":141,"start_character":3,"end_line":141,"end_character":55},"updated":"2017-02-27 21:05:55.000000000","message":"Is there a common practice of how to do this? For example, should a configurable interval be defined in zun.conf? What is the default value?","commit_id":"4aad6ba3bb7ee3275da2844f3bfec8f1e6624b04"},{"author":{"_account_id":9775,"name":"Sudipta Biswas","email":"sbiswas7@in.ibm.com","username":"sudswas"},"change_message_id":"b7b71931e79d915ee7c40a146a040d2e3e062f0c","unresolved":false,"context_lines":[{"line_number":138,"context_line":"1. Create the new schema."},{"line_number":139,"context_line":"2. Add cpu_policy field in the REST APIs and zun client."},{"line_number":140,"context_line":"3. Write logic to hydrate the inventory tables."},{"line_number":141,"context_line":"4. Implement a periodic task that inventories the host."},{"line_number":142,"context_line":"5. Write logic to check the cpusets of a given host."},{"line_number":143,"context_line":"5. Implement unit/integration test."},{"line_number":144,"context_line":""}],"source_content_type":"text/x-rst","patch_set":8,"id":"ba2be162_0bd9cbf3","line":141,"range":{"start_line":141,"start_character":3,"end_line":141,"end_character":55},"in_reply_to":"da36d5c6_40f742f5","updated":"2017-04-05 08:47:49.000000000","message":"I have left the implementation details to the reviews. But necessarily in abstraction, it appears that we need some kind of a reconciliation task.\n\nThis has been laid out already as a part of the periodic task in zun.","commit_id":"4aad6ba3bb7ee3275da2844f3bfec8f1e6624b04"},{"author":{"_account_id":12175,"name":"Eli Qiao","email":"qiaoliyong@gmail.com","username":"Eli"},"change_message_id":"4003adb8df1bfa9f305e81bf2fcc44ef090669ed","unresolved":false,"context_lines":[{"line_number":140,"context_line":"3. Write logic to hydrate the inventory tables."},{"line_number":141,"context_line":"4. Implement a periodic task that inventories the host."},{"line_number":142,"context_line":"5. Write logic to check the cpusets of a given host."},{"line_number":143,"context_line":"5. Implement unit/integration test."},{"line_number":144,"context_line":""},{"line_number":145,"context_line":""},{"line_number":146,"context_line":"Dependencies"}],"source_content_type":"text/x-rst","patch_set":8,"id":"da36d5c6_a8129756","line":143,"range":{"start_line":143,"start_character":0,"end_line":143,"end_character":1},"updated":"2017-02-28 07:06:27.000000000","message":"6","commit_id":"4aad6ba3bb7ee3275da2844f3bfec8f1e6624b04"},{"author":{"_account_id":9775,"name":"Sudipta Biswas","email":"sbiswas7@in.ibm.com","username":"sudswas"},"change_message_id":"b7b71931e79d915ee7c40a146a040d2e3e062f0c","unresolved":false,"context_lines":[{"line_number":140,"context_line":"3. Write logic to hydrate the inventory tables."},{"line_number":141,"context_line":"4. Implement a periodic task that inventories the host."},{"line_number":142,"context_line":"5. Write logic to check the cpusets of a given host."},{"line_number":143,"context_line":"5. Implement unit/integration test."},{"line_number":144,"context_line":""},{"line_number":145,"context_line":""},{"line_number":146,"context_line":"Dependencies"}],"source_content_type":"text/x-rst","patch_set":8,"id":"ba2be162_6beec78c","line":143,"range":{"start_line":143,"start_character":0,"end_line":143,"end_character":1},"in_reply_to":"da36d5c6_a8129756","updated":"2017-04-05 08:47:49.000000000","message":"Good catch.","commit_id":"4aad6ba3bb7ee3275da2844f3bfec8f1e6624b04"},{"author":{"_account_id":12175,"name":"Eli Qiao","email":"qiaoliyong@gmail.com","username":"Eli"},"change_message_id":"4003adb8df1bfa9f305e81bf2fcc44ef090669ed","unresolved":false,"context_lines":[{"line_number":141,"context_line":"4. Implement a periodic task that inventories the host."},{"line_number":142,"context_line":"5. Write logic to check the cpusets of a given host."},{"line_number":143,"context_line":"5. Implement unit/integration test."},{"line_number":144,"context_line":""},{"line_number":145,"context_line":""},{"line_number":146,"context_line":"Dependencies"},{"line_number":147,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":8,"id":"da36d5c6_0831e3ec","line":144,"updated":"2017-02-28 07:06:27.000000000","message":"Seems missing, how to passing vcpu-\u003epcpu pin in docker?\ncan you pass some docs of how docker do cpu pinning?\n\nAnd, what about nova-docker driver? does nova docker support vcpu-\u003epcpu pinning?","commit_id":"4aad6ba3bb7ee3275da2844f3bfec8f1e6624b04"},{"author":{"_account_id":9775,"name":"Sudipta Biswas","email":"sbiswas7@in.ibm.com","username":"sudswas"},"change_message_id":"b7b71931e79d915ee7c40a146a040d2e3e062f0c","unresolved":false,"context_lines":[{"line_number":141,"context_line":"4. Implement a periodic task that inventories the host."},{"line_number":142,"context_line":"5. Write logic to check the cpusets of a given host."},{"line_number":143,"context_line":"5. Implement unit/integration test."},{"line_number":144,"context_line":""},{"line_number":145,"context_line":""},{"line_number":146,"context_line":"Dependencies"},{"line_number":147,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":8,"id":"ba2be162_0b00eb33","line":144,"in_reply_to":"da36d5c6_0831e3ec","updated":"2017-04-05 08:47:49.000000000","message":"Sure i will add that info too.","commit_id":"4aad6ba3bb7ee3275da2844f3bfec8f1e6624b04"},{"author":{"_account_id":12175,"name":"Eli Qiao","email":"qiaoliyong@gmail.com","username":"Eli"},"change_message_id":"053ecda520e6d94eb566491265e03a95f3bdd5bf","unresolved":false,"context_lines":[{"line_number":81,"context_line":""},{"line_number":82,"context_line":"docker run -d ubuntu --cpusets-cpu\u003d\"1,3\" --cpuset-mems\u003d\"1,3\""},{"line_number":83,"context_line":""},{"line_number":84,"context_line":"The cpuset-mems would allow the memory access for the cpusets to stay localized.  "},{"line_number":85,"context_line":""},{"line_number":86,"context_line":""},{"line_number":87,"context_line":"Design Principles"}],"source_content_type":"text/x-rst","patch_set":9,"id":"bff0334d_882662b7","line":84,"range":{"start_line":84,"start_character":80,"end_line":84,"end_character":82},"updated":"2017-04-06 01:15:32.000000000","message":"remove this.","commit_id":"f2cd11c6dcc8c377ab82b0819cc22066553454f1"},{"author":{"_account_id":12175,"name":"Eli Qiao","email":"qiaoliyong@gmail.com","username":"Eli"},"change_message_id":"053ecda520e6d94eb566491265e03a95f3bdd5bf","unresolved":false,"context_lines":[{"line_number":82,"context_line":"docker run -d ubuntu --cpusets-cpu\u003d\"1,3\" --cpuset-mems\u003d\"1,3\""},{"line_number":83,"context_line":""},{"line_number":84,"context_line":"The cpuset-mems would allow the memory access for the cpusets to stay localized.  "},{"line_number":85,"context_line":""},{"line_number":86,"context_line":""},{"line_number":87,"context_line":"Design Principles"},{"line_number":88,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":9,"id":"bff0334d_284ab654","line":85,"updated":"2017-04-06 01:15:32.000000000","message":"Another question, what is a container process is paused/stoped, does the cpu pin still there? I mean if other container can reuse these cpus?\n\nif it can be reused, we need to consider pause/stop case for a created container on the host.","commit_id":"f2cd11c6dcc8c377ab82b0819cc22066553454f1"},{"author":{"_account_id":25026,"name":"Bin Zhou","email":"lakerzhou@gmail.com","username":"lakerzhou"},"change_message_id":"5ea765684f63672c0ea5b7ce4b472fcd5fa614e3","unresolved":false,"context_lines":[{"line_number":82,"context_line":"docker run -d ubuntu --cpusets-cpu\u003d\"1,3\" --cpuset-mems\u003d\"1,3\""},{"line_number":83,"context_line":""},{"line_number":84,"context_line":"The cpuset-mems would allow the memory access for the cpusets to stay localized.  "},{"line_number":85,"context_line":""},{"line_number":86,"context_line":""},{"line_number":87,"context_line":"Design Principles"},{"line_number":88,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":9,"id":"df140735_2704eb49","line":85,"in_reply_to":"bff0334d_163fcf25","updated":"2017-05-31 18:43:21.000000000","message":"+1","commit_id":"f2cd11c6dcc8c377ab82b0819cc22066553454f1"},{"author":{"_account_id":9775,"name":"Sudipta Biswas","email":"sbiswas7@in.ibm.com","username":"sudswas"},"change_message_id":"dd826b942a6e582cbffe36241a87e54194c8524c","unresolved":false,"context_lines":[{"line_number":82,"context_line":"docker run -d ubuntu --cpusets-cpu\u003d\"1,3\" --cpuset-mems\u003d\"1,3\""},{"line_number":83,"context_line":""},{"line_number":84,"context_line":"The cpuset-mems would allow the memory access for the cpusets to stay localized.  "},{"line_number":85,"context_line":""},{"line_number":86,"context_line":""},{"line_number":87,"context_line":"Design Principles"},{"line_number":88,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":9,"id":"bff0334d_163fcf25","line":85,"in_reply_to":"bff0334d_284ab654","updated":"2017-04-07 10:48:34.000000000","message":"Very good point. I think we should not update the pinset unless we have killed a container. Since, we really can\u0027t afford to migrate the container unlike VMs.","commit_id":"f2cd11c6dcc8c377ab82b0819cc22066553454f1"}]}