)]}'
{"specs/mitaka/approved/resource-providers-scheduler.rst":[{"author":{"_account_id":6873,"name":"Matt Riedemann","email":"mriedem.os@gmail.com","username":"mriedem"},"change_message_id":"ecdc7b30822f8414e25c9c763cb75abe596b9bec","unresolved":false,"context_lines":[{"line_number":8,"context_line":"Resource Providers - Scheduler Integration"},{"line_number":9,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":10,"context_line":""},{"line_number":11,"context_line":"https://blueprints.launchpad.net/nova/+spec/resource-providers-scheduler"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"This blueprint completes the `resource-providers` series of blueprints by"},{"line_number":14,"context_line":"having the scheduler make use of the `inventories` and `allocations` table to"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3a57f1b5_183c3ce1","line":11,"updated":"2016-02-11 15:57:40.000000000","message":"This doesn\u0027t exist yet. Would be nice to create the stubs since I\u0027m trying to link the dependency tree in launchpad based on this:\n\n(9:38:20 AM) jaypipes: mriedem: in order:\n(9:38:21 AM) jaypipes: * resource-classes (bp MERGED, code MERGED)\n(9:38:21 AM) jaypipes: * pci-generate-stats (bp MERGED, code IN REVIEW)\n(9:38:21 AM) jaypipes: * resource-providers (bp MERGED, code IN REVIEW)\n(9:38:21 AM) jaypipes: * generic-resource-pools (bp IN REVIEW, code TODO)\n(9:38:21 AM) jaypipes: * compute-node-inventory (bp IN REVIEW, code TODO)\n(9:38:23 AM) jaypipes: * resource-providers-allocations (bp IN REVIEW, code TODO)\n(9:38:25 AM) jaypipes: * resource-providers-scheduler (bp IN REVIEW, code TODO)","commit_id":"17f8c7797c16e4752a7c86fa6b288c5b9d311adb"},{"author":{"_account_id":15648,"name":"Yingxin","email":"yingxin.cheng@intel.com","username":"Yingxin"},"change_message_id":"3c941fcdd77186f48a91bfe82509c45a30e6aad8","unresolved":false,"context_lines":[{"line_number":14,"context_line":"having the scheduler make use of the `inventories` and `allocations` table to"},{"line_number":15,"context_line":"pre-filter compute nodes from evaluation during `select_destinations()`. In"},{"line_number":16,"context_line":"addition, this blueprint makes a dramatic change in the design of the scheduler"},{"line_number":17,"context_line":"and resource tracker in Nova by moving the process of resource claiming into"},{"line_number":18,"context_line":"the scheduler instead of on the distributed compute nodes."},{"line_number":19,"context_line":""},{"line_number":20,"context_line":"Problem description"},{"line_number":21,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":1,"id":"dae33548_f7dd8471","line":18,"range":{"start_line":17,"start_character":32,"end_line":18,"end_character":57},"updated":"2016-02-15 16:28:40.000000000","message":"Does it have the same meaning after changing some wording? \"moving the process of distributed and in-memory resource claiming into the global database\".\n\nIf it is, I believe the distributed in-memory lock is faster than the database lock.","commit_id":"17f8c7797c16e4752a7c86fa6b288c5b9d311adb"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"3d9add8c8bb7d71dba399e95677edd024d79a148","unresolved":false,"context_lines":[{"line_number":31,"context_line":"Many of these scheduler filters do nothing more than calculate the amount of a"},{"line_number":32,"context_line":"particular resource that a compute node has available to it and return `False`"},{"line_number":33,"context_line":"if the amount requested is greater than the available amount of that type of"},{"line_number":34,"context_line":"resource."},{"line_number":35,"context_line":""},{"line_number":36,"context_line":"After the scheduler has sorted the list of compute nodes that meet the"},{"line_number":37,"context_line":"requested constraints, the scheduler returns the ID of the compute node to the"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3a57f1b5_78566d5d","line":34,"updated":"2016-02-12 08:55:07.000000000","message":"I totally agree, but like I said during the midcycle, the related filters can be not being called just by changing an opt.\nIt can also be easily modified by an other out-of-tree filter.","commit_id":"17f8c7797c16e4752a7c86fa6b288c5b9d311adb"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"ca9cd05ad547ab6559a1f54a6c486d295a659f9c","unresolved":false,"context_lines":[{"line_number":39,"context_line":"target destination compute node over RPC which (usually) ends in a call to"},{"line_number":40,"context_line":"`nova.compute.resource_tracker.ResourceTracker.instance_claim()`. At the end of"},{"line_number":41,"context_line":"this `instance_claim()` call, the resource tracker updates the `ComputeNode`"},{"line_number":42,"context_line":"object and updates the compute node\u0027s \"inventory\" and \"allocation\" fields, then"},{"line_number":43,"context_line":"calls `save()` on the object. It is *only* at this time that the database has"},{"line_number":44,"context_line":"an accurate record of the resources claimed on that compute node. The scheduler"},{"line_number":45,"context_line":"updates the host state object\u0027s resource counters before sending the result of"}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a68dd71_5c253908","line":42,"updated":"2016-01-26 09:44:18.000000000","message":"@vilobh: Please see the following blueprints for information on which fields are inventory and which fields are allocation:\n\n compute-node-inventory: https://review.openstack.org/260048\n resource-providers-allocations: https://review.openstack.org/271779","commit_id":"17f8c7797c16e4752a7c86fa6b288c5b9d311adb"},{"author":{"_account_id":10263,"name":"Vilobh Meshram","email":"vilobhmm@yahoo-inc.com","username":"vilobhmm"},"change_message_id":"cde0d96e9c101a7c0c1df6130aecc8fc5862031e","unresolved":false,"context_lines":[{"line_number":39,"context_line":"target destination compute node over RPC which (usually) ends in a call to"},{"line_number":40,"context_line":"`nova.compute.resource_tracker.ResourceTracker.instance_claim()`. At the end of"},{"line_number":41,"context_line":"this `instance_claim()` call, the resource tracker updates the `ComputeNode`"},{"line_number":42,"context_line":"object and updates the compute node\u0027s \"inventory\" and \"allocation\" fields, then"},{"line_number":43,"context_line":"calls `save()` on the object. It is *only* at this time that the database has"},{"line_number":44,"context_line":"an accurate record of the resources claimed on that compute node. The scheduler"},{"line_number":45,"context_line":"updates the host state object\u0027s resource counters before sending the result of"}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a68dd71_df02c3b3","line":42,"range":{"start_line":42,"start_character":39,"end_line":42,"end_character":48},"updated":"2016-01-24 21:30:17.000000000","message":"It would be nice to descrive which fields as part of compute nodes object [1] you would classify as \"inventory\" fields and which as \"allocation\" fields\n\n[1] https://github.com/openstack/nova/blob/master/nova/objects/compute_node.py","commit_id":"17f8c7797c16e4752a7c86fa6b288c5b9d311adb"},{"author":{"_account_id":15648,"name":"Yingxin","email":"yingxin.cheng@intel.com","username":"Yingxin"},"change_message_id":"3c941fcdd77186f48a91bfe82509c45a30e6aad8","unresolved":false,"context_lines":[{"line_number":41,"context_line":"this `instance_claim()` call, the resource tracker updates the `ComputeNode`"},{"line_number":42,"context_line":"object and updates the compute node\u0027s \"inventory\" and \"allocation\" fields, then"},{"line_number":43,"context_line":"calls `save()` on the object. It is *only* at this time that the database has"},{"line_number":44,"context_line":"an accurate record of the resources claimed on that compute node. The scheduler"},{"line_number":45,"context_line":"updates the host state object\u0027s resource counters before sending the result of"},{"line_number":46,"context_line":"`select_destinations()` back to the caller and has a check to ensure that the"},{"line_number":47,"context_line":"complicated dance of messages and database interactions between the scheduler,"}],"source_content_type":"text/x-rst","patch_set":1,"id":"dae33548_17e3c8a4","line":44,"range":{"start_line":44,"start_character":3,"end_line":44,"end_character":18},"updated":"2016-02-15 16:28:40.000000000","message":"From my different viewpoint: before *that* time the db also has an accurate record of resources, because the scheduler decision haven\u0027t reached the resource tracker yet. The resource tracker has the final say about whether a scheduler decision is successful or not, then it will update its resources. Only scheduler doesn\u0027t know about that.\n\nThe problem is that scheduler doesn\u0027t remember its recent decisions, so when the scheduler refreshes its resource view from db, all the recent decisions are lost. The ideal scheduler should still apply its recent decisions to the cache until it knows whether the decision is successful on the resource tracker side.","commit_id":"17f8c7797c16e4752a7c86fa6b288c5b9d311adb"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"ca9cd05ad547ab6559a1f54a6c486d295a659f9c","unresolved":false,"context_lines":[{"line_number":43,"context_line":"calls `save()` on the object. It is *only* at this time that the database has"},{"line_number":44,"context_line":"an accurate record of the resources claimed on that compute node. The scheduler"},{"line_number":45,"context_line":"updates the host state object\u0027s resource counters before sending the result of"},{"line_number":46,"context_line":"`select_destinations()` back to the caller and has a check to ensure that the"},{"line_number":47,"context_line":"complicated dance of messages and database interactions between the scheduler,"},{"line_number":48,"context_line":"conductor, and target compute node doesn\u0027t give it a stale view of the"},{"line_number":49,"context_line":"deployment."}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a68dd71_7c227d03","line":46,"updated":"2016-01-26 09:44:18.000000000","message":"@vilobh: yes:\n\n https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L259\n\nOn respin, I\u0027ll add that link.","commit_id":"17f8c7797c16e4752a7c86fa6b288c5b9d311adb"},{"author":{"_account_id":10263,"name":"Vilobh Meshram","email":"vilobhmm@yahoo-inc.com","username":"vilobhmm"},"change_message_id":"cde0d96e9c101a7c0c1df6130aecc8fc5862031e","unresolved":false,"context_lines":[{"line_number":43,"context_line":"calls `save()` on the object. It is *only* at this time that the database has"},{"line_number":44,"context_line":"an accurate record of the resources claimed on that compute node. The scheduler"},{"line_number":45,"context_line":"updates the host state object\u0027s resource counters before sending the result of"},{"line_number":46,"context_line":"`select_destinations()` back to the caller and has a check to ensure that the"},{"line_number":47,"context_line":"complicated dance of messages and database interactions between the scheduler,"},{"line_number":48,"context_line":"conductor, and target compute node doesn\u0027t give it a stale view of the"},{"line_number":49,"context_line":"deployment."}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a68dd71_df196323","line":46,"range":{"start_line":46,"start_character":53,"end_line":46,"end_character":58},"updated":"2016-01-24 21:30:17.000000000","message":"Providing a reference for this check in the scheduler code would be helpful IMHO. Are you talking about consume-from-request ?","commit_id":"17f8c7797c16e4752a7c86fa6b288c5b9d311adb"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"3d9add8c8bb7d71dba399e95677edd024d79a148","unresolved":false,"context_lines":[{"line_number":53,"context_line":"First, having to return all compute node records in the entire deployment is"},{"line_number":54,"context_line":"extremely wasteful and this inefficiency gets worse the larger the deployment"},{"line_number":55,"context_line":"is. The filter loop is essentially implementing a SQL WHERE clause, but in"},{"line_number":56,"context_line":"Python instead of a much faster and more efficient database query."},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"Second, because of the way that the scheduler keeps a cache of host state"},{"line_number":59,"context_line":"objects to avoid operating on stale data during the time between the scheduler"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3a57f1b5_38e315a6","line":56,"updated":"2016-02-12 08:55:07.000000000","message":"so, from a performance perspective, I think John told us that the main issue is getting all the computes, but having RAM, CPU and disk calls in the filters is not really a performance problem.\n\nThat said, I agree, getting less computes to check looks better from a DB PoV.","commit_id":"17f8c7797c16e4752a7c86fa6b288c5b9d311adb"},{"author":{"_account_id":15648,"name":"Yingxin","email":"yingxin.cheng@intel.com","username":"Yingxin"},"change_message_id":"3c941fcdd77186f48a91bfe82509c45a30e6aad8","unresolved":false,"context_lines":[{"line_number":53,"context_line":"First, having to return all compute node records in the entire deployment is"},{"line_number":54,"context_line":"extremely wasteful and this inefficiency gets worse the larger the deployment"},{"line_number":55,"context_line":"is. The filter loop is essentially implementing a SQL WHERE clause, but in"},{"line_number":56,"context_line":"Python instead of a much faster and more efficient database query."},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"Second, because of the way that the scheduler keeps a cache of host state"},{"line_number":59,"context_line":"objects to avoid operating on stale data during the time between the scheduler"}],"source_content_type":"text/x-rst","patch_set":1,"id":"dae33548_3221bafc","line":56,"in_reply_to":"3a57f1b5_38e315a6","updated":"2016-02-15 16:28:40.000000000","message":"Agreed, there still exists the problem of refreshing compute node records in the mentioned scheduler, although to some degree the number of records can be reduced by the SQL WHERE clause. I\u0027m also interested in 3 questions: \nHow many compute nodes can be filtered in this approach?\nCan the final nodes be selected using only SQL queries and by some relatively static properties such as host aggregates, system metadatas etc?\nWhat\u0027s the actual speed of SQL queries compared with in-memory calculation of caches?\n\nIn my performance tests, the \"refreshing\" operation from db to conductor and to scheduler through oslo_messaging costs about 120 times longer[1] than in-memory calculations using filters and weighers. In the multiple scheduler deployment, the performance is even worse[2].\n\n[1] http://paste.openstack.org/show/486923/\n[2] http://paste.openstack.org/show/486925/","commit_id":"17f8c7797c16e4752a7c86fa6b288c5b9d311adb"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"3d9add8c8bb7d71dba399e95677edd024d79a148","unresolved":false,"context_lines":[{"line_number":55,"context_line":"is. The filter loop is essentially implementing a SQL WHERE clause, but in"},{"line_number":56,"context_line":"Python instead of a much faster and more efficient database query."},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"Second, because of the way that the scheduler keeps a cache of host state"},{"line_number":59,"context_line":"objects to avoid operating on stale data during the time between the scheduler"},{"line_number":60,"context_line":"selected a destination host and that target compute node updated the database"},{"line_number":61,"context_line":"with the claimed resource information, two separate scheduler processes have a"},{"line_number":62,"context_line":"significant chance of selecting the same target host (because only the host"},{"line_number":63,"context_line":"state cache on the one of them will be updated appropriately) and causing a"},{"line_number":64,"context_line":"Retry to occur relatively frequently. The \"solution\" to this problem has been"},{"line_number":65,"context_line":"to only run a single scheduler daemon, which introduces performance issues and,"},{"line_number":66,"context_line":"in part, led to the creation of the original cells concept."},{"line_number":67,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"3a57f1b5_fbbb37e3","line":64,"range":{"start_line":58,"start_character":0,"end_line":64,"end_character":37},"updated":"2016-02-12 08:55:07.000000000","message":"So, see, I\u0027m not really seeing that as a problem. Having the consistency managed by computes nodes seems to me the right approach, because the scheduler can\u0027t be omniscient.\nTwo reasons for that :\n\n- #1 If we consider that an external system, as a resource provider, will provide a single resource class usage (like network segment availability), it will still require the instance to be spawned *for* consuming that resource class, even if the scheduler accounts for it. That would mean that the scheduler would have to manage a list of allocations with TTL, and periodically verify that the allocation succeeded by asking the external system (or getting feedback from the external system). See, that\u0027s racy.\n- #2 the scheduler is just a decision maker, by any case it doesn\u0027t account for the real instance creation (it doesn\u0027t hold the ownership of the instance). Having it being accountable for the instances usage is heavily difficult. Take for example a request for CPU pinning or NUMA affinity. The user can\u0027t really express which pin of the pCPU he will get, that\u0027s the compute node which will do that for him. Of course, the scheduler will help picking an host that can fit the request, but the real pinning will happen in the compute node.\n\nRetries are not bad, they\u0027re necessary (I know this is a 2PC but are we considering all 2PC bad?) . What we can hugely improve is rather the \"distributed scheduler consistency\" and make sure that two instances of the same scheduler will handle the same set of inventories and allocations, that I truly +2 it.","commit_id":"17f8c7797c16e4752a7c86fa6b288c5b9d311adb"},{"author":{"_account_id":15648,"name":"Yingxin","email":"yingxin.cheng@intel.com","username":"Yingxin"},"change_message_id":"3c941fcdd77186f48a91bfe82509c45a30e6aad8","unresolved":false,"context_lines":[{"line_number":55,"context_line":"is. The filter loop is essentially implementing a SQL WHERE clause, but in"},{"line_number":56,"context_line":"Python instead of a much faster and more efficient database query."},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"Second, because of the way that the scheduler keeps a cache of host state"},{"line_number":59,"context_line":"objects to avoid operating on stale data during the time between the scheduler"},{"line_number":60,"context_line":"selected a destination host and that target compute node updated the database"},{"line_number":61,"context_line":"with the claimed resource information, two separate scheduler processes have a"},{"line_number":62,"context_line":"significant chance of selecting the same target host (because only the host"},{"line_number":63,"context_line":"state cache on the one of them will be updated appropriately) and causing a"},{"line_number":64,"context_line":"Retry to occur relatively frequently. The \"solution\" to this problem has been"},{"line_number":65,"context_line":"to only run a single scheduler daemon, which introduces performance issues and,"},{"line_number":66,"context_line":"in part, led to the creation of the original cells concept."},{"line_number":67,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"dae33548_70b5f5cb","line":64,"range":{"start_line":58,"start_character":0,"end_line":64,"end_character":37},"in_reply_to":"3a57f1b5_fbbb37e3","updated":"2016-02-15 16:28:40.000000000","message":"If I\u0027m correct, Jay\u0027s idea is to eliminate the time between the scheduler selected a destination host and the following db update, so the racing problem is over. However it introduces database locks and the performance relies mostly on the implementation of database and conductor.\n\nAnother way is to let schedulers try their decisions on separate remote targets(based on claim) and minimize the time gap. No locks are introduced however there are retries.\n\nIt is very hard for me to determine which approach is better.","commit_id":"17f8c7797c16e4752a7c86fa6b288c5b9d311adb"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"3d9add8c8bb7d71dba399e95677edd024d79a148","unresolved":false,"context_lines":[{"line_number":62,"context_line":"significant chance of selecting the same target host (because only the host"},{"line_number":63,"context_line":"state cache on the one of them will be updated appropriately) and causing a"},{"line_number":64,"context_line":"Retry to occur relatively frequently. The \"solution\" to this problem has been"},{"line_number":65,"context_line":"to only run a single scheduler daemon, which introduces performance issues and,"},{"line_number":66,"context_line":"in part, led to the creation of the original cells concept."},{"line_number":67,"context_line":""},{"line_number":68,"context_line":"Use Cases"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3a57f1b5_b8414565","line":65,"range":{"start_line":65,"start_character":39,"end_line":65,"end_character":74},"updated":"2016-02-12 08:55:07.000000000","message":"only for specific cases :\n\n- when the cloud capacity is close to be full\n- when the cloud only has a very few hosts\n\nFor very large clouds, having this design is by far more performant than having a distributed scheduler which would synchronize on all of its instances.","commit_id":"17f8c7797c16e4752a7c86fa6b288c5b9d311adb"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"1aa32088d4e9f62059bdf3ce55f122fab74a7d57","unresolved":false,"context_lines":[{"line_number":63,"context_line":"state cache on the one of them will be updated appropriately) and causing a"},{"line_number":64,"context_line":"Retry to occur relatively frequently. The \"solution\" to this problem has been"},{"line_number":65,"context_line":"to only run a single scheduler daemon, which introduces performance issues and,"},{"line_number":66,"context_line":"in part, led to the creation of the original cells concept."},{"line_number":67,"context_line":""},{"line_number":68,"context_line":"Use Cases"},{"line_number":69,"context_line":"----------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"7a5de9d1_f0d0b4f9","line":66,"range":{"start_line":66,"start_character":9,"end_line":66,"end_character":58},"updated":"2016-02-02 18:43:12.000000000","message":"well its created due to network switch limitations really, but I suppose this is partly true.","commit_id":"17f8c7797c16e4752a7c86fa6b288c5b9d311adb"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"3d9add8c8bb7d71dba399e95677edd024d79a148","unresolved":false,"context_lines":[{"line_number":69,"context_line":"----------"},{"line_number":70,"context_line":""},{"line_number":71,"context_line":"As a deployer, I want to be able to run multiple scheduler processes for"},{"line_number":72,"context_line":"scaling purposes and also reduce (or eliminate) the cause of Retry failures"},{"line_number":73,"context_line":"which cause a not-insignificant amount of work for the system."},{"line_number":74,"context_line":""},{"line_number":75,"context_line":"Proposed change"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3a57f1b5_9b7b2328","line":72,"range":{"start_line":72,"start_character":25,"end_line":72,"end_character":75},"updated":"2016-02-12 08:55:07.000000000","message":"see my comment above L64. I want to be able to run multiple schedulers, yes, but I don\u0027t want to eliminate retries.","commit_id":"17f8c7797c16e4752a7c86fa6b288c5b9d311adb"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"1aa32088d4e9f62059bdf3ce55f122fab74a7d57","unresolved":false,"context_lines":[{"line_number":69,"context_line":"----------"},{"line_number":70,"context_line":""},{"line_number":71,"context_line":"As a deployer, I want to be able to run multiple scheduler processes for"},{"line_number":72,"context_line":"scaling purposes and also reduce (or eliminate) the cause of Retry failures"},{"line_number":73,"context_line":"which cause a not-insignificant amount of work for the system."},{"line_number":74,"context_line":""},{"line_number":75,"context_line":"Proposed change"}],"source_content_type":"text/x-rst","patch_set":1,"id":"7a5de9d1_30340c49","line":72,"range":{"start_line":72,"start_character":33,"end_line":72,"end_character":47},"updated":"2016-02-02 18:43:12.000000000","message":"we don\u0027t fix the NUMA related races here.","commit_id":"17f8c7797c16e4752a7c86fa6b288c5b9d311adb"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"3d9add8c8bb7d71dba399e95677edd024d79a148","unresolved":false,"context_lines":[{"line_number":69,"context_line":"----------"},{"line_number":70,"context_line":""},{"line_number":71,"context_line":"As a deployer, I want to be able to run multiple scheduler processes for"},{"line_number":72,"context_line":"scaling purposes and also reduce (or eliminate) the cause of Retry failures"},{"line_number":73,"context_line":"which cause a not-insignificant amount of work for the system."},{"line_number":74,"context_line":""},{"line_number":75,"context_line":"Proposed change"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3a57f1b5_5b712b43","line":72,"range":{"start_line":72,"start_character":33,"end_line":72,"end_character":47},"in_reply_to":"7a5de9d1_30340c49","updated":"2016-02-12 08:55:07.000000000","message":"Yeah, see my comment L64","commit_id":"17f8c7797c16e4752a7c86fa6b288c5b9d311adb"},{"author":{"_account_id":15648,"name":"Yingxin","email":"yingxin.cheng@intel.com","username":"Yingxin"},"change_message_id":"3c941fcdd77186f48a91bfe82509c45a30e6aad8","unresolved":false,"context_lines":[{"line_number":77,"context_line":""},{"line_number":78,"context_line":"We propose modifying the scheduler in two fundamental ways."},{"line_number":79,"context_line":""},{"line_number":80,"context_line":"We propose to winnow the set of compute nodes it evaluates for matching to a"},{"line_number":81,"context_line":"launch request. This will dramatically reduce the amount of compute node"},{"line_number":82,"context_line":"records that need to be pulled from the database on every call to"},{"line_number":83,"context_line":"`select_destinations()`."}],"source_content_type":"text/x-rst","patch_set":1,"id":"dae33548_6b7344b2","line":80,"range":{"start_line":80,"start_character":14,"end_line":80,"end_character":20},"updated":"2016-02-15 16:28:40.000000000","message":"Can the final nodes be selected using only SQL queries and based on some relatively static properties such as host aggregates, system metadata, image metadata etc?\n\nIf it is possible, I agree that the host state cache can be eliminated, even there is no need to have multiple schedulers because the bottleneck will be in executing the SQLs.","commit_id":"17f8c7797c16e4752a7c86fa6b288c5b9d311adb"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"3d9add8c8bb7d71dba399e95677edd024d79a148","unresolved":false,"context_lines":[{"line_number":80,"context_line":"We propose to winnow the set of compute nodes it evaluates for matching to a"},{"line_number":81,"context_line":"launch request. This will dramatically reduce the amount of compute node"},{"line_number":82,"context_line":"records that need to be pulled from the database on every call to"},{"line_number":83,"context_line":"`select_destinations()`."},{"line_number":84,"context_line":""},{"line_number":85,"context_line":"We also propose to rid ourselves of the host state cache entirely and instead"},{"line_number":86,"context_line":"have the scheduler actually do the claiming of resources for a compute node"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3a57f1b5_bb57ff58","line":83,"updated":"2016-02-12 08:55:07.000000000","message":"I\u0027m okay provided you make that SQL where clause optional (like filters).","commit_id":"17f8c7797c16e4752a7c86fa6b288c5b9d311adb"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"3d9add8c8bb7d71dba399e95677edd024d79a148","unresolved":false,"context_lines":[{"line_number":83,"context_line":"`select_destinations()`."},{"line_number":84,"context_line":""},{"line_number":85,"context_line":"We also propose to rid ourselves of the host state cache entirely and instead"},{"line_number":86,"context_line":"have the scheduler actually do the claiming of resources for a compute node"},{"line_number":87,"context_line":"object. When the scheduler determines the target compute node, the scheduler"},{"line_number":88,"context_line":"will insert a set of records into the `allocations` table representing the"},{"line_number":89,"context_line":"amounts of various resources that the instance will consume on the target host."},{"line_number":90,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"3a57f1b5_1bf5f355","line":87,"range":{"start_line":86,"start_character":35,"end_line":87,"end_character":8},"updated":"2016-02-12 08:55:07.000000000","message":"So, I explained in L47 why I think we still need compute claims, I just feel we need to focus on making sure that multiple scheduler processes are accessing the same datastore consistently (if you want to use scheduler claims, why not), but I don\u0027t want to remove compute claims.","commit_id":"17f8c7797c16e4752a7c86fa6b288c5b9d311adb"},{"author":{"_account_id":15648,"name":"Yingxin","email":"yingxin.cheng@intel.com","username":"Yingxin"},"change_message_id":"3c941fcdd77186f48a91bfe82509c45a30e6aad8","unresolved":false,"context_lines":[{"line_number":83,"context_line":"`select_destinations()`."},{"line_number":84,"context_line":""},{"line_number":85,"context_line":"We also propose to rid ourselves of the host state cache entirely and instead"},{"line_number":86,"context_line":"have the scheduler actually do the claiming of resources for a compute node"},{"line_number":87,"context_line":"object. When the scheduler determines the target compute node, the scheduler"},{"line_number":88,"context_line":"will insert a set of records into the `allocations` table representing the"},{"line_number":89,"context_line":"amounts of various resources that the instance will consume on the target host."},{"line_number":90,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"dae33548_2b9adc14","line":87,"range":{"start_line":86,"start_character":35,"end_line":87,"end_character":8},"in_reply_to":"3a57f1b5_1bf5f355","updated":"2016-02-15 16:28:40.000000000","message":"Again, one way is to eliminate the time between the scheduler selected a destination host and the following db update(i.e. the scheduler db claim) like this, but the db locks are introduced; Another way is to let schedulers try their decisions on separate remote compute nodes, no locks are introduced however there are retries.\n\nIt is still hard for me to determine which approach is better.","commit_id":"17f8c7797c16e4752a7c86fa6b288c5b9d311adb"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"ca9cd05ad547ab6559a1f54a6c486d295a659f9c","unresolved":false,"context_lines":[{"line_number":86,"context_line":"have the scheduler actually do the claiming of resources for a compute node"},{"line_number":87,"context_line":"object. When the scheduler determines the target compute node, the scheduler"},{"line_number":88,"context_line":"will insert a set of records into the `allocations` table representing the"},{"line_number":89,"context_line":"amounts of various resources that the instance will consume on the target host."},{"line_number":90,"context_line":""},{"line_number":91,"context_line":"Alternatives"},{"line_number":92,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a68dd71_1c1fc1b8","line":89,"updated":"2016-01-26 09:44:18.000000000","message":"@vilobh: See above. The entire insert of allocations will be a single atomic database transaction.","commit_id":"17f8c7797c16e4752a7c86fa6b288c5b9d311adb"},{"author":{"_account_id":10263,"name":"Vilobh Meshram","email":"vilobhmm@yahoo-inc.com","username":"vilobhmm"},"change_message_id":"cde0d96e9c101a7c0c1df6130aecc8fc5862031e","unresolved":false,"context_lines":[{"line_number":86,"context_line":"have the scheduler actually do the claiming of resources for a compute node"},{"line_number":87,"context_line":"object. When the scheduler determines the target compute node, the scheduler"},{"line_number":88,"context_line":"will insert a set of records into the `allocations` table representing the"},{"line_number":89,"context_line":"amounts of various resources that the instance will consume on the target host."},{"line_number":90,"context_line":""},{"line_number":91,"context_line":"Alternatives"},{"line_number":92,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a68dd71_1fd5bbe6","line":89,"range":{"start_line":89,"start_character":63,"end_line":89,"end_character":66},"updated":"2016-01-24 21:30:17.000000000","message":"what happens when this fails ?Who takes responsibility to clear off the allocations ?","commit_id":"17f8c7797c16e4752a7c86fa6b288c5b9d311adb"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"1aa32088d4e9f62059bdf3ce55f122fab74a7d57","unresolved":false,"context_lines":[{"line_number":86,"context_line":"have the scheduler actually do the claiming of resources for a compute node"},{"line_number":87,"context_line":"object. When the scheduler determines the target compute node, the scheduler"},{"line_number":88,"context_line":"will insert a set of records into the `allocations` table representing the"},{"line_number":89,"context_line":"amounts of various resources that the instance will consume on the target host."},{"line_number":90,"context_line":""},{"line_number":91,"context_line":"Alternatives"},{"line_number":92,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"7a5de9d1_10231007","line":89,"in_reply_to":"9a68dd71_1c1fc1b8","updated":"2016-02-02 18:43:12.000000000","message":"We still need retries for the NUMA cases, but thats fine, you just drop the claim and try get a new one.","commit_id":"17f8c7797c16e4752a7c86fa6b288c5b9d311adb"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"1aa32088d4e9f62059bdf3ce55f122fab74a7d57","unresolved":false,"context_lines":[{"line_number":100,"context_line":"need to essentially add an entirely new code path and scheduler interface and"},{"line_number":101,"context_line":"code path on the compute node that didn\u0027t do the \"claim\" logic and attempt to"},{"line_number":102,"context_line":"update the inventory of the compute node (since the scheduler will have already"},{"line_number":103,"context_line":"done so)."},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"Data model impact"},{"line_number":106,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"7a5de9d1_d0e1f89b","line":103,"updated":"2016-02-02 18:43:12.000000000","message":"I was wondering about that, making this optional at first, while we deprecate the old filterscheduler and cachingscheduler.\n\nThe claim logic would just be part of select_destinations here, I think? Or are you saying the resource tracker is also involved? I guess it shouldn\u0027t be?","commit_id":"17f8c7797c16e4752a7c86fa6b288c5b9d311adb"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"3d9add8c8bb7d71dba399e95677edd024d79a148","unresolved":false,"context_lines":[{"line_number":158,"context_line":"Work Items"},{"line_number":159,"context_line":"----------"},{"line_number":160,"context_line":""},{"line_number":161,"context_line":"TODO"},{"line_number":162,"context_line":""},{"line_number":163,"context_line":"Dependencies"},{"line_number":164,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":1,"id":"3a57f1b5_bb9cdf0e","line":161,"updated":"2016-02-12 08:55:07.000000000","message":"That would help *a lot* the conversation to get some high-level work items you want to do.","commit_id":"17f8c7797c16e4752a7c86fa6b288c5b9d311adb"}]}
