)]}'
{"specs/backlog/approved/parallel-scheduler.rst":[{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"abe5f66ac36df66fbc62f9fa413910d2652783ab","unresolved":false,"context_lines":[{"line_number":16,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":17,"context_line":""},{"line_number":18,"context_line":"We need the nova filter scheduler to work well in typical public cloud"},{"line_number":19,"context_line":"even after they have migrated from cells v1 to cells v2."},{"line_number":20,"context_line":""},{"line_number":21,"context_line":"Some key observations about the current nova-scheduler:"},{"line_number":22,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_26198988","line":19,"updated":"2015-06-24 21:39:02.000000000","message":"it leaves non explained why we care about cells v2. Here the main reason is that we don\u0027t trust the scheduler to be able to scale out.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"abe5f66ac36df66fbc62f9fa413910d2652783ab","unresolved":false,"context_lines":[{"line_number":22,"context_line":""},{"line_number":23,"context_line":"* If you running two nova-scheduler processes they race each other, they"},{"line_number":24,"context_line":"  don\u0027t find out about each others choices until the DB gets updated"},{"line_number":25,"context_line":"  by the nova-compute resource tracker."},{"line_number":26,"context_line":"  This has lead to many deployments opting for an Active/Passive HA setup"},{"line_number":27,"context_line":"  for the nova-scheduler process."},{"line_number":28,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_f9f4a6f1","line":25,"updated":"2015-06-24 21:39:02.000000000","message":"That\u0027s even worst than that because the HostState is not shared, so it is only refreshed at the beginning of the scheduler call. That way, two distinct schedulers can diverge from the same point of time if they have 2 or more instances to schedule.\n\nHaving RT refreshing the resources immediatly would just makes sure that a new _schedule() call would get the fresh data, but wouldn\u0027t prevent two concurrent schedulers to race.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":11642,"name":"Jesse J. Cook","email":"jesse.cook@rackspace.com","username":"crashenx"},"change_message_id":"e3a6a73be652c6d208bad7e6a7bff20be0ebb0da","unresolved":false,"context_lines":[{"line_number":29,"context_line":"* The resource tracker has the final say on if an instance can fit."},{"line_number":30,"context_line":"  If the request ends up on a full compute node, the build errors out,"},{"line_number":31,"context_line":"  and lets the retry system find an different compute node to try."},{"line_number":32,"context_line":"  However, we stop retrying after three attempts, and this extends build"},{"line_number":33,"context_line":"  times for users, so its best to avoid these retries."},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"* Deployers often chose to fill first, to ensure they keep room for"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_ac2cb602","line":32,"updated":"2015-06-24 01:20:01.000000000","message":"As worded, I\u0027m not sure if we should retry more or not at all, or both?","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"abe5f66ac36df66fbc62f9fa413910d2652783ab","unresolved":false,"context_lines":[{"line_number":29,"context_line":"* The resource tracker has the final say on if an instance can fit."},{"line_number":30,"context_line":"  If the request ends up on a full compute node, the build errors out,"},{"line_number":31,"context_line":"  and lets the retry system find an different compute node to try."},{"line_number":32,"context_line":"  However, we stop retrying after three attempts, and this extends build"},{"line_number":33,"context_line":"  times for users, so its best to avoid these retries."},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"* Deployers often chose to fill first, to ensure they keep room for"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_d9220254","line":32,"in_reply_to":"fa32b979_ac2cb602","updated":"2015-06-24 21:39:02.000000000","message":"MHO is that there is no golden rules for the number of retries. It certainly depends on the frequency of requests and the number of running instances (and the workload).\n\nIn general, the state of art speaks about a ratio retries/total requests and tries to keep it below a certain level.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"abe5f66ac36df66fbc62f9fa413910d2652783ab","unresolved":false,"context_lines":[{"line_number":30,"context_line":"  If the request ends up on a full compute node, the build errors out,"},{"line_number":31,"context_line":"  and lets the retry system find an different compute node to try."},{"line_number":32,"context_line":"  However, we stop retrying after three attempts, and this extends build"},{"line_number":33,"context_line":"  times for users, so its best to avoid these retries."},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"* Deployers often chose to fill first, to ensure they keep room for"},{"line_number":36,"context_line":"  larger flavors they offer. They then use the IO-ops filter to ensure you"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_d9ac428a","line":33,"updated":"2015-06-24 21:39:02.000000000","message":"I\u0027m not really sure retries are bad. The real problem is that we retry very late in the boot process (here, that\u0027s only when claiming in the compute manager).\n\nKeeping a retry mechanism done earlier in the process is IMHO good and would potentially reduce the problem to a sustainable level.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":11642,"name":"Jesse J. Cook","email":"jesse.cook@rackspace.com","username":"crashenx"},"change_message_id":"e3a6a73be652c6d208bad7e6a7bff20be0ebb0da","unresolved":false,"context_lines":[{"line_number":32,"context_line":"  However, we stop retrying after three attempts, and this extends build"},{"line_number":33,"context_line":"  times for users, so its best to avoid these retries."},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"* Deployers often chose to fill first, to ensure they keep room for"},{"line_number":36,"context_line":"  larger flavors they offer. They then use the IO-ops filter to ensure you"},{"line_number":37,"context_line":"  don\u0027t have too many builds happening on the same node at any time."},{"line_number":38,"context_line":"  Adding these together, this makes the above races much worse."}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_4c3e4a29","line":35,"updated":"2015-06-24 01:20:01.000000000","message":"Fill smaller slots first?","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"0f0bd1fda6cce20285abef1493e12d033eec55b2","unresolved":false,"context_lines":[{"line_number":32,"context_line":"  However, we stop retrying after three attempts, and this extends build"},{"line_number":33,"context_line":"  times for users, so its best to avoid these retries."},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"* Deployers often chose to fill first, to ensure they keep room for"},{"line_number":36,"context_line":"  larger flavors they offer. They then use the IO-ops filter to ensure you"},{"line_number":37,"context_line":"  don\u0027t have too many builds happening on the same node at any time."},{"line_number":38,"context_line":"  Adding these together, this makes the above races much worse."}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_ed161948","line":35,"in_reply_to":"fa32b979_4c3e4a29","updated":"2015-06-24 15:30:01.000000000","message":"no, basically you either fill first or spread first, depending on which way you configure the the RAM weighted.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"abe5f66ac36df66fbc62f9fa413910d2652783ab","unresolved":false,"context_lines":[{"line_number":35,"context_line":"* Deployers often chose to fill first, to ensure they keep room for"},{"line_number":36,"context_line":"  larger flavors they offer. They then use the IO-ops filter to ensure you"},{"line_number":37,"context_line":"  don\u0027t have too many builds happening on the same node at any time."},{"line_number":38,"context_line":"  Adding these together, this makes the above races much worse."},{"line_number":39,"context_line":""},{"line_number":40,"context_line":"* Randomization of decisions has been added to reduce the impact of the"},{"line_number":41,"context_line":"  races, but this means its making \u0027worse\u0027 decisions."}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_d9fa0277","line":38,"updated":"2015-06-24 21:39:02.000000000","message":"Nothing to say here, ++ on that","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":11642,"name":"Jesse J. Cook","email":"jesse.cook@rackspace.com","username":"crashenx"},"change_message_id":"e3a6a73be652c6d208bad7e6a7bff20be0ebb0da","unresolved":false,"context_lines":[{"line_number":37,"context_line":"  don\u0027t have too many builds happening on the same node at any time."},{"line_number":38,"context_line":"  Adding these together, this makes the above races much worse."},{"line_number":39,"context_line":""},{"line_number":40,"context_line":"* Randomization of decisions has been added to reduce the impact of the"},{"line_number":41,"context_line":"  races, but this means its making \u0027worse\u0027 decisions."},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"* Querying the DB is the most expensive part of the scheduling process."}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_cc181ad6","line":40,"updated":"2015-06-24 01:20:01.000000000","message":"Is this a reference to random weighted distribution or something that is merged upstream already (i.e. Random scheduler, which TBH I\"m not familiar with)?","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"0f0bd1fda6cce20285abef1493e12d033eec55b2","unresolved":false,"context_lines":[{"line_number":37,"context_line":"  don\u0027t have too many builds happening on the same node at any time."},{"line_number":38,"context_line":"  Adding these together, this makes the above races much worse."},{"line_number":39,"context_line":""},{"line_number":40,"context_line":"* Randomization of decisions has been added to reduce the impact of the"},{"line_number":41,"context_line":"  races, but this means its making \u0027worse\u0027 decisions."},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"* Querying the DB is the most expensive part of the scheduling process."}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_6a13cf65","line":40,"in_reply_to":"fa32b979_cc181ad6","updated":"2015-06-24 15:30:01.000000000","message":"Yes.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"abe5f66ac36df66fbc62f9fa413910d2652783ab","unresolved":false,"context_lines":[{"line_number":74,"context_line":"  within that cell."},{"line_number":75,"context_line":"  If you fail to build in a cell, there is no way to try another cell."},{"line_number":76,"context_line":"  If a cell gets full, there is no way to move a VM to another cell if it"},{"line_number":77,"context_line":"  needs to be resized up."},{"line_number":78,"context_line":""},{"line_number":79,"context_line":"* The reported slots have hidden dependencies."},{"line_number":80,"context_line":"  If you have space for a 2 GB VM, the system also reports two slots for a"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_b724e50c","line":77,"updated":"2015-06-24 21:39:02.000000000","message":"I\u0027m not sure the whole paragraph is necessary unless to provide why it\u0027s wrong : cells v1 is having 2 schedulers, one for providing a cell and one for vanilla scheduler, while we want only one scheduler for the whole cells.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":11642,"name":"Jesse J. Cook","email":"jesse.cook@rackspace.com","username":"crashenx"},"change_message_id":"e3a6a73be652c6d208bad7e6a7bff20be0ebb0da","unresolved":false,"context_lines":[{"line_number":92,"context_line":"  There are plans to randomly distribute builds between those slots, but"},{"line_number":93,"context_line":"  that just limits the impact of this problem, rather than eliminating it."},{"line_number":94,"context_line":""},{"line_number":95,"context_line":"* This is a described in the Google omega paper as \"two level scheduling\""},{"line_number":96,"context_line":""},{"line_number":97,"context_line":"So in summary, the current nova-scheduler works best when:"},{"line_number":98,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_2cdd06e0","line":95,"updated":"2015-06-24 01:20:01.000000000","message":"What is \"this\"?","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"0f0bd1fda6cce20285abef1493e12d033eec55b2","unresolved":false,"context_lines":[{"line_number":92,"context_line":"  There are plans to randomly distribute builds between those slots, but"},{"line_number":93,"context_line":"  that just limits the impact of this problem, rather than eliminating it."},{"line_number":94,"context_line":""},{"line_number":95,"context_line":"* This is a described in the Google omega paper as \"two level scheduling\""},{"line_number":96,"context_line":""},{"line_number":97,"context_line":"So in summary, the current nova-scheduler works best when:"},{"line_number":98,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_6d58c931","line":95,"in_reply_to":"fa32b979_2cdd06e0","updated":"2015-06-24 15:30:01.000000000","message":"cells scheduling, I think this should be part of the previous bullet point, oops.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":11642,"name":"Jesse J. Cook","email":"jesse.cook@rackspace.com","username":"crashenx"},"change_message_id":"e3a6a73be652c6d208bad7e6a7bff20be0ebb0da","unresolved":false,"context_lines":[{"line_number":114,"context_line":"----------"},{"line_number":115,"context_line":""},{"line_number":116,"context_line":"Consider a deployment where you have more than 10k hypervisors,"},{"line_number":117,"context_line":"with build request bursts of at least 1k within a 15 min period."},{"line_number":118,"context_line":""},{"line_number":119,"context_line":"This is particularly relevant to cells v1 users that are going to be"},{"line_number":120,"context_line":"migrate to cells v2."}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_4caf2a2a","line":117,"updated":"2015-06-24 01:20:01.000000000","message":"Good place for a scenario outline (http://pythonhosted.org/behave/tutorial.html#scenario-outlines), just sayin.\n\nI think it\u0027s worth considering a burst of 3K in 3s, and others.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"0f0bd1fda6cce20285abef1493e12d033eec55b2","unresolved":false,"context_lines":[{"line_number":114,"context_line":"----------"},{"line_number":115,"context_line":""},{"line_number":116,"context_line":"Consider a deployment where you have more than 10k hypervisors,"},{"line_number":117,"context_line":"with build request bursts of at least 1k within a 15 min period."},{"line_number":118,"context_line":""},{"line_number":119,"context_line":"This is particularly relevant to cells v1 users that are going to be"},{"line_number":120,"context_line":"migrate to cells v2."}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_cd49d561","line":117,"in_reply_to":"fa32b979_4caf2a2a","updated":"2015-06-24 15:30:01.000000000","message":"sure, it could be.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":11642,"name":"Jesse J. Cook","email":"jesse.cook@rackspace.com","username":"crashenx"},"change_message_id":"e3a6a73be652c6d208bad7e6a7bff20be0ebb0da","unresolved":false,"context_lines":[{"line_number":117,"context_line":"with build request bursts of at least 1k within a 15 min period."},{"line_number":118,"context_line":""},{"line_number":119,"context_line":"This is particularly relevant to cells v1 users that are going to be"},{"line_number":120,"context_line":"migrate to cells v2."},{"line_number":121,"context_line":""},{"line_number":122,"context_line":"Project Priority"},{"line_number":123,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_2c7986c8","line":120,"updated":"2015-06-24 01:20:01.000000000","message":"\"migrating\"","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"abe5f66ac36df66fbc62f9fa413910d2652783ab","unresolved":false,"context_lines":[{"line_number":142,"context_line":"that work well for particular use cases and usage patterns."},{"line_number":143,"context_line":""},{"line_number":144,"context_line":"With that in mind, I am going to focus on replacing the typical"},{"line_number":145,"context_line":"public cloud use cases as described above."},{"line_number":146,"context_line":""},{"line_number":147,"context_line":"Moving away from filters and weights"},{"line_number":148,"context_line":"+++++++++++++++++++++++++++++++++++++"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_d222d753","line":145,"updated":"2015-06-24 21:39:02.000000000","message":"I\u0027m not against leaving the door for any out-of-tree support, given it\u0027s very easy to redefine a separate scheduler driver.\n\nThat said, I think we need to draw a line and say \"okay, for the good or worst, we support ChanceScheduler and any scheduler driver inheriting from FilterScheduler, others are not relevant to that scope\" is scoping enough the backlog spec.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"7db0d0e904c1cafe2f41d29da1097442e76fc661","unresolved":false,"context_lines":[{"line_number":149,"context_line":""},{"line_number":150,"context_line":"An alternative to the existing scheduler is to have nova-compute nodes pull"},{"line_number":151,"context_line":"build requests from a shared queue, rather than being pushed work from a"},{"line_number":152,"context_line":"central scheduler."},{"line_number":153,"context_line":""},{"line_number":154,"context_line":"This works well for spread first, but requires some careful co-ordination"},{"line_number":155,"context_line":"to make a fill first scheduler work. You probably need a central system to"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_67877261","line":152,"updated":"2015-06-24 23:20:51.000000000","message":"What about mesos, doesn\u0027t it use a central-scheduler and then assigns work to nodes? Any idea what other scheduling systems do? Shared queues are nice as u say for scale-out but they do make it hard to have a global view and best placement (as the nodes pull eagerly), reminds me of https://en.wikipedia.org/wiki/A*_search_algorithm and such...\n\nDoes raise an interesting question though, could other projects (like mesos? or other) replace the scheduler completely (letting those projects handle resource allocation to nodes and best-fit and such...)? Nova-api then could just drop a request for some VM \u0027resource\u0027 to that other project/system (mesos, other..), and then let that other project decide where it goes (and then eventually some nova-compute will get a request to build a VM \u0027resource\u0027 from that project/system and carry on fulfilling that request using the existing code that already works).","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"288fa7bb67eae3e679d1eb6e21c4ec4ddb28da7b","unresolved":false,"context_lines":[{"line_number":149,"context_line":""},{"line_number":150,"context_line":"An alternative to the existing scheduler is to have nova-compute nodes pull"},{"line_number":151,"context_line":"build requests from a shared queue, rather than being pushed work from a"},{"line_number":152,"context_line":"central scheduler."},{"line_number":153,"context_line":""},{"line_number":154,"context_line":"This works well for spread first, but requires some careful co-ordination"},{"line_number":155,"context_line":"to make a fill first scheduler work. You probably need a central system to"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_e2ffedff","line":152,"in_reply_to":"fa32b979_67877261","updated":"2015-06-25 03:11:17.000000000","message":"For example take the diagram @ https://mesos.apache.org/documentation/latest/mesos-architecture/\n\nNow replace hadoop executor, or mpi executor with \u0027vm executor\u0027 and replace hadoop scheduler or mpi scheduler with nova-api (or nova-scheduler?), seems like that may just be a good fit (from the high level...)","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"e1b2740916cef04191355196a9ad6360200679f4","unresolved":false,"context_lines":[{"line_number":149,"context_line":""},{"line_number":150,"context_line":"An alternative to the existing scheduler is to have nova-compute nodes pull"},{"line_number":151,"context_line":"build requests from a shared queue, rather than being pushed work from a"},{"line_number":152,"context_line":"central scheduler."},{"line_number":153,"context_line":""},{"line_number":154,"context_line":"This works well for spread first, but requires some careful co-ordination"},{"line_number":155,"context_line":"to make a fill first scheduler work. You probably need a central system to"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_be356445","line":152,"in_reply_to":"fa32b979_80328ef0","updated":"2015-06-25 16:00:04.000000000","message":"Sure, and I understand all that, but this did seem alot like a brainstorming spec, so that was my brainstorm :-P\n\nIt seems like if hadoop, or others have been able to externalize there scheduler (via mesos, or yarn, or other) that we should be able to as well, although I know that\u0027s likely a long ways off (imho if we don\u0027t try to build a architecture that can do that, then that path will never be reached...); of course the gantt project has been trying to do something similar for a while now so perhaps this brainstorm idea is to \u0027drastic\u0027.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"6d0442e77c0424dcbb3067f8b74a6be5ed3bfae7","unresolved":false,"context_lines":[{"line_number":149,"context_line":""},{"line_number":150,"context_line":"An alternative to the existing scheduler is to have nova-compute nodes pull"},{"line_number":151,"context_line":"build requests from a shared queue, rather than being pushed work from a"},{"line_number":152,"context_line":"central scheduler."},{"line_number":153,"context_line":""},{"line_number":154,"context_line":"This works well for spread first, but requires some careful co-ordination"},{"line_number":155,"context_line":"to make a fill first scheduler work. You probably need a central system to"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ba15a1d1_b25602cc","line":152,"in_reply_to":"fa32b979_be356445","updated":"2015-09-18 15:09:30.000000000","message":"Anyone can just provide another out-of-tree scheduler backend and create a new scheduler driver inheriting from the base SchedulerDriver, that would work without needing to add it to the code because anyone can modify the default nova scheduler driver in CONF.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"9ed0987e70946e1920cd2bcfc446f8545c3cde76","unresolved":false,"context_lines":[{"line_number":149,"context_line":""},{"line_number":150,"context_line":"An alternative to the existing scheduler is to have nova-compute nodes pull"},{"line_number":151,"context_line":"build requests from a shared queue, rather than being pushed work from a"},{"line_number":152,"context_line":"central scheduler."},{"line_number":153,"context_line":""},{"line_number":154,"context_line":"This works well for spread first, but requires some careful co-ordination"},{"line_number":155,"context_line":"to make a fill first scheduler work. You probably need a central system to"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_80328ef0","line":152,"in_reply_to":"fa32b979_e2ffedff","updated":"2015-06-25 09:42:45.000000000","message":"Why not Mesos ? Because we want to have an internal in-tree solution for providing resources... Also, the current implementation is very tightly coupled with Nova so using a general Mesos proposal would just not help the problem.\n\nThat said, I\u0027m not against thinking of a scheduler driver able to use Mesos or something else, but we need stable APIs and an internal solution IMHO.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":11642,"name":"Jesse J. Cook","email":"jesse.cook@rackspace.com","username":"crashenx"},"change_message_id":"e3a6a73be652c6d208bad7e6a7bff20be0ebb0da","unresolved":false,"context_lines":[{"line_number":154,"context_line":"This works well for spread first, but requires some careful co-ordination"},{"line_number":155,"context_line":"to make a fill first scheduler work. You probably need a central system to"},{"line_number":156,"context_line":"give a new compute node permission to pull from the queue, or something like"},{"line_number":157,"context_line":"that."},{"line_number":158,"context_line":""},{"line_number":159,"context_line":"If you pull a message from the queue, and discover you are unable to service"},{"line_number":160,"context_line":"that request you could put that message back on the queue. Ideally you would"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_ce0a4f20","line":157,"updated":"2015-06-24 01:20:01.000000000","message":"Does it? Perhaps I\u0027m thinking about this differently, but I\u0027m not seeing this problem.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":11642,"name":"Jesse J. Cook","email":"jesse.cook@rackspace.com","username":"crashenx"},"change_message_id":"2481bd89d82403d72d001ddc66da780fd33da522","unresolved":false,"context_lines":[{"line_number":154,"context_line":"This works well for spread first, but requires some careful co-ordination"},{"line_number":155,"context_line":"to make a fill first scheduler work. You probably need a central system to"},{"line_number":156,"context_line":"give a new compute node permission to pull from the queue, or something like"},{"line_number":157,"context_line":"that."},{"line_number":158,"context_line":""},{"line_number":159,"context_line":"If you pull a message from the queue, and discover you are unable to service"},{"line_number":160,"context_line":"that request you could put that message back on the queue. Ideally you would"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ba3cc151_0edc2911","line":157,"in_reply_to":"fa32b979_cdbc7508","updated":"2015-06-29 23:25:05.000000000","message":"One simple solution is to have a rules system where you prefer to stay empty if you are empty or avoid small builds if you don\u0027t have any or w/e. Think about the problem from the standpoint of a rational actor that knows what the global state should look like but cannot see the global state. Game theory and robotics swarming come to mind.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"0f0bd1fda6cce20285abef1493e12d033eec55b2","unresolved":false,"context_lines":[{"line_number":154,"context_line":"This works well for spread first, but requires some careful co-ordination"},{"line_number":155,"context_line":"to make a fill first scheduler work. You probably need a central system to"},{"line_number":156,"context_line":"give a new compute node permission to pull from the queue, or something like"},{"line_number":157,"context_line":"that."},{"line_number":158,"context_line":""},{"line_number":159,"context_line":"If you pull a message from the queue, and discover you are unable to service"},{"line_number":160,"context_line":"that request you could put that message back on the queue. Ideally you would"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_cdbc7508","line":157,"in_reply_to":"fa32b979_ce0a4f20","updated":"2015-06-24 15:30:01.000000000","message":"So fill first means you want to keep as many hosts totally free as possible. As an empty host, you only want to take the build if it can\u0027t find on any other host that already has come servers on it. Thats quite hard to do without knowing the state of your peers. It all gets very racey.\n\nBut I am not sure I fully under stand the extent of the other approach.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":11642,"name":"Jesse J. Cook","email":"jesse.cook@rackspace.com","username":"crashenx"},"change_message_id":"e3a6a73be652c6d208bad7e6a7bff20be0ebb0da","unresolved":false,"context_lines":[{"line_number":156,"context_line":"give a new compute node permission to pull from the queue, or something like"},{"line_number":157,"context_line":"that."},{"line_number":158,"context_line":""},{"line_number":159,"context_line":"If you pull a message from the queue, and discover you are unable to service"},{"line_number":160,"context_line":"that request you could put that message back on the queue. Ideally you would"},{"line_number":161,"context_line":"shard the number of queues to limit the cases where such retries are needed."},{"line_number":162,"context_line":"Although, its hard to do that with per tenant affinity and anit-affinity"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_4e171f45","line":159,"updated":"2015-06-24 01:20:01.000000000","message":"How would that happen?","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"0f0bd1fda6cce20285abef1493e12d033eec55b2","unresolved":false,"context_lines":[{"line_number":156,"context_line":"give a new compute node permission to pull from the queue, or something like"},{"line_number":157,"context_line":"that."},{"line_number":158,"context_line":""},{"line_number":159,"context_line":"If you pull a message from the queue, and discover you are unable to service"},{"line_number":160,"context_line":"that request you could put that message back on the queue. Ideally you would"},{"line_number":161,"context_line":"shard the number of queues to limit the cases where such retries are needed."},{"line_number":162,"context_line":"Although, its hard to do that with per tenant affinity and anit-affinity"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_8d60ada6","line":159,"in_reply_to":"fa32b979_4e171f45","updated":"2015-06-24 15:30:01.000000000","message":"I don\u0027t want to add that detail here, I was assuming some oslo.messaging cast queue, but I am sure there is a better way.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":11642,"name":"Jesse J. Cook","email":"jesse.cook@rackspace.com","username":"crashenx"},"change_message_id":"2481bd89d82403d72d001ddc66da780fd33da522","unresolved":false,"context_lines":[{"line_number":156,"context_line":"give a new compute node permission to pull from the queue, or something like"},{"line_number":157,"context_line":"that."},{"line_number":158,"context_line":""},{"line_number":159,"context_line":"If you pull a message from the queue, and discover you are unable to service"},{"line_number":160,"context_line":"that request you could put that message back on the queue. Ideally you would"},{"line_number":161,"context_line":"shard the number of queues to limit the cases where such retries are needed."},{"line_number":162,"context_line":"Although, its hard to do that with per tenant affinity and anit-affinity"}],"source_content_type":"text/x-rst","patch_set":5,"id":"ba3cc151_eea6a5c0","line":159,"in_reply_to":"fa32b979_8d60ada6","updated":"2015-06-29 23:25:05.000000000","message":"No, I mean I don\u0027t think that should be a problem if you are pulling from the queue. If you pull something from the queue and discover you can\u0027t service it, you have a serious architectural issue.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":11642,"name":"Jesse J. Cook","email":"jesse.cook@rackspace.com","username":"crashenx"},"change_message_id":"e3a6a73be652c6d208bad7e6a7bff20be0ebb0da","unresolved":false,"context_lines":[{"line_number":162,"context_line":"Although, its hard to do that with per tenant affinity and anit-affinity"},{"line_number":163,"context_line":"rules."},{"line_number":164,"context_line":""},{"line_number":165,"context_line":"While I really want someone to explore building a driver like this,"},{"line_number":166,"context_line":"this spec is not considering this approach. It will instead focus on a more"},{"line_number":167,"context_line":"direct replacement for the current filter scheduler."},{"line_number":168,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_ee4dd372","line":165,"updated":"2015-06-24 01:20:01.000000000","message":"I have some thoughts about architecture for this approach, but I\u0027m not prepared to articulate them in this comment block.\n\nI really like systems that do not share or leak state, and I think pulling is likely an easier path to getting closer to such a system.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"0f0bd1fda6cce20285abef1493e12d033eec55b2","unresolved":false,"context_lines":[{"line_number":162,"context_line":"Although, its hard to do that with per tenant affinity and anit-affinity"},{"line_number":163,"context_line":"rules."},{"line_number":164,"context_line":""},{"line_number":165,"context_line":"While I really want someone to explore building a driver like this,"},{"line_number":166,"context_line":"this spec is not considering this approach. It will instead focus on a more"},{"line_number":167,"context_line":"direct replacement for the current filter scheduler."},{"line_number":168,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_2df721d4","line":165,"in_reply_to":"fa32b979_ee4dd372","updated":"2015-06-24 15:30:01.000000000","message":"Right, I support adding this, please submit a backlog spec for that.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"abe5f66ac36df66fbc62f9fa413910d2652783ab","unresolved":false,"context_lines":[{"line_number":164,"context_line":""},{"line_number":165,"context_line":"While I really want someone to explore building a driver like this,"},{"line_number":166,"context_line":"this spec is not considering this approach. It will instead focus on a more"},{"line_number":167,"context_line":"direct replacement for the current filter scheduler."},{"line_number":168,"context_line":""},{"line_number":169,"context_line":"Partitioning the Filter Scheduler"},{"line_number":170,"context_line":"++++++++++++++++++++++++++++++++++"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_08900e42","line":167,"updated":"2015-06-24 21:39:02.000000000","message":"That\u0027s why I think you need to explain more the scope of that spec (my remark L145) and say that even if it\u0027s valid to consider to get rid of nova-scheduler entirely, that\u0027s not here that we should talk about.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"abe5f66ac36df66fbc62f9fa413910d2652783ab","unresolved":false,"context_lines":[{"line_number":175,"context_line":"One major issue is interference between different schedulers. Ideally"},{"line_number":176,"context_line":"we don\u0027t want multiple schedulers assigning work to the same nova-compute"},{"line_number":177,"context_line":"nodes, as they will be competing with each other for the same resources."},{"line_number":178,"context_line":"Ideally each scheduler would be looking a different subset of hosts."},{"line_number":179,"context_line":""},{"line_number":180,"context_line":"Fighting this requirement are cluster wide behaviors, such as affinity and"},{"line_number":181,"context_line":"anti-affinity rules, where ideally we need to know the full state of the"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_93652e1f","line":178,"updated":"2015-06-24 21:39:02.000000000","message":"Still referring to the Google Omega paper, it\u0027s referred as sub-optimal. So, I\u0027m not particularly against that, but we need to be very specific about the sharding line. Honestly, I have serious doubts about that unless we leave the possibility to the operator to define how to shard.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":11642,"name":"Jesse J. Cook","email":"jesse.cook@rackspace.com","username":"crashenx"},"change_message_id":"e3a6a73be652c6d208bad7e6a7bff20be0ebb0da","unresolved":false,"context_lines":[{"line_number":178,"context_line":"Ideally each scheduler would be looking a different subset of hosts."},{"line_number":179,"context_line":""},{"line_number":180,"context_line":"Fighting this requirement are cluster wide behaviors, such as affinity and"},{"line_number":181,"context_line":"anti-affinity rules, where ideally we need to know the full state of the"},{"line_number":182,"context_line":"system, rather than just looking at a subset of the system."},{"line_number":183,"context_line":""},{"line_number":184,"context_line":"Its possible to have a dynamic partitioning, but for simplicity, I am going"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_ce6fafc2","line":181,"updated":"2015-06-24 01:20:01.000000000","message":"Ugh, full system state knowledge \u003d\u003d lots of complexity. It\u0027d be nice to instead aim for an intelligent rules based system that doesn\u0027t require state about other parts of the system / partitions / w/e.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"0f0bd1fda6cce20285abef1493e12d033eec55b2","unresolved":false,"context_lines":[{"line_number":178,"context_line":"Ideally each scheduler would be looking a different subset of hosts."},{"line_number":179,"context_line":""},{"line_number":180,"context_line":"Fighting this requirement are cluster wide behaviors, such as affinity and"},{"line_number":181,"context_line":"anti-affinity rules, where ideally we need to know the full state of the"},{"line_number":182,"context_line":"system, rather than just looking at a subset of the system."},{"line_number":183,"context_line":""},{"line_number":184,"context_line":"Its possible to have a dynamic partitioning, but for simplicity, I am going"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_4ddb854f","line":181,"in_reply_to":"fa32b979_ce6fafc2","updated":"2015-06-24 15:30:01.000000000","message":"Thats what the filter scheduler is, we can build other schedulers, but this is not the scope of this idea here.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"abe5f66ac36df66fbc62f9fa413910d2652783ab","unresolved":false,"context_lines":[{"line_number":182,"context_line":"system, rather than just looking at a subset of the system."},{"line_number":183,"context_line":""},{"line_number":184,"context_line":"Its possible to have a dynamic partitioning, but for simplicity, I am going"},{"line_number":185,"context_line":"to focus on static partitions of the system."},{"line_number":186,"context_line":"The problem with a static partition is that they tend to have capacity"},{"line_number":187,"context_line":"planning implications. If a subset of all requests get routed to a particular"},{"line_number":188,"context_line":"set of hosts, then you need to ensure you increase the number of hosts to"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_d37f36a9","line":185,"updated":"2015-06-24 21:39:02.000000000","message":"Hell yeah...","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":11642,"name":"Jesse J. Cook","email":"jesse.cook@rackspace.com","username":"crashenx"},"change_message_id":"e3a6a73be652c6d208bad7e6a7bff20be0ebb0da","unresolved":false,"context_lines":[{"line_number":185,"context_line":"to focus on static partitions of the system."},{"line_number":186,"context_line":"The problem with a static partition is that they tend to have capacity"},{"line_number":187,"context_line":"planning implications. If a subset of all requests get routed to a particular"},{"line_number":188,"context_line":"set of hosts, then you need to ensure you increase the number of hosts to"},{"line_number":189,"context_line":"match the demand for that subset of hosts."},{"line_number":190,"context_line":""},{"line_number":191,"context_line":"In cells v1, the top level scheduling was used to try and spread the load"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_0e883770","line":188,"updated":"2015-06-24 01:20:01.000000000","message":"\"increase the number of hosts to match the demand for that subset of hosts\"\n\nwha?","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"0f0bd1fda6cce20285abef1493e12d033eec55b2","unresolved":false,"context_lines":[{"line_number":185,"context_line":"to focus on static partitions of the system."},{"line_number":186,"context_line":"The problem with a static partition is that they tend to have capacity"},{"line_number":187,"context_line":"planning implications. If a subset of all requests get routed to a particular"},{"line_number":188,"context_line":"set of hosts, then you need to ensure you increase the number of hosts to"},{"line_number":189,"context_line":"match the demand for that subset of hosts."},{"line_number":190,"context_line":""},{"line_number":191,"context_line":"In cells v1, the top level scheduling was used to try and spread the load"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_ed985977","line":188,"in_reply_to":"fa32b979_0e883770","updated":"2015-06-24 15:30:01.000000000","message":"If you have more requests for spinning disks than SSDs, you need to add more capacity to your spinning disks than your SSDs.\n\nNow if you do a hash of the tenant id to some subset of hosts, you get the same thing. If that partition wants more capacity than the other partition, you need to increase the capacity accordingly.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"abe5f66ac36df66fbc62f9fa413910d2652783ab","unresolved":false,"context_lines":[{"line_number":186,"context_line":"The problem with a static partition is that they tend to have capacity"},{"line_number":187,"context_line":"planning implications. If a subset of all requests get routed to a particular"},{"line_number":188,"context_line":"set of hosts, then you need to ensure you increase the number of hosts to"},{"line_number":189,"context_line":"match the demand for that subset of hosts."},{"line_number":190,"context_line":""},{"line_number":191,"context_line":"In cells v1, the top level scheduling was used to try and spread the load"},{"line_number":192,"context_line":"between lots of groups that get added as you expand, but this two level"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_b34f1291","line":189,"updated":"2015-06-24 21:39:02.000000000","message":"That\u0027s what I call suboptimal, because you could exhaust the capacity of your shard while some other shards could yet accept some requests.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"abe5f66ac36df66fbc62f9fa413910d2652783ab","unresolved":false,"context_lines":[{"line_number":197,"context_line":""},{"line_number":198,"context_line":"* Certain groups of hosts can have specific hardware mapped to"},{"line_number":199,"context_line":"  specific flavors."},{"line_number":200,"context_line":"  i.e. SSD vs non-SSD local storage vs all storage form cinder (no local disk)"},{"line_number":201,"context_line":""},{"line_number":202,"context_line":"* Keeping Windows and Linux VMs on different sets of hypervisors is common"},{"line_number":203,"context_line":"  place, to allow for the best utilization of bulk license savings."}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_9e4f756a","line":200,"updated":"2015-06-24 21:39:02.000000000","message":"that necessarly implies that one filter is enabled with the images correctly tagged. All of that seems very operator-specific.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"abe5f66ac36df66fbc62f9fa413910d2652783ab","unresolved":false,"context_lines":[{"line_number":202,"context_line":"* Keeping Windows and Linux VMs on different sets of hypervisors is common"},{"line_number":203,"context_line":"  place, to allow for the best utilization of bulk license savings."},{"line_number":204,"context_line":"  This is a very similar distinct split between hosts based on the users"},{"line_number":205,"context_line":"  build request."},{"line_number":206,"context_line":""},{"line_number":207,"context_line":"Lets consider having a separate nova-scheduler cluster for each of these"},{"line_number":208,"context_line":"groups of hosts. We can route requests to each scheduler cluster based on"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_7e0ab121","line":205,"updated":"2015-06-24 21:39:02.000000000","message":"Yeah, the sharding has to be exclusive so that we\u0027re 100% sure that a request would end up to a specific shard and not to the other anyway.\n\nAvailability zones could be one other way to shard, since we know that one host is only part of one AZ, don\u0027t you think ?","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":5441,"name":"Andrew Laski","email":"andrew@lascii.com","username":"alaski"},"change_message_id":"18ab6652e7479119eb04643a6ca560907ffc2d66","unresolved":false,"context_lines":[{"line_number":209,"context_line":"the request spec. The flavor is required in all build requests, and can route"},{"line_number":210,"context_line":"you to one of each subset. Requests for global concepts such as affinity don\u0027t"},{"line_number":211,"context_line":"really make sense across these groups of hosts, and its possible that the"},{"line_number":212,"context_line":"request router could check these kinds of constraints."},{"line_number":213,"context_line":""},{"line_number":214,"context_line":"In a cells v2 world, you would have multiple cells in each group. For"},{"line_number":215,"context_line":"simplicity we can assume each complete cell would be registered to one"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_55233858","line":212,"updated":"2015-06-23 21:22:35.000000000","message":"I\u0027m not sure what this is saying.  Affinity doesn\u0027t make sense with clustering, but somehow a router could make it work?  How would affinity be handled?","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"abe5f66ac36df66fbc62f9fa413910d2652783ab","unresolved":false,"context_lines":[{"line_number":209,"context_line":"the request spec. The flavor is required in all build requests, and can route"},{"line_number":210,"context_line":"you to one of each subset. Requests for global concepts such as affinity don\u0027t"},{"line_number":211,"context_line":"really make sense across these groups of hosts, and its possible that the"},{"line_number":212,"context_line":"request router could check these kinds of constraints."},{"line_number":213,"context_line":""},{"line_number":214,"context_line":"In a cells v2 world, you would have multiple cells in each group. For"},{"line_number":215,"context_line":"simplicity we can assume each complete cell would be registered to one"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_3ead09e9","line":212,"in_reply_to":"fa32b979_55233858","updated":"2015-06-24 21:39:02.000000000","message":"We could route based on RequestSpec, for sure, but again, it would mean that we need to be sure that hosts are properly splitted. That\u0027s why I only consider AZs as a valid option...","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":11642,"name":"Jesse J. Cook","email":"jesse.cook@rackspace.com","username":"crashenx"},"change_message_id":"e3a6a73be652c6d208bad7e6a7bff20be0ebb0da","unresolved":false,"context_lines":[{"line_number":214,"context_line":"In a cells v2 world, you would have multiple cells in each group. For"},{"line_number":215,"context_line":"simplicity we can assume each complete cell would be registered to one"},{"line_number":216,"context_line":"(and only one) of the scheduler clusters. In practice, we probably want each"},{"line_number":217,"context_line":"host to know what scheduler it should report things two."},{"line_number":218,"context_line":""},{"line_number":219,"context_line":"The nice property of this partition is that you need to do capacity planning"},{"line_number":220,"context_line":"for each of these groups of hosts independently, regardless of how the"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_4efddfaf","line":217,"updated":"2015-06-24 01:20:01.000000000","message":"Think I\u0027m a bit lost at this point or missing some important bits to grok the importance of this bit.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"abe5f66ac36df66fbc62f9fa413910d2652783ab","unresolved":false,"context_lines":[{"line_number":214,"context_line":"In a cells v2 world, you would have multiple cells in each group. For"},{"line_number":215,"context_line":"simplicity we can assume each complete cell would be registered to one"},{"line_number":216,"context_line":"(and only one) of the scheduler clusters. In practice, we probably want each"},{"line_number":217,"context_line":"host to know what scheduler it should report things two."},{"line_number":218,"context_line":""},{"line_number":219,"context_line":"The nice property of this partition is that you need to do capacity planning"},{"line_number":220,"context_line":"for each of these groups of hosts independently, regardless of how the"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_7ecb91a4","line":217,"in_reply_to":"fa32b979_4efddfaf","updated":"2015-06-24 21:39:02.000000000","message":"Sharding per cell would be working, but it would imply that the user expresses which cell he wants to be in, which breaks the model IMHO because it\u0027s implicit.\n\nMapping the cell with an AZ could work, because it would make the split explicit.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":11642,"name":"Jesse J. Cook","email":"jesse.cook@rackspace.com","username":"crashenx"},"change_message_id":"e3a6a73be652c6d208bad7e6a7bff20be0ebb0da","unresolved":false,"context_lines":[{"line_number":228,"context_line":"you end up having to expand capacity differently across each of the groups"},{"line_number":229,"context_line":"as the demand from those different tenants goes up and down."},{"line_number":230,"context_line":"When each of those schedulers look at overlapping subsets of the nodes, you"},{"line_number":231,"context_line":"improve the spread of resources, but you tent to end up with some interference"},{"line_number":232,"context_line":"between the different scheduler clusters."},{"line_number":233,"context_line":""},{"line_number":234,"context_line":"While some of these alternative partitioning schemes may well be useful once"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_8e1e870a","line":231,"updated":"2015-06-24 01:20:01.000000000","message":"tend","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":11642,"name":"Jesse J. Cook","email":"jesse.cook@rackspace.com","username":"crashenx"},"change_message_id":"e3a6a73be652c6d208bad7e6a7bff20be0ebb0da","unresolved":false,"context_lines":[{"line_number":229,"context_line":"as the demand from those different tenants goes up and down."},{"line_number":230,"context_line":"When each of those schedulers look at overlapping subsets of the nodes, you"},{"line_number":231,"context_line":"improve the spread of resources, but you tent to end up with some interference"},{"line_number":232,"context_line":"between the different scheduler clusters."},{"line_number":233,"context_line":""},{"line_number":234,"context_line":"While some of these alternative partitioning schemes may well be useful once"},{"line_number":235,"context_line":"we have some of the other enhancements discussed here, I am limiting the scope"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_eebab3e0","line":232,"updated":"2015-06-24 01:20:01.000000000","message":"Tenants are important to an affinity algorithm and partitioning on would likely add a fair bit of complexity to manage. Maybe I\u0027m missing something though.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"abe5f66ac36df66fbc62f9fa413910d2652783ab","unresolved":false,"context_lines":[{"line_number":230,"context_line":"When each of those schedulers look at overlapping subsets of the nodes, you"},{"line_number":231,"context_line":"improve the spread of resources, but you tent to end up with some interference"},{"line_number":232,"context_line":"between the different scheduler clusters."},{"line_number":233,"context_line":""},{"line_number":234,"context_line":"While some of these alternative partitioning schemes may well be useful once"},{"line_number":235,"context_line":"we have some of the other enhancements discussed here, I am limiting the scope"},{"line_number":236,"context_line":"of this spec to the simplest partitioning scheme, a distinct partitioning of"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_1ed12523","line":233,"updated":"2015-06-24 21:39:02.000000000","message":"Tenant sharding would be OK, but since it\u0027s suboptimal, it would be left to the operator to enable it or not. That\u0027s not really far from the concept of AggregateMultiTenancy filter you know...","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":5441,"name":"Andrew Laski","email":"andrew@lascii.com","username":"alaski"},"change_message_id":"18ab6652e7479119eb04643a6ca560907ffc2d66","unresolved":false,"context_lines":[{"line_number":237,"context_line":"hosts based on the requested flavor, for the initial version."},{"line_number":238,"context_line":"The major downside of this approach is it limits the impact of partitioning to"},{"line_number":239,"context_line":"the very largest cloud deployments, those where there are several distinct"},{"line_number":240,"context_line":"groups of hosts that have their capacity managed separately."},{"line_number":241,"context_line":""},{"line_number":242,"context_line":"Using the Resource Tracker to implement \"distributed\" locking"},{"line_number":243,"context_line":"++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_2a496eb7","line":240,"updated":"2015-06-23 21:22:35.000000000","message":"All I can think while reading this is that this should be an alternate to the filtering scheduler and not an enhancement on top of it.\n\nThere are many ways to partition, and some constraints around deployments (homogenous hardware profiles in a cell) that I don\u0027t think should be enforced in the current filter scheduler.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"0f0bd1fda6cce20285abef1493e12d033eec55b2","unresolved":false,"context_lines":[{"line_number":237,"context_line":"hosts based on the requested flavor, for the initial version."},{"line_number":238,"context_line":"The major downside of this approach is it limits the impact of partitioning to"},{"line_number":239,"context_line":"the very largest cloud deployments, those where there are several distinct"},{"line_number":240,"context_line":"groups of hosts that have their capacity managed separately."},{"line_number":241,"context_line":""},{"line_number":242,"context_line":"Using the Resource Tracker to implement \"distributed\" locking"},{"line_number":243,"context_line":"++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_cdc23534","line":240,"in_reply_to":"fa32b979_2a496eb7","updated":"2015-06-24 15:30:01.000000000","message":"Its a layering on top, I think, true.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"abe5f66ac36df66fbc62f9fa413910d2652783ab","unresolved":false,"context_lines":[{"line_number":237,"context_line":"hosts based on the requested flavor, for the initial version."},{"line_number":238,"context_line":"The major downside of this approach is it limits the impact of partitioning to"},{"line_number":239,"context_line":"the very largest cloud deployments, those where there are several distinct"},{"line_number":240,"context_line":"groups of hosts that have their capacity managed separately."},{"line_number":241,"context_line":""},{"line_number":242,"context_line":"Using the Resource Tracker to implement \"distributed\" locking"},{"line_number":243,"context_line":"++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_7e839103","line":240,"in_reply_to":"fa32b979_2a496eb7","updated":"2015-06-24 21:39:02.000000000","message":"Yup, I just wonder if we could leave the routing able to be defined by the operator without providing a filterscheduler on top. Quite difficult, because it could be confusing people...","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":11642,"name":"Jesse J. Cook","email":"jesse.cook@rackspace.com","username":"crashenx"},"change_message_id":"e3a6a73be652c6d208bad7e6a7bff20be0ebb0da","unresolved":false,"context_lines":[{"line_number":244,"context_line":""},{"line_number":245,"context_line":"There have been various discussions about having the resource tracker persist"},{"line_number":246,"context_line":"the resource claims it hands out, so those claims persist across a"},{"line_number":247,"context_line":"nova-compute service restart. On top of that, we can add some RPC calls so the"},{"line_number":248,"context_line":"nova-conductor, or any other node, would be able to acquire one of these"},{"line_number":249,"context_line":"claims during VM move operations, such as resize and live-migrate, where you"},{"line_number":250,"context_line":"don\u0027t want new VM builds taking up space you are about to use once you have"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_8e93e75a","line":247,"updated":"2015-06-24 01:20:01.000000000","message":"This sounds like a giant ball of complexity, so I just stopped reading here. Locking in this context in general seems like a bandaid for a design bug.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"0f0bd1fda6cce20285abef1493e12d033eec55b2","unresolved":false,"context_lines":[{"line_number":244,"context_line":""},{"line_number":245,"context_line":"There have been various discussions about having the resource tracker persist"},{"line_number":246,"context_line":"the resource claims it hands out, so those claims persist across a"},{"line_number":247,"context_line":"nova-compute service restart. On top of that, we can add some RPC calls so the"},{"line_number":248,"context_line":"nova-conductor, or any other node, would be able to acquire one of these"},{"line_number":249,"context_line":"claims during VM move operations, such as resize and live-migrate, where you"},{"line_number":250,"context_line":"don\u0027t want new VM builds taking up space you are about to use once you have"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_2d24010b","line":247,"in_reply_to":"fa32b979_8e93e75a","updated":"2015-06-24 15:30:01.000000000","message":"We do this already, this just changes where it happens.\n\nI see it as partitioning your lock space to remove contention on those locks. It doesn\u0027t feel to bad to me, but I could be missing something.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"abe5f66ac36df66fbc62f9fa413910d2652783ab","unresolved":false,"context_lines":[{"line_number":254,"context_line":"get a leak of capacity due to un-used resource tracker claims."},{"line_number":255,"context_line":"This moves what could be a distributed locking mechanism to a per nova-compute"},{"line_number":256,"context_line":"locking system, that should mean there is much less lock contention, and"},{"line_number":257,"context_line":"generally its a much easier problem to solve."},{"line_number":258,"context_line":""},{"line_number":259,"context_line":"When the resource tracker reports its current available resources up to the"},{"line_number":260,"context_line":"scheduler it would reduce the amount of free resources to take account of the"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_d96af780","line":257,"updated":"2015-06-24 21:39:02.000000000","message":"I\u0027m not a big fan of persistent claims, because I consider that the reason for persisting them is not the root cause. The root cause is that we don\u0027t have a scaling-out scheduler able to do transaction locking on a distributed HostState, so we rely on the compute manager for ensuring that the request can be fulfilled.\nFrom my perspective, I\u0027d rather try to claim on the scheduler when consuming_from_instance() so it would then recall the scheduler if not fine, and leave the compute manager to just easily verify if the request can be done as a last-standing-man. That way, it would not need to persist the claims on the compute manager IMHO.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"abe5f66ac36df66fbc62f9fa413910d2652783ab","unresolved":false,"context_lines":[{"line_number":263,"context_line":"Now consider if the scheduler was able to acquire one of these claims before"},{"line_number":264,"context_line":"returning the chosen host to the nova-conductor. This would be moving the"},{"line_number":265,"context_line":"claim request from the very start of the build process in nova-compute into"},{"line_number":266,"context_line":"the scheduler."},{"line_number":267,"context_line":"This would allow the scheduler to build up a collection of claims for the"},{"line_number":268,"context_line":"requested resources before returning the choice to the caller what resources"},{"line_number":269,"context_line":"the scheduler has chosen. Should there be a problem detected, the scheduler"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_590d87e6","line":266,"updated":"2015-06-24 21:39:02.000000000","message":"++","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"abe5f66ac36df66fbc62f9fa413910d2652783ab","unresolved":false,"context_lines":[{"line_number":268,"context_line":"requested resources before returning the choice to the caller what resources"},{"line_number":269,"context_line":"the scheduler has chosen. Should there be a problem detected, the scheduler"},{"line_number":270,"context_line":"can perform retries until it gets all the claims required for the given"},{"line_number":271,"context_line":"resource request made to the scheduler."},{"line_number":272,"context_line":""},{"line_number":273,"context_line":"Putting this all together, you now see that the schedulers will start to see"},{"line_number":274,"context_line":"each others decisions because the claims acquired by another scheduler show up"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_79272b63","line":271,"updated":"2015-06-24 21:39:02.000000000","message":"You could mention http://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/host-state-level-locking.html","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"abe5f66ac36df66fbc62f9fa413910d2652783ab","unresolved":false,"context_lines":[{"line_number":277,"context_line":"Taking this a step further we could ensure that a scheduler waits for the"},{"line_number":278,"context_line":"claim it just took to show up in the shared state before returning the"},{"line_number":279,"context_line":"compute node choice to the scheduler\u0027s caller."},{"line_number":280,"context_line":""},{"line_number":281,"context_line":"Another possible twist is to consider a claim system very similar to the"},{"line_number":282,"context_line":"\"compare and swap\" DB call system. When the scheduler makes a claim, it could"},{"line_number":283,"context_line":"tell the compute node only to give out that claim if the compute node still"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_59b26707","line":280,"updated":"2015-06-24 21:39:02.000000000","message":"The missing part of Nikola\u0027s spec is that it doesn\u0027t consider a distributed HostState. IMHO, that\u0027s where the effort has to be done.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"abe5f66ac36df66fbc62f9fa413910d2652783ab","unresolved":false,"context_lines":[{"line_number":287,"context_line":"It could be done by having a hash of the currently reported node state,"},{"line_number":288,"context_line":"and comparing that. Its assumed such a hash would not change when an instance"},{"line_number":289,"context_line":"goes from the claimed state to a state where it is using that claim."},{"line_number":290,"context_line":""},{"line_number":291,"context_line":"It seems likely that a combination of these strategies should help ensure the"},{"line_number":292,"context_line":"scheduler is able to deal with most races between other parallel schedulers"},{"line_number":293,"context_line":"before returning the chosen compute node to the scheduler caller. This should"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_d99d1767","line":290,"updated":"2015-06-24 21:39:02.000000000","message":"Probably, but I don\u0027t see much benefit of that, with all the issues given by passing the hash to the compute manager (TTL etc.)","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":11642,"name":"Jesse J. Cook","email":"jesse.cook@rackspace.com","username":"crashenx"},"change_message_id":"e3a6a73be652c6d208bad7e6a7bff20be0ebb0da","unresolved":false,"context_lines":[{"line_number":305,"context_line":"being stale."},{"line_number":306,"context_line":"And interesting alternative is to just consume the updates to the current"},{"line_number":307,"context_line":"state, rather than having to fetch the full copy of the host state every time:"},{"line_number":308,"context_line":"https://blueprints.launchpad.net/nova/+spec/no-db-scheduler"},{"line_number":309,"context_line":""},{"line_number":310,"context_line":"This is very similar to a shared state scheduler discussed in the omega paper."},{"line_number":311,"context_line":"In this case the shared state is implemented using an in memory structure in"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_aefa6b75","line":308,"updated":"2015-06-24 01:20:01.000000000","message":"This is a long spec, and I\u0027m running out of steam, so I\u0027m gonna skip the details on this bit for now.\n\nI really do like the idea of consuming update notifications. Although, I prefer getting rid of global state and consuming build requests instead.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"0f0bd1fda6cce20285abef1493e12d033eec55b2","unresolved":false,"context_lines":[{"line_number":305,"context_line":"being stale."},{"line_number":306,"context_line":"And interesting alternative is to just consume the updates to the current"},{"line_number":307,"context_line":"state, rather than having to fetch the full copy of the host state every time:"},{"line_number":308,"context_line":"https://blueprints.launchpad.net/nova/+spec/no-db-scheduler"},{"line_number":309,"context_line":""},{"line_number":310,"context_line":"This is very similar to a shared state scheduler discussed in the omega paper."},{"line_number":311,"context_line":"In this case the shared state is implemented using an in memory structure in"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_4dcf0529","line":308,"in_reply_to":"fa32b979_aefa6b75","updated":"2015-06-24 15:30:01.000000000","message":"Right, but thats not the filter scheduler. Agreed we should explore both. My preference is this one right now, because of my rambling in the alternatives section. Its not very well informed ramblings though.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"abe5f66ac36df66fbc62f9fa413910d2652783ab","unresolved":false,"context_lines":[{"line_number":318,"context_line":"starting to service any requests."},{"line_number":319,"context_line":"Making sure all computes report their full state occasionally means there is a"},{"line_number":320,"context_line":"point where you can trim the old updates and still get a full view of the"},{"line_number":321,"context_line":"the full system."},{"line_number":322,"context_line":""},{"line_number":323,"context_line":"The pain point of friction with the no-db-scheduler was the complexity of"},{"line_number":324,"context_line":"maintaining the code that look a lot like the implementation of a DB log."}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_dba501a2","line":321,"updated":"2015-06-24 21:39:02.000000000","message":"Interesting proposal for giving a shared state. It also helps by providing the stats incrementally and not for every 60 secs.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"abe5f66ac36df66fbc62f9fa413910d2652783ab","unresolved":false,"context_lines":[{"line_number":323,"context_line":"The pain point of friction with the no-db-scheduler was the complexity of"},{"line_number":324,"context_line":"maintaining the code that look a lot like the implementation of a DB log."},{"line_number":325,"context_line":"Being able to efficiently trim old updates, so any new schedulers have only"},{"line_number":326,"context_line":"have a small amount of data to catch up."},{"line_number":327,"context_line":"It turns our Kafka has already implemented at lot of these semantics and is"},{"line_number":328,"context_line":"has already been proven to work at an extremely large scale:"},{"line_number":329,"context_line":"http://kafka.apache.org/documentation.html#introduction"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_fbcec558","line":326,"updated":"2015-06-24 21:39:02.000000000","message":"The main problem with no-db-scheduler was that it was a huge change, not only providing incremental updates but more than that.\nHere, if we are clear about just providing a system for updating the HostState, it would be easier.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"abe5f66ac36df66fbc62f9fa413910d2652783ab","unresolved":false,"context_lines":[{"line_number":326,"context_line":"have a small amount of data to catch up."},{"line_number":327,"context_line":"It turns our Kafka has already implemented at lot of these semantics and is"},{"line_number":328,"context_line":"has already been proven to work at an extremely large scale:"},{"line_number":329,"context_line":"http://kafka.apache.org/documentation.html#introduction"},{"line_number":330,"context_line":""},{"line_number":331,"context_line":"It seems we should be able to create a kafka based system to get efficient"},{"line_number":332,"context_line":"incremental updates to the current state of the system, rather than having to"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_1b336963","line":329,"updated":"2015-06-24 21:39:02.000000000","message":"What I would love to see is how we could have Kafka as a driver and possibly not something mandatory. Thinking about a DB incremental driver would be cool too IMHO.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":5441,"name":"Andrew Laski","email":"andrew@lascii.com","username":"alaski"},"change_message_id":"18ab6652e7479119eb04643a6ca560907ffc2d66","unresolved":false,"context_lines":[{"line_number":331,"context_line":"It seems we should be able to create a kafka based system to get efficient"},{"line_number":332,"context_line":"incremental updates to the current state of the system, rather than having to"},{"line_number":333,"context_line":"make the expensive DB call to get the state for all the hosts we are"},{"line_number":334,"context_line":"interested in."},{"line_number":335,"context_line":""},{"line_number":336,"context_line":"Memory concerns"},{"line_number":337,"context_line":"++++++++++++++++"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_4a8b92c3","line":334,"updated":"2015-06-23 21:22:35.000000000","message":"This is similar to https://blueprints.launchpad.net/nova/+spec/isolate-scheduler-db right?  Kafka looks nifty and I could see it as an optional component used in a scheduler.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"0f0bd1fda6cce20285abef1493e12d033eec55b2","unresolved":false,"context_lines":[{"line_number":331,"context_line":"It seems we should be able to create a kafka based system to get efficient"},{"line_number":332,"context_line":"incremental updates to the current state of the system, rather than having to"},{"line_number":333,"context_line":"make the expensive DB call to get the state for all the hosts we are"},{"line_number":334,"context_line":"interested in."},{"line_number":335,"context_line":""},{"line_number":336,"context_line":"Memory concerns"},{"line_number":337,"context_line":"++++++++++++++++"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_8d974deb","line":334,"in_reply_to":"fa32b979_4a8b92c3","updated":"2015-06-24 15:30:01.000000000","message":"The above blueprint having been completed should make it easier.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"abe5f66ac36df66fbc62f9fa413910d2652783ab","unresolved":false,"context_lines":[{"line_number":331,"context_line":"It seems we should be able to create a kafka based system to get efficient"},{"line_number":332,"context_line":"incremental updates to the current state of the system, rather than having to"},{"line_number":333,"context_line":"make the expensive DB call to get the state for all the hosts we are"},{"line_number":334,"context_line":"interested in."},{"line_number":335,"context_line":""},{"line_number":336,"context_line":"Memory concerns"},{"line_number":337,"context_line":"++++++++++++++++"}],"source_content_type":"text/x-rst","patch_set":5,"id":"fa32b979_7b1715a3","line":334,"in_reply_to":"fa32b979_8d974deb","updated":"2015-06-24 21:39:02.000000000","message":"A queue like Kafka would help the above blueprint by removing the fanout RPC cast that I really dislike.","commit_id":"abd35eedfccd899adc0a2280bbcfe8551641a8c1"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"00cb75f3e07ae236abedff0ebc62e3e487d35c32","unresolved":false,"context_lines":[{"line_number":45,"context_line":"* The C based DB driver and eventlet means the scheduler performs best"},{"line_number":46,"context_line":"  when the eventlet thread pool is very small, ideally less than 5."},{"line_number":47,"context_line":"  Without that, you find it makes several DB calls before processing"},{"line_number":48,"context_line":"  the (now stale) information it has fetched from the DB."},{"line_number":49,"context_line":""},{"line_number":50,"context_line":"* The Caching Scheduler was added to periodically update the host state"},{"line_number":51,"context_line":"  in a periodic task, so no user request has to wait for the DB query."}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_5145cb6d","line":48,"updated":"2015-11-10 12:23:34.000000000","message":"Say what? The scheduler doesn\u0027t do direct DB queries... it calls the conductor via the nova.objects.ComputeNode.get_all() interface. In any case, this mention of the eventlet thread pool is distracting and not germane to the spec at hand, IMHO.","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"11c241013f5c87f31506d27e40ce11b8c2c3f838","unresolved":false,"context_lines":[{"line_number":45,"context_line":"* The C based DB driver and eventlet means the scheduler performs best"},{"line_number":46,"context_line":"  when the eventlet thread pool is very small, ideally less than 5."},{"line_number":47,"context_line":"  Without that, you find it makes several DB calls before processing"},{"line_number":48,"context_line":"  the (now stale) information it has fetched from the DB."},{"line_number":49,"context_line":""},{"line_number":50,"context_line":"* The Caching Scheduler was added to periodically update the host state"},{"line_number":51,"context_line":"  in a periodic task, so no user request has to wait for the DB query."}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_53d5cab6","line":48,"in_reply_to":"da85f559_5145cb6d","updated":"2015-11-10 13:51:25.000000000","message":"No, the scheduler does direct DB access thru the NovaObject layer because it doesn\u0027t have the indirection API compared to n-cpu :\nhttps://github.com/openstack/nova/blob/master/nova/cmd/scheduler.py\n\nhttps://github.com/openstack/nova/blob/master/nova/cmd/compute.py#L67-L68","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"84e1d4d39896453f091e48e753b3cc40bd3959ce","unresolved":false,"context_lines":[{"line_number":87,"context_line":"  scheduling decisions, and has no randomization."},{"line_number":88,"context_line":"  Consider two cells, one with 12x1GB slots another with 10x1GB slots."},{"line_number":89,"context_line":"  If you get 15 requests for a 1GB slot, they all get sent to the cell"},{"line_number":90,"context_line":"  reporting 12 slots. 2 of those build request will fail because that cell"},{"line_number":91,"context_line":"  has no room."},{"line_number":92,"context_line":"  There are plans to randomly distribute builds between those slots, but"},{"line_number":93,"context_line":"  that just limits the impact of this problem, rather than eliminating it."}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_80e34005","line":90,"updated":"2015-11-12 21:51:37.000000000","message":"3 will fail. 15 - 12 :)","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"00cb75f3e07ae236abedff0ebc62e3e487d35c32","unresolved":false,"context_lines":[{"line_number":98,"context_line":""},{"line_number":99,"context_line":"* there is a single nova-scheduler process running"},{"line_number":100,"context_line":""},{"line_number":101,"context_line":"* it periodically refreshes its state from the DB"},{"line_number":102,"context_line":""},{"line_number":103,"context_line":"* it updates its in memory state with any previous decisions it has made"},{"line_number":104,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_513cebe1","line":101,"updated":"2015-11-10 12:23:34.000000000","message":"\"periodically\" \u003d\u003d every time select_destinations() is called...","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"84e1d4d39896453f091e48e753b3cc40bd3959ce","unresolved":false,"context_lines":[{"line_number":119,"context_line":"This is particularly relevant to cells v1 users that are going to be"},{"line_number":120,"context_line":"migrate to cells v2."},{"line_number":121,"context_line":""},{"line_number":122,"context_line":"Project Priority"},{"line_number":123,"context_line":"-----------------"},{"line_number":124,"context_line":""},{"line_number":125,"context_line":"None"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_40c4785e","line":122,"updated":"2015-11-12 21:51:37.000000000","message":"This can be removed.","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"84e1d4d39896453f091e48e753b3cc40bd3959ce","unresolved":false,"context_lines":[{"line_number":129,"context_line":""},{"line_number":130,"context_line":"None"},{"line_number":131,"context_line":""},{"line_number":132,"context_line":"This spec is to agree the problem, and list some possible solutions."},{"line_number":133,"context_line":""},{"line_number":134,"context_line":"Alternatives"},{"line_number":135,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_a0a9a492","line":132,"updated":"2015-11-12 21:51:37.000000000","message":"s/agree the/agree about the/","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"00cb75f3e07ae236abedff0ebc62e3e487d35c32","unresolved":false,"context_lines":[{"line_number":157,"context_line":"that."},{"line_number":158,"context_line":""},{"line_number":159,"context_line":"If you pull a message from the queue, and discover you are unable to service"},{"line_number":160,"context_line":"that request you could put that message back on the queue. Ideally you would"},{"line_number":161,"context_line":"shard the number of queues to limit the cases where such retries are needed."},{"line_number":162,"context_line":"Although, its hard to do that with per tenant affinity and anit-affinity"},{"line_number":163,"context_line":"rules."}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_c46337db","line":160,"updated":"2015-11-10 12:23:34.000000000","message":"But then you get into problems of needing to track the original message. If the compute node were to fail after pulling the message from the queue and before pushing the message back to the queue, your schedule request would be lost.","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":5441,"name":"Andrew Laski","email":"andrew@lascii.com","username":"alaski"},"change_message_id":"2dd5b363e6d43a1de94bab66a641e7c3af61287c","unresolved":false,"context_lines":[{"line_number":159,"context_line":"If you pull a message from the queue, and discover you are unable to service"},{"line_number":160,"context_line":"that request you could put that message back on the queue. Ideally you would"},{"line_number":161,"context_line":"shard the number of queues to limit the cases where such retries are needed."},{"line_number":162,"context_line":"Although, its hard to do that with per tenant affinity and anit-affinity"},{"line_number":163,"context_line":"rules."},{"line_number":164,"context_line":""},{"line_number":165,"context_line":"While I really want someone to explore building a driver like this,"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9a8ffd7b_2d531f3b","line":162,"range":{"start_line":162,"start_character":59,"end_line":162,"end_character":63},"updated":"2015-12-01 17:33:35.000000000","message":"s/anit/anti/","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"84e1d4d39896453f091e48e753b3cc40bd3959ce","unresolved":false,"context_lines":[{"line_number":164,"context_line":""},{"line_number":165,"context_line":"While I really want someone to explore building a driver like this,"},{"line_number":166,"context_line":"this spec is not considering this approach. It will instead focus on a more"},{"line_number":167,"context_line":"direct replacement for the current filter scheduler."},{"line_number":168,"context_line":""},{"line_number":169,"context_line":"Partitioning the Filter Scheduler"},{"line_number":170,"context_line":"++++++++++++++++++++++++++++++++++"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_a081a4c5","line":167,"updated":"2015-11-12 21:51:37.000000000","message":"There are also non-linear schedulers, such as the solver scheduler: https://www.youtube.com/watch?v\u003dFsw-hjtQ8e4","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"00cb75f3e07ae236abedff0ebc62e3e487d35c32","unresolved":false,"context_lines":[{"line_number":170,"context_line":"++++++++++++++++++++++++++++++++++"},{"line_number":171,"context_line":""},{"line_number":172,"context_line":"A great way to reduce the size of a problem, is to split the larger"},{"line_number":173,"context_line":"problem into smaller pieces. Lets look at this in more detail."},{"line_number":174,"context_line":""},{"line_number":175,"context_line":"One major issue is interference between different schedulers. Ideally"},{"line_number":176,"context_line":"we don\u0027t want multiple schedulers assigning work to the same nova-compute"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_04b13f47","line":173,"updated":"2015-11-10 12:23:34.000000000","message":"++ This is my top priority for the future scheduler design. \"Divide et impera\"","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"ac6a5463946ef1208b2129e9ae2b72483da50be5","unresolved":false,"context_lines":[{"line_number":170,"context_line":"++++++++++++++++++++++++++++++++++"},{"line_number":171,"context_line":""},{"line_number":172,"context_line":"A great way to reduce the size of a problem, is to split the larger"},{"line_number":173,"context_line":"problem into smaller pieces. Lets look at this in more detail."},{"line_number":174,"context_line":""},{"line_number":175,"context_line":"One major issue is interference between different schedulers. Ideally"},{"line_number":176,"context_line":"we don\u0027t want multiple schedulers assigning work to the same nova-compute"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_33136668","line":173,"in_reply_to":"da85f559_04b13f47","updated":"2015-11-10 14:19:55.000000000","message":"I\u0027d rather see a list of scheduler drivers with each one answering a problem. I\u0027m not a big fan of a divide-and-conquer scheduler like Mesos does for many reasons - in particular because our workloads aren\u0027t that specialized so that it would be underperformant - but I can understand that *some* deployments could have that and accordinly have this kind of scheduler driver, but IMHO it shouldn\u0027t be the default config","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"00cb75f3e07ae236abedff0ebc62e3e487d35c32","unresolved":false,"context_lines":[{"line_number":175,"context_line":"One major issue is interference between different schedulers. Ideally"},{"line_number":176,"context_line":"we don\u0027t want multiple schedulers assigning work to the same nova-compute"},{"line_number":177,"context_line":"nodes, as they will be competing with each other for the same resources."},{"line_number":178,"context_line":"Ideally each scheduler would be looking a different subset of hosts."},{"line_number":179,"context_line":""},{"line_number":180,"context_line":"Fighting this requirement are cluster wide behaviors, such as affinity and"},{"line_number":181,"context_line":"anti-affinity rules, where ideally we need to know the full state of the"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_04985fbf","line":178,"updated":"2015-11-10 12:23:34.000000000","message":"Right.","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"00cb75f3e07ae236abedff0ebc62e3e487d35c32","unresolved":false,"context_lines":[{"line_number":179,"context_line":""},{"line_number":180,"context_line":"Fighting this requirement are cluster wide behaviors, such as affinity and"},{"line_number":181,"context_line":"anti-affinity rules, where ideally we need to know the full state of the"},{"line_number":182,"context_line":"system, rather than just looking at a subset of the system."},{"line_number":183,"context_line":""},{"line_number":184,"context_line":"Its possible to have a dynamic partitioning, but for simplicity, I am going"},{"line_number":185,"context_line":"to focus on static partitions of the system."}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_c46d7799","line":182,"updated":"2015-11-10 12:23:34.000000000","message":"Yes, but affinity and anti-affinity policies can be satisfied using a partitioned space algorithm. For instance, if the affinity policy is \"close to rack A\", then a scheduler processes that handle the partitions of the deployment that contains racks B through E would necessarily not be involved in the decision making...","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"84e1d4d39896453f091e48e753b3cc40bd3959ce","unresolved":false,"context_lines":[{"line_number":179,"context_line":""},{"line_number":180,"context_line":"Fighting this requirement are cluster wide behaviors, such as affinity and"},{"line_number":181,"context_line":"anti-affinity rules, where ideally we need to know the full state of the"},{"line_number":182,"context_line":"system, rather than just looking at a subset of the system."},{"line_number":183,"context_line":""},{"line_number":184,"context_line":"Its possible to have a dynamic partitioning, but for simplicity, I am going"},{"line_number":185,"context_line":"to focus on static partitions of the system."}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_c0a2a84b","line":182,"in_reply_to":"da85f559_c46d7799","updated":"2015-11-12 21:51:37.000000000","message":"This would imply the need for some pre-scheduling scheduler, which would route the request to the scheduler with the necessary knowledge. And what if the dividing line for the schedulers was near the request? IOW, a request for \"near rack L\", but the schedulers are split A-L and M-Z.","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"00cb75f3e07ae236abedff0ebc62e3e487d35c32","unresolved":false,"context_lines":[{"line_number":186,"context_line":"The problem with a static partition is that they tend to have capacity"},{"line_number":187,"context_line":"planning implications. If a subset of all requests get routed to a particular"},{"line_number":188,"context_line":"set of hosts, then you need to ensure you increase the number of hosts to"},{"line_number":189,"context_line":"match the demand for that subset of hosts."},{"line_number":190,"context_line":""},{"line_number":191,"context_line":"In cells v1, the top level scheduling was used to try and spread the load"},{"line_number":192,"context_line":"between lots of groups that get added as you expand, but this two level"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_44274764","line":189,"updated":"2015-11-10 12:23:34.000000000","message":"Right... the typical partition or hash ring function problem.","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"ac6a5463946ef1208b2129e9ae2b72483da50be5","unresolved":false,"context_lines":[{"line_number":186,"context_line":"The problem with a static partition is that they tend to have capacity"},{"line_number":187,"context_line":"planning implications. If a subset of all requests get routed to a particular"},{"line_number":188,"context_line":"set of hosts, then you need to ensure you increase the number of hosts to"},{"line_number":189,"context_line":"match the demand for that subset of hosts."},{"line_number":190,"context_line":""},{"line_number":191,"context_line":"In cells v1, the top level scheduling was used to try and spread the load"},{"line_number":192,"context_line":"between lots of groups that get added as you expand, but this two level"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_f642ac3d","line":189,"in_reply_to":"da85f559_44274764","updated":"2015-11-10 14:19:55.000000000","message":"The main problem of static partitioning is that you always subperform if you have pretty identical workloads like I said above. I can point you out some interesting papers on that, hence me not a big fan of that driver as the default one for the OpenStack vanilla environment","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"00cb75f3e07ae236abedff0ebc62e3e487d35c32","unresolved":false,"context_lines":[{"line_number":197,"context_line":""},{"line_number":198,"context_line":"* Certain groups of hosts can have specific hardware mapped to"},{"line_number":199,"context_line":"  specific flavors."},{"line_number":200,"context_line":"  i.e. SSD vs non-SSD local storage vs all storage form cinder (no local disk)"},{"line_number":201,"context_line":""},{"line_number":202,"context_line":"* Keeping Windows and Linux VMs on different sets of hypervisors is common"},{"line_number":203,"context_line":"  place, to allow for the best utilization of bulk license savings."}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_04a5bfa0","line":200,"updated":"2015-11-10 12:23:34.000000000","message":"Yes, but... let\u0027s make sure any partitioning scheme uses the *components* of a flavor -- i.e. its quantitative parts like RAM/CPU/Disk and its qualitative parts like whether it has SSD local drives or particular CPU models -- instead of the clunky flavor object as a whole.","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"00cb75f3e07ae236abedff0ebc62e3e487d35c32","unresolved":false,"context_lines":[{"line_number":214,"context_line":"In a cells v2 world, you would have multiple cells in each group. For"},{"line_number":215,"context_line":"simplicity we can assume each complete cell would be registered to one"},{"line_number":216,"context_line":"(and only one) of the scheduler clusters. In practice, we probably want each"},{"line_number":217,"context_line":"host to know what scheduler it should report things two."},{"line_number":218,"context_line":""},{"line_number":219,"context_line":"The nice property of this partition is that you need to do capacity planning"},{"line_number":220,"context_line":"for each of these groups of hosts independently, regardless of how the"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_04d37f05","line":217,"updated":"2015-11-10 12:23:34.000000000","message":"s/things two/things to/","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"00cb75f3e07ae236abedff0ebc62e3e487d35c32","unresolved":false,"context_lines":[{"line_number":222,"context_line":""},{"line_number":223,"context_line":"There are many other possible partitions, but this seems one of the simplest"},{"line_number":224,"context_line":"and well help many of the large cloud users moving from cells v1 to cells v2."},{"line_number":225,"context_line":"Lets consider another partition, such as using hash of a tenant to"},{"line_number":226,"context_line":"choose between some distinct subset of hosts. Here you need to have a very"},{"line_number":227,"context_line":"large number of tenants and/or even usage across your tenants, otherwise"},{"line_number":228,"context_line":"you end up having to expand capacity differently across each of the groups"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_04fa9f75","line":225,"updated":"2015-11-10 12:23:34.000000000","message":"s/Lets/Let\u0027s/","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"00cb75f3e07ae236abedff0ebc62e3e487d35c32","unresolved":false,"context_lines":[{"line_number":228,"context_line":"you end up having to expand capacity differently across each of the groups"},{"line_number":229,"context_line":"as the demand from those different tenants goes up and down."},{"line_number":230,"context_line":"When each of those schedulers look at overlapping subsets of the nodes, you"},{"line_number":231,"context_line":"improve the spread of resources, but you tent to end up with some interference"},{"line_number":232,"context_line":"between the different scheduler clusters."},{"line_number":233,"context_line":""},{"line_number":234,"context_line":"While some of these alternative partitioning schemes may well be useful once"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_e4ee7bb3","line":231,"updated":"2015-11-10 12:23:34.000000000","message":"s/tent/tend/","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"00cb75f3e07ae236abedff0ebc62e3e487d35c32","unresolved":false,"context_lines":[{"line_number":234,"context_line":"While some of these alternative partitioning schemes may well be useful once"},{"line_number":235,"context_line":"we have some of the other enhancements discussed here, I am limiting the scope"},{"line_number":236,"context_line":"of this spec to the simplest partitioning scheme, a distinct partitioning of"},{"line_number":237,"context_line":"hosts based on the requested flavor, for the initial version."},{"line_number":238,"context_line":"The major downside of this approach is it limits the impact of partitioning to"},{"line_number":239,"context_line":"the very largest cloud deployments, those where there are several distinct"},{"line_number":240,"context_line":"groups of hosts that have their capacity managed separately."}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_e42bdb92","line":237,"updated":"2015-11-10 12:23:34.000000000","message":"as mentioned above, I\u0027d greatly prefer to *not* use the flavor object itself, but rather its components. My vision is to decompose the flavor object *immediately* upon hitting the API service and only pass the component parts of the flavor on to other parts of the system after that.\n\nSo, instead of saying \"this partition services requests for m1.xlarge flavors\", you would instead be able to say \"this partition services requests for instances that have between 16GB and 64GB of RAM and less than 8 vCPU\" or \"this partition services requests for any instance that requires SSD local drives\"","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"84e1d4d39896453f091e48e753b3cc40bd3959ce","unresolved":false,"context_lines":[{"line_number":234,"context_line":"While some of these alternative partitioning schemes may well be useful once"},{"line_number":235,"context_line":"we have some of the other enhancements discussed here, I am limiting the scope"},{"line_number":236,"context_line":"of this spec to the simplest partitioning scheme, a distinct partitioning of"},{"line_number":237,"context_line":"hosts based on the requested flavor, for the initial version."},{"line_number":238,"context_line":"The major downside of this approach is it limits the impact of partitioning to"},{"line_number":239,"context_line":"the very largest cloud deployments, those where there are several distinct"},{"line_number":240,"context_line":"groups of hosts that have their capacity managed separately."}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_a0420467","line":237,"in_reply_to":"da85f559_e42bdb92","updated":"2015-11-12 21:51:37.000000000","message":"+1","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"00cb75f3e07ae236abedff0ebc62e3e487d35c32","unresolved":false,"context_lines":[{"line_number":237,"context_line":"hosts based on the requested flavor, for the initial version."},{"line_number":238,"context_line":"The major downside of this approach is it limits the impact of partitioning to"},{"line_number":239,"context_line":"the very largest cloud deployments, those where there are several distinct"},{"line_number":240,"context_line":"groups of hosts that have their capacity managed separately."},{"line_number":241,"context_line":""},{"line_number":242,"context_line":"Using the Resource Tracker to implement \"distributed\" locking"},{"line_number":243,"context_line":"++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_a703950c","line":240,"updated":"2015-11-10 12:23:34.000000000","message":"I don\u0027t necessarily see this as being limited to only the largest cloud deployers. You could use this scheme for clouds of even 50-100 nodes in size pretty effectively.","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"ac6a5463946ef1208b2129e9ae2b72483da50be5","unresolved":false,"context_lines":[{"line_number":237,"context_line":"hosts based on the requested flavor, for the initial version."},{"line_number":238,"context_line":"The major downside of this approach is it limits the impact of partitioning to"},{"line_number":239,"context_line":"the very largest cloud deployments, those where there are several distinct"},{"line_number":240,"context_line":"groups of hosts that have their capacity managed separately."},{"line_number":241,"context_line":""},{"line_number":242,"context_line":"Using the Resource Tracker to implement \"distributed\" locking"},{"line_number":243,"context_line":"++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_56861814","line":240,"in_reply_to":"da85f559_a703950c","updated":"2015-11-10 14:19:55.000000000","message":"No, again, it\u0027s an underperformance unless you have very different workload types - proof eg. \n- http://ijcsse.org/published/volume4/issue6/p2-V4I6.pdf\n- http://eurosys2013.tudos.org/wp-content/uploads/2013/paper/Schwarzkopf.pdf\n- http://www.umbrant.com/blog/2015/mesos_omega_borg_survey.html\n\nSplitting your cloud in chunks is understandable for very specific deployments but the decision (give me an host) always takes the same whatever the future user workload is. \nSo, I consider that scheduler partitioning is maybe understandable when you want to route a request to a specific subset because you\u0027re 100% sure that your chunks are mutually exclusive, but again, that\u0027s very specific and should only be seen as a non-default scheduler driver.","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"00cb75f3e07ae236abedff0ebc62e3e487d35c32","unresolved":false,"context_lines":[{"line_number":254,"context_line":"get a leak of capacity due to un-used resource tracker claims."},{"line_number":255,"context_line":"This moves what could be a distributed locking mechanism to a per nova-compute"},{"line_number":256,"context_line":"locking system, that should mean there is much less lock contention, and"},{"line_number":257,"context_line":"generally its a much easier problem to solve."},{"line_number":258,"context_line":""},{"line_number":259,"context_line":"When the resource tracker reports its current available resources up to the"},{"line_number":260,"context_line":"scheduler it would reduce the amount of free resources to take account of the"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_87a211ad","line":257,"updated":"2015-11-10 12:23:34.000000000","message":"Erm, I don\u0027t quite follow how persisting claims on the compute nodes will lead to a situation where there is less lock contention... if the scheduler does *not* have a non-stale view of the world (i.e. if we continue with the current state of the system where the scheduler does not own the resources in its partition) then the scheduler will continue to suffer from the extended interval that currently exists between when the scheduler makes a decision (on stale data) and the destination compute node ends up rejecting the launch request because another scheduler beat the first to the punch.","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"84e1d4d39896453f091e48e753b3cc40bd3959ce","unresolved":false,"context_lines":[{"line_number":254,"context_line":"get a leak of capacity due to un-used resource tracker claims."},{"line_number":255,"context_line":"This moves what could be a distributed locking mechanism to a per nova-compute"},{"line_number":256,"context_line":"locking system, that should mean there is much less lock contention, and"},{"line_number":257,"context_line":"generally its a much easier problem to solve."},{"line_number":258,"context_line":""},{"line_number":259,"context_line":"When the resource tracker reports its current available resources up to the"},{"line_number":260,"context_line":"scheduler it would reduce the amount of free resources to take account of the"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_cb7169aa","line":257,"in_reply_to":"da85f559_794ead70","updated":"2015-11-12 21:51:37.000000000","message":"It sounds to me like there are two notions of optimizations. One is pick something fast and fail fast, retrying as needed. The other is pick something with a much higher confidence that it is going to succeed, and only retry in those rare cases of failure.\n\nUntil we choose one of those directions, all these proposed improvements are going to conflict with each other.","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"ac6a5463946ef1208b2129e9ae2b72483da50be5","unresolved":false,"context_lines":[{"line_number":254,"context_line":"get a leak of capacity due to un-used resource tracker claims."},{"line_number":255,"context_line":"This moves what could be a distributed locking mechanism to a per nova-compute"},{"line_number":256,"context_line":"locking system, that should mean there is much less lock contention, and"},{"line_number":257,"context_line":"generally its a much easier problem to solve."},{"line_number":258,"context_line":""},{"line_number":259,"context_line":"When the resource tracker reports its current available resources up to the"},{"line_number":260,"context_line":"scheduler it would reduce the amount of free resources to take account of the"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_794ead70","line":257,"in_reply_to":"da85f559_87a211ad","updated":"2015-11-10 14:19:55.000000000","message":"So, I feel that we suffer from a situation where the scheduler could be far better having a non-stale view of the world, but we should still tolerate that the compute node owns the final decision. If not, it would mean that by any extend, the scheduler should wait for the compute node to acknowledge the decision, which would be an active lock - which is totally what I want to avoid because we want a *fast* scheduler - even if it does mistakes.","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"00cb75f3e07ae236abedff0ebc62e3e487d35c32","unresolved":false,"context_lines":[{"line_number":263,"context_line":"Now consider if the scheduler was able to acquire one of these claims before"},{"line_number":264,"context_line":"returning the chosen host to the nova-conductor. This would be moving the"},{"line_number":265,"context_line":"claim request from the very start of the build process in nova-compute into"},{"line_number":266,"context_line":"the scheduler."},{"line_number":267,"context_line":"This would allow the scheduler to build up a collection of claims for the"},{"line_number":268,"context_line":"requested resources before returning the choice to the caller what resources"},{"line_number":269,"context_line":"the scheduler has chosen. Should there be a problem detected, the scheduler"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_07cce168","line":266,"updated":"2015-11-10 12:23:34.000000000","message":"? OK, so here you are saying that the scheduler is the thing that owns the resources... which I agree with. But what precisely do you mean by \"acquire one of these claims\" above? If you are recommending that before finalizing the placement decision the scheduler should contact the compute host to \"get a claim\" then I\u0027m afraid that will introduce too much unnecessary latency into the system. The scheduler should just be able to \"create a claim on resources\" by updating a single record in a data store without needing to contact the target host at all.","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"84e1d4d39896453f091e48e753b3cc40bd3959ce","unresolved":false,"context_lines":[{"line_number":263,"context_line":"Now consider if the scheduler was able to acquire one of these claims before"},{"line_number":264,"context_line":"returning the chosen host to the nova-conductor. This would be moving the"},{"line_number":265,"context_line":"claim request from the very start of the build process in nova-compute into"},{"line_number":266,"context_line":"the scheduler."},{"line_number":267,"context_line":"This would allow the scheduler to build up a collection of claims for the"},{"line_number":268,"context_line":"requested resources before returning the choice to the caller what resources"},{"line_number":269,"context_line":"the scheduler has chosen. Should there be a problem detected, the scheduler"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_6bdcfd74","line":266,"in_reply_to":"da85f559_07cce168","updated":"2015-11-12 21:51:37.000000000","message":"I prefer Jay\u0027s approach, but that is not the current design of the interaction among the various moving parts of resource scheduling. Right now, the resource provider owns the resource. That\u0027s backwards, IMO, but until we agree that this needs to be changed, we have to work with that design.","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"ac6a5463946ef1208b2129e9ae2b72483da50be5","unresolved":false,"context_lines":[{"line_number":263,"context_line":"Now consider if the scheduler was able to acquire one of these claims before"},{"line_number":264,"context_line":"returning the chosen host to the nova-conductor. This would be moving the"},{"line_number":265,"context_line":"claim request from the very start of the build process in nova-compute into"},{"line_number":266,"context_line":"the scheduler."},{"line_number":267,"context_line":"This would allow the scheduler to build up a collection of claims for the"},{"line_number":268,"context_line":"requested resources before returning the choice to the caller what resources"},{"line_number":269,"context_line":"the scheduler has chosen. Should there be a problem detected, the scheduler"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_39dee53f","line":266,"in_reply_to":"da85f559_07cce168","updated":"2015-11-10 14:19:55.000000000","message":"So, there are 2 points here: \n- which component is the single source of knowledge for accuracy \u003d\u003e from my perspecitive of a fast scheduler, that\u0027s still necessarly the compute node which gives the final word\n- the scheduler can still have an internal view of the world which can be modified by a transaction (that would be scheduler claim) when 2 concurrent scheduling requests would try to insert at the same time the resource usage - but that\u0027s only a scheduler internal process IMHO","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"ac6a5463946ef1208b2129e9ae2b72483da50be5","unresolved":false,"context_lines":[{"line_number":272,"context_line":""},{"line_number":273,"context_line":"Putting this all together, you now see that the schedulers will start to see"},{"line_number":274,"context_line":"each others decisions because the claims acquired by another scheduler show up"},{"line_number":275,"context_line":"more quickly in the shared state."},{"line_number":276,"context_line":""},{"line_number":277,"context_line":"Taking this a step further we could ensure that a scheduler waits for the"},{"line_number":278,"context_line":"claim it just took to show up in the shared state before returning the"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_d99b1971","line":275,"range":{"start_line":275,"start_character":20,"end_line":275,"end_character":32},"updated":"2015-11-10 14:19:55.000000000","message":"that means that distributed schedulers should have a shared state - and for that I agree","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"84e1d4d39896453f091e48e753b3cc40bd3959ce","unresolved":false,"context_lines":[{"line_number":272,"context_line":""},{"line_number":273,"context_line":"Putting this all together, you now see that the schedulers will start to see"},{"line_number":274,"context_line":"each others decisions because the claims acquired by another scheduler show up"},{"line_number":275,"context_line":"more quickly in the shared state."},{"line_number":276,"context_line":""},{"line_number":277,"context_line":"Taking this a step further we could ensure that a scheduler waits for the"},{"line_number":278,"context_line":"claim it just took to show up in the shared state before returning the"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_eb354da8","line":275,"range":{"start_line":275,"start_character":20,"end_line":275,"end_character":32},"in_reply_to":"da85f559_d99b1971","updated":"2015-11-12 21:51:37.000000000","message":"Oooh - we\u0027re getting close to distributed data stores... :)","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"00cb75f3e07ae236abedff0ebc62e3e487d35c32","unresolved":false,"context_lines":[{"line_number":286,"context_line":"internal state to see if this is still the best node to send the request."},{"line_number":287,"context_line":"It could be done by having a hash of the currently reported node state,"},{"line_number":288,"context_line":"and comparing that. Its assumed such a hash would not change when an instance"},{"line_number":289,"context_line":"goes from the claimed state to a state where it is using that claim."},{"line_number":290,"context_line":""},{"line_number":291,"context_line":"It seems likely that a combination of these strategies should help ensure the"},{"line_number":292,"context_line":"scheduler is able to deal with most races between other parallel schedulers"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_a7cb951d","line":289,"updated":"2015-11-10 12:23:34.000000000","message":"Yes, this!\n\nIn fact, this is the design I had proposed way back at the beginning of last year:\n\nhttps://review.openstack.org/#/c/103598/4/nova/placement/__init__.py\n\nhttps://review.openstack.org/#/c/103598/4/nova/placement/resource_tracker.py\n\nNote the second link is actually a resource tracker *in the scheduler*, not on the compute node. See the \"compare_and_update_resources()\" method there.","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"ac6a5463946ef1208b2129e9ae2b72483da50be5","unresolved":false,"context_lines":[{"line_number":286,"context_line":"internal state to see if this is still the best node to send the request."},{"line_number":287,"context_line":"It could be done by having a hash of the currently reported node state,"},{"line_number":288,"context_line":"and comparing that. Its assumed such a hash would not change when an instance"},{"line_number":289,"context_line":"goes from the claimed state to a state where it is using that claim."},{"line_number":290,"context_line":""},{"line_number":291,"context_line":"It seems likely that a combination of these strategies should help ensure the"},{"line_number":292,"context_line":"scheduler is able to deal with most races between other parallel schedulers"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_79b78dd8","line":289,"in_reply_to":"da85f559_a7cb951d","updated":"2015-11-10 14:19:55.000000000","message":"I\u0027m not opposed to that, but that requires a shared HostManager for all concurrent schedulers IMHO","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"84e1d4d39896453f091e48e753b3cc40bd3959ce","unresolved":false,"context_lines":[{"line_number":286,"context_line":"internal state to see if this is still the best node to send the request."},{"line_number":287,"context_line":"It could be done by having a hash of the currently reported node state,"},{"line_number":288,"context_line":"and comparing that. Its assumed such a hash would not change when an instance"},{"line_number":289,"context_line":"goes from the claimed state to a state where it is using that claim."},{"line_number":290,"context_line":""},{"line_number":291,"context_line":"It seems likely that a combination of these strategies should help ensure the"},{"line_number":292,"context_line":"scheduler is able to deal with most races between other parallel schedulers"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_cb296939","line":289,"in_reply_to":"da85f559_a7cb951d","updated":"2015-11-12 21:51:37.000000000","message":"This is the same approach philosophically as lightweight transactions in Cassandra. It makes sense to have such a sanity check in an async environment.","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"00cb75f3e07ae236abedff0ebc62e3e487d35c32","unresolved":false,"context_lines":[{"line_number":297,"context_line":"+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++"},{"line_number":298,"context_line":""},{"line_number":299,"context_line":"As mentioned above, the most expensive part of the scheduling process is not"},{"line_number":300,"context_line":"running through the list of filters and weights, it is getting updating the"},{"line_number":301,"context_line":"current host state from the database."},{"line_number":302,"context_line":""},{"line_number":303,"context_line":"We currently use the Caching Scheduler to reduce the cost of these DB calls,"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_47e9897f","line":300,"updated":"2015-11-10 12:23:34.000000000","message":"s/getting updating/updating/","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"00cb75f3e07ae236abedff0ebc62e3e487d35c32","unresolved":false,"context_lines":[{"line_number":323,"context_line":"The pain point of friction with the no-db-scheduler was the complexity of"},{"line_number":324,"context_line":"maintaining the code that look a lot like the implementation of a DB log."},{"line_number":325,"context_line":"Being able to efficiently trim old updates, so any new schedulers have only"},{"line_number":326,"context_line":"have a small amount of data to catch up."},{"line_number":327,"context_line":"It turns our Kafka has already implemented at lot of these semantics and is"},{"line_number":328,"context_line":"has already been proven to work at an extremely large scale:"},{"line_number":329,"context_line":"http://kafka.apache.org/documentation.html#introduction"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_47b74986","line":326,"updated":"2015-11-10 12:23:34.000000000","message":"s/have a small amount/a small amount/","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"84e1d4d39896453f091e48e753b3cc40bd3959ce","unresolved":false,"context_lines":[{"line_number":331,"context_line":"It seems we should be able to create a kafka based system to get efficient"},{"line_number":332,"context_line":"incremental updates to the current state of the system, rather than having to"},{"line_number":333,"context_line":"make the expensive DB call to get the state for all the hosts we are"},{"line_number":334,"context_line":"interested in."},{"line_number":335,"context_line":""},{"line_number":336,"context_line":"Memory concerns"},{"line_number":337,"context_line":"++++++++++++++++"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_eb86ad15","line":334,"updated":"2015-11-12 21:51:37.000000000","message":"Kafka does indeed look promising, and it would be very informative to be able to put together a POC system to see if it does fit well.","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"84e1d4d39896453f091e48e753b3cc40bd3959ce","unresolved":false,"context_lines":[{"line_number":340,"context_line":"of all the hosts in the system, and their current state."},{"line_number":341,"context_line":""},{"line_number":342,"context_line":"It seems that, in practice, this will be the least of our worries when it"},{"line_number":343,"context_line":"comes to finding what limits the level of scale this solution can reach."},{"line_number":344,"context_line":""},{"line_number":345,"context_line":"Data model impact"},{"line_number":346,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_0b08d190","line":343,"updated":"2015-11-12 21:51:37.000000000","message":"We can throw hardware at the problem by maxing out the memory for the scheduler machines, so that really isn\u0027t an issue in any serious deployment. But the critical thing, IMO, is how you act on that in-memory data. DBs have the advantage that they have been designed to quickly find the thing you want, so a good choice might just be an in-memory database?","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"00cb75f3e07ae236abedff0ebc62e3e487d35c32","unresolved":false,"context_lines":[{"line_number":375,"context_line":"Other deployer impact"},{"line_number":376,"context_line":"---------------------"},{"line_number":377,"context_line":""},{"line_number":378,"context_line":"Any solution will need a way to live upgrade from the existing scheduler."},{"line_number":379,"context_line":""},{"line_number":380,"context_line":"Developer impact"},{"line_number":381,"context_line":"----------------"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_473c29e2","line":378,"updated":"2015-11-10 12:23:34.000000000","message":"Why? All that really needs to happen is the old scheduler processes need to complete all their in-flight requests and then just swap in the new scheduler processes.","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"ac6a5463946ef1208b2129e9ae2b72483da50be5","unresolved":false,"context_lines":[{"line_number":375,"context_line":"Other deployer impact"},{"line_number":376,"context_line":"---------------------"},{"line_number":377,"context_line":""},{"line_number":378,"context_line":"Any solution will need a way to live upgrade from the existing scheduler."},{"line_number":379,"context_line":""},{"line_number":380,"context_line":"Developer impact"},{"line_number":381,"context_line":"----------------"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_d9d799eb","line":378,"in_reply_to":"da85f559_473c29e2","updated":"2015-11-10 14:19:55.000000000","message":"I feel we should rather propose a collection of scheduler drivers, each one proposing to solve a very specific implementation","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"84e1d4d39896453f091e48e753b3cc40bd3959ce","unresolved":false,"context_lines":[{"line_number":375,"context_line":"Other deployer impact"},{"line_number":376,"context_line":"---------------------"},{"line_number":377,"context_line":""},{"line_number":378,"context_line":"Any solution will need a way to live upgrade from the existing scheduler."},{"line_number":379,"context_line":""},{"line_number":380,"context_line":"Developer impact"},{"line_number":381,"context_line":"----------------"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_2b76f506","line":378,"in_reply_to":"da85f559_d9d799eb","updated":"2015-11-12 21:51:37.000000000","message":"Jay - that would only be true if there were no additional changes to the data flow. If the schedulers relied on updated resource trackers, or new technology like Kafka, then a lot of thought and testing needs to be done before these changes can be released.","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"84e1d4d39896453f091e48e753b3cc40bd3959ce","unresolved":false,"context_lines":[{"line_number":380,"context_line":"Developer impact"},{"line_number":381,"context_line":"----------------"},{"line_number":382,"context_line":""},{"line_number":383,"context_line":"None"},{"line_number":384,"context_line":""},{"line_number":385,"context_line":"Implementation"},{"line_number":386,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_6b539d6d","line":383,"updated":"2015-11-12 21:51:37.000000000","message":"True, unless you develop parts of Nova that involve scheduling requests.","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"ac6a5463946ef1208b2129e9ae2b72483da50be5","unresolved":false,"context_lines":[{"line_number":423,"context_line":"References"},{"line_number":424,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":425,"context_line":""},{"line_number":426,"context_line":"Google omega paper: http://research.google.com/pubs/pub41684.html"},{"line_number":427,"context_line":""},{"line_number":428,"context_line":"History"},{"line_number":429,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":6,"id":"da85f559_1623b006","line":426,"updated":"2015-11-10 14:19:55.000000000","message":"I\u0027d add some other good papers like the ones I mentioned above :\n- http://ijcsse.org/published/volume4/issue6/p2-V4I6.pdf\n- http://www.umbrant.com/blog/2015/mesos_omega_borg_survey.html\n\nIt makes you understand that there is no one-size-fits-all solution but it also allows you to forge an opinion about what should be the default scheduler for clouds.","commit_id":"76b49813922d320fbbf6aafab8a0cd7ed0655837"}]}