)]}'
{"specs/convergence-engine.rst":[{"author":{"_account_id":4328,"name":"Steven Hardy","email":"steven.hardy@suse.com","username":"shardy"},"change_message_id":"469f4f5b2f4ad7ed1c4eeadea8f958d794280cf3","unresolved":false,"context_lines":[{"line_number":12,"context_line":""},{"line_number":13,"context_line":"In order to scale out and also to enable the convergence model, tasks"},{"line_number":14,"context_line":"normally done inside a single engine with a stack-wide lock will need"},{"line_number":15,"context_line":"to be performed in isolation. This process will be the process which"},{"line_number":16,"context_line":"performs those tasks."},{"line_number":17,"context_line":""},{"line_number":18,"context_line":"Problem description"}],"source_content_type":"text/x-rst","patch_set":1,"id":"1ae5cdf2_26142543","line":15,"updated":"2014-06-03 09:54:49.000000000","message":"\"This process will be the process\" reads awkwardly.  It means \"The convergence engine will be the process\"?\n\nIMO we really need a diagram linked here, as it\u0027s fairly hard to visualize the architecture being described.","commit_id":"23464e2d9405ec5c591764db57e4c760fcfcef08"},{"author":{"_account_id":6488,"name":"Clint Byrum","email":"clint@fewbar.com","username":"clint-fewbar"},"change_message_id":"c20ca4f9303e6a13c56a7fcc0a2459fc2a26ca59","unresolved":false,"context_lines":[{"line_number":12,"context_line":""},{"line_number":13,"context_line":"In order to scale out and also to enable the convergence model, tasks"},{"line_number":14,"context_line":"normally done inside a single engine with a stack-wide lock will need"},{"line_number":15,"context_line":"to be performed in isolation. This process will be the process which"},{"line_number":16,"context_line":"performs those tasks."},{"line_number":17,"context_line":""},{"line_number":18,"context_line":"Problem description"}],"source_content_type":"text/x-rst","patch_set":1,"id":"1ae5cdf2_0a5e3e76","line":15,"in_reply_to":"1ae5cdf2_26142543","updated":"2014-06-03 16:58:21.000000000","message":"Good idea. I\u0027ll work up a diagram with before/after highlighted.","commit_id":"23464e2d9405ec5c591764db57e4c760fcfcef08"},{"author":{"_account_id":4328,"name":"Steven Hardy","email":"steven.hardy@suse.com","username":"shardy"},"change_message_id":"469f4f5b2f4ad7ed1c4eeadea8f958d794280cf3","unresolved":false,"context_lines":[{"line_number":30,"context_line":"* Create a new process named heat-convergence-engine"},{"line_number":31,"context_line":""},{"line_number":32,"context_line":"    * New process will only act on resources one at a time, no stack-wide"},{"line_number":33,"context_line":"      operations will be done."},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"* Create a new \"job queue\" style RPC for performing convergence operations"},{"line_number":36,"context_line":"  on resources."}],"source_content_type":"text/x-rst","patch_set":1,"id":"1ae5cdf2_46abf1d5","line":33,"updated":"2014-06-03 09:54:49.000000000","message":"So if the process works on only one resource, this implies either a pool of $many processes, or the ability to dynamically spawn more worker processes.  \n\nHow are we planning to do that?","commit_id":"23464e2d9405ec5c591764db57e4c760fcfcef08"},{"author":{"_account_id":6488,"name":"Clint Byrum","email":"clint@fewbar.com","username":"clint-fewbar"},"change_message_id":"c20ca4f9303e6a13c56a7fcc0a2459fc2a26ca59","unresolved":false,"context_lines":[{"line_number":30,"context_line":"* Create a new process named heat-convergence-engine"},{"line_number":31,"context_line":""},{"line_number":32,"context_line":"    * New process will only act on resources one at a time, no stack-wide"},{"line_number":33,"context_line":"      operations will be done."},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"* Create a new \"job queue\" style RPC for performing convergence operations"},{"line_number":36,"context_line":"  on resources."}],"source_content_type":"text/x-rst","patch_set":1,"id":"1ae5cdf2_8a496eb9","line":33,"in_reply_to":"1ae5cdf2_46abf1d5","updated":"2014-06-03 16:58:21.000000000","message":"You only need one process, it will just do things one after the other. It\u0027s a bit awkward to send an RPC message to yourself, but it should work fine.","commit_id":"23464e2d9405ec5c591764db57e4c760fcfcef08"},{"author":{"_account_id":8246,"name":"Qiming Teng","email":"tengqm@outlook.com","username":"tengqm"},"change_message_id":"51a1866b2db887168d9f64720855eb32ec2b49db","unresolved":false,"context_lines":[{"line_number":30,"context_line":"* Create a new process named heat-convergence-engine"},{"line_number":31,"context_line":""},{"line_number":32,"context_line":"    * New process will only act on resources one at a time, no stack-wide"},{"line_number":33,"context_line":"      operations will be done."},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"* Create a new \"job queue\" style RPC for performing convergence operations"},{"line_number":36,"context_line":"  on resources."}],"source_content_type":"text/x-rst","patch_set":1,"id":"1ae5cdf2_b82e458a","line":33,"in_reply_to":"1ae5cdf2_8a496eb9","updated":"2014-06-05 03:25:46.000000000","message":"Need some education on how scalability issue is solved here.  Is this design meant to be a better implementation of serialized processing?\n\nNote that a stack can be a resource itself.  Any special consideration paid on nested stacks?","commit_id":"23464e2d9405ec5c591764db57e4c760fcfcef08"},{"author":{"_account_id":4715,"name":"Angus Salkeld","email":"asalkeld@redhat.com","username":"asalkeld"},"change_message_id":"7ae86cf7037a9ddec9e352b2e1a0101694f6f243","unresolved":false,"context_lines":[{"line_number":40,"context_line":""},{"line_number":41,"context_line":"* Migrate existing resources which do require refactoring (because of"},{"line_number":42,"context_line":"  use of extensive stack details)"},{"line_number":43,"context_line":""},{"line_number":44,"context_line":"Alternatives"},{"line_number":45,"context_line":"------------"},{"line_number":46,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"1ae5cdf2_0a52da12","line":43,"updated":"2014-06-03 00:55:26.000000000","message":"some thoughts on the plugin api would be good. and an answer to \"why can\u0027t we use the current plugin api\". - Not suggesting that necessarily, but I think we need a compelling argument for changing the plugin api. I even go so far as saying that it\u0027s worth looking at trying really hard to not have to change that api. I worry that we are piling too much logic onto the  new resource and it would be good if they could be simply implemented. Maybe a discussion on where the logic for repair goes? I\u0027d hope in a layer above the resource plugin (to keep the plugin super simple).","commit_id":"23464e2d9405ec5c591764db57e4c760fcfcef08"},{"author":{"_account_id":4328,"name":"Steven Hardy","email":"steven.hardy@suse.com","username":"shardy"},"change_message_id":"469f4f5b2f4ad7ed1c4eeadea8f958d794280cf3","unresolved":false,"context_lines":[{"line_number":40,"context_line":""},{"line_number":41,"context_line":"* Migrate existing resources which do require refactoring (because of"},{"line_number":42,"context_line":"  use of extensive stack details)"},{"line_number":43,"context_line":""},{"line_number":44,"context_line":"Alternatives"},{"line_number":45,"context_line":"------------"},{"line_number":46,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"1ae5cdf2_66638ddc","line":43,"in_reply_to":"1ae5cdf2_0a52da12","updated":"2014-06-03 09:54:49.000000000","message":"I agree, unless it\u0027s totally unavoidable I\u0027d really, really like to try and avoid breaking existing plugins.  I know from various conversations that there are a relatively significant number of folks maintaining out of tree plugins, so we should think hard before we break them all.","commit_id":"23464e2d9405ec5c591764db57e4c760fcfcef08"},{"author":{"_account_id":6488,"name":"Clint Byrum","email":"clint@fewbar.com","username":"clint-fewbar"},"change_message_id":"c20ca4f9303e6a13c56a7fcc0a2459fc2a26ca59","unresolved":false,"context_lines":[{"line_number":40,"context_line":""},{"line_number":41,"context_line":"* Migrate existing resources which do require refactoring (because of"},{"line_number":42,"context_line":"  use of extensive stack details)"},{"line_number":43,"context_line":""},{"line_number":44,"context_line":"Alternatives"},{"line_number":45,"context_line":"------------"},{"line_number":46,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"1ae5cdf2_6ae342a4","line":43,"in_reply_to":"1ae5cdf2_66638ddc","updated":"2014-06-03 16:58:21.000000000","message":"I will add explicit guarantee that we won\u0027t break the plugin API. We already have the appropriate split between create/delete/update/check to support the observer running checks, and the converger running create/update/delete. The difference happens in the engine, not the plugins.","commit_id":"23464e2d9405ec5c591764db57e4c760fcfcef08"},{"author":{"_account_id":7253,"name":"Jason Dunsmore","email":"jasondunsmore@gmail.com","username":"jbd"},"change_message_id":"99a6803ddfbebc276284c7aac20b028560ad26ca","unresolved":false,"context_lines":[{"line_number":22,"context_line":"from robustness and scalability limitations due to the need to keep the"},{"line_number":23,"context_line":"entire stack in RAM while creating or updating. This prevents horizontally"},{"line_number":24,"context_line":"scaling to address a single large stack, and it also requires a single"},{"line_number":25,"context_line":"coarse grained lock per action, preventing concurrent access to the stack."},{"line_number":26,"context_line":""},{"line_number":27,"context_line":"Proposed change"},{"line_number":28,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":2,"id":"baada198_11e6e916","line":25,"updated":"2014-07-08 21:00:11.000000000","message":"Can you provide a use case for when \"concurrent access to the stack\" is needed?","commit_id":"8d983732064b3b0a0eba611099f79da026f42ac1"},{"author":{"_account_id":7253,"name":"Jason Dunsmore","email":"jasondunsmore@gmail.com","username":"jbd"},"change_message_id":"5731a4512a0b896b69ed3e2dc98e42b4f028ba88","unresolved":false,"context_lines":[{"line_number":22,"context_line":"from robustness and scalability limitations due to the need to keep the"},{"line_number":23,"context_line":"entire stack in RAM while creating or updating. This prevents horizontally"},{"line_number":24,"context_line":"scaling to address a single large stack, and it also requires a single"},{"line_number":25,"context_line":"coarse grained lock per action, preventing concurrent access to the stack."},{"line_number":26,"context_line":""},{"line_number":27,"context_line":"Proposed change"},{"line_number":28,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":2,"id":"baada198_b409abdf","line":25,"in_reply_to":"baada198_11e6e916","updated":"2014-07-08 21:18:22.000000000","message":"I found a use case in https://review.openstack.org/#/c/95907/6/specs/convergence.rst  I will comment on that review.","commit_id":"8d983732064b3b0a0eba611099f79da026f42ac1"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"84ed843a5f45d6ac39c55ddd94620a5432ef472c","unresolved":false,"context_lines":[{"line_number":36,"context_line":"    * Storage backends will be expected to serialize concurrent access"},{"line_number":37,"context_line":"      to the same resource."},{"line_number":38,"context_line":""},{"line_number":39,"context_line":"* Create a new \"job queue\" style RPC for performing convergence operations"},{"line_number":40,"context_line":"  on resources."},{"line_number":41,"context_line":""},{"line_number":42,"context_line":"* Migrate existing resources which do not require refactoring (resources"}],"source_content_type":"text/x-rst","patch_set":2,"id":"1ae5cdf2_0acef182","line":39,"updated":"2014-06-27 23:18:59.000000000","message":"What is this? Could this just use http://docs.openstack.org/developer/taskflow/jobs.html or is this something different (unsure about what job queue and RPC mean when combined together).","commit_id":"8d983732064b3b0a0eba611099f79da026f42ac1"},{"author":{"_account_id":6488,"name":"Clint Byrum","email":"clint@fewbar.com","username":"clint-fewbar"},"change_message_id":"2337fd32cc17fd6384cb1a3e09e28a3af54831bd","unresolved":false,"context_lines":[{"line_number":36,"context_line":"    * Storage backends will be expected to serialize concurrent access"},{"line_number":37,"context_line":"      to the same resource."},{"line_number":38,"context_line":""},{"line_number":39,"context_line":"* Create a new \"job queue\" style RPC for performing convergence operations"},{"line_number":40,"context_line":"  on resources."},{"line_number":41,"context_line":""},{"line_number":42,"context_line":"* Migrate existing resources which do not require refactoring (resources"}],"source_content_type":"text/x-rst","patch_set":2,"id":"baada198_31f1de6a","line":39,"in_reply_to":"1ae5cdf2_0acef182","updated":"2014-07-17 17:01:39.000000000","message":"When I wrote this I didn\u0027t really understand how RPC worked fully. I\u0027m more familiar, and it already kind of works this way. So I\u0027m going to change this to just say \"Create new converge RPC commands.\"\n\nThe taskflow tie-in will come in after we get the current system working with the disconnected observer/convergence engine, so that we can more reliably retry and resume resource workflows. This is primarily a way to eliminate the FAILED state that will happen whenever we do maintenance on the engine, and may even turn into a child spec of the higher level convergence spec which states this as one of its goals.","commit_id":"8d983732064b3b0a0eba611099f79da026f42ac1"},{"author":{"_account_id":4571,"name":"Steve Baker","email":"sbaker@redhat.com","username":"steve-stevebaker"},"change_message_id":"f26550b81ef8299eb3153c3345d0e6fb4bf9a3e7","unresolved":false,"context_lines":[{"line_number":43,"context_line":"  whose create/update/delete/check methods can run in isolation)"},{"line_number":44,"context_line":""},{"line_number":45,"context_line":"* Migrate existing resources which do require refactoring (because of"},{"line_number":46,"context_line":"  use of extensive stack details)"},{"line_number":47,"context_line":""},{"line_number":48,"context_line":"Note that the actual resource plugin API will not be broken. Loading"},{"line_number":49,"context_line":"of details from the stack will still function as always. However, such"}],"source_content_type":"text/x-rst","patch_set":2,"id":"1ae5cdf2_bf866f63","line":46,"updated":"2014-06-25 23:51:59.000000000","message":"There should probably be a distinction between resources which reference their stack, and resource which reference their sibling resources.\n\nIt should be possible to optimise the latter so the other resources in the stack are only loaded if requested.","commit_id":"8d983732064b3b0a0eba611099f79da026f42ac1"},{"author":{"_account_id":6488,"name":"Clint Byrum","email":"clint@fewbar.com","username":"clint-fewbar"},"change_message_id":"2337fd32cc17fd6384cb1a3e09e28a3af54831bd","unresolved":false,"context_lines":[{"line_number":43,"context_line":"  whose create/update/delete/check methods can run in isolation)"},{"line_number":44,"context_line":""},{"line_number":45,"context_line":"* Migrate existing resources which do require refactoring (because of"},{"line_number":46,"context_line":"  use of extensive stack details)"},{"line_number":47,"context_line":""},{"line_number":48,"context_line":"Note that the actual resource plugin API will not be broken. Loading"},{"line_number":49,"context_line":"of details from the stack will still function as always. However, such"}],"source_content_type":"text/x-rst","patch_set":2,"id":"baada198_7147d677","line":46,"in_reply_to":"1ae5cdf2_bf866f63","updated":"2014-07-17 17:01:39.000000000","message":"Done","commit_id":"8d983732064b3b0a0eba611099f79da026f42ac1"},{"author":{"_account_id":4328,"name":"Steven Hardy","email":"steven.hardy@suse.com","username":"shardy"},"change_message_id":"c939b387744aff252106cb8063c6efad7209fc5a","unresolved":false,"context_lines":[{"line_number":27,"context_line":"Proposed change"},{"line_number":28,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":29,"context_line":""},{"line_number":30,"context_line":"* Create a new process named heat-convergence-engine"},{"line_number":31,"context_line":""},{"line_number":32,"context_line":"    * New process will only act on resources one at a time, no stack-wide"},{"line_number":33,"context_line":"      operations will be done."}],"source_content_type":"text/x-rst","patch_set":3,"id":"1abeadc6_9f09292f","line":30,"updated":"2014-08-18 10:26:49.000000000","message":"One thought, would it be better to just add an additional interface to the existing heat-engine, so we can scale them out as is already done, but then have an engine call via RPC to round-robin the resources to $n other engines?\n\nSeems like this could be marginally cleaner and more convenient from both a conceptual and distribution/documentation/packaging perspective?","commit_id":"2837422b38c1cbf8853fc363d06b3e314e4dc9b9"},{"author":{"_account_id":10487,"name":"Kanagaraj Manickam","email":"mkr1481@gmail.com","username":"kanagarajm"},"change_message_id":"535f7b7b0623d91a0d990c3a755ef34cb909b6bc","unresolved":false,"context_lines":[{"line_number":28,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":29,"context_line":""},{"line_number":30,"context_line":"* Create a new process named heat-convergence-engine"},{"line_number":31,"context_line":""},{"line_number":32,"context_line":"    * New process will only act on resources one at a time, no stack-wide"},{"line_number":33,"context_line":"      operations will be done."},{"line_number":34,"context_line":"    * Many processes will be able to act on many resources, allowing scale-out"}],"source_content_type":"text/x-rst","patch_set":3,"id":"1abeadc6_0bade888","line":31,"updated":"2014-08-18 05:48:22.000000000","message":"Hi Clint,\n\nI would like to provide my view points on creating new process:\n\nToday  the stack is getting provisioned completely from a given heat-engine. and The intent of this blue print is, to provision a stack across multiple engines provided each one can process at the resource level than a stack level to support scaling.\n\nCurrently the nested-stacks are considered as resource in heat-engine. In the similar line, by considering the \"the stack also as resource\" we can achieve the scaling of heat-engine for a stack as follows:\n\n1. update the heat-engine to treat the stack as a resource. stack-resource module will perform the initial graph creation and graph traversing to provision the stack as being done today. To process the resource action part, stack-resource will publish the message to the same queue where all heat-engines are subscribed to. \n2. update the heat-engine to process the message from the queue and appropriately call the respective resource module to perform the resource action. Here the resource module could be either of stack-resource module or granular resource module.\n\nHere, the same heat-engine is enhanced to perform the actions on stack as well as it\u0027s resources. This will avoid the need of a new process.\n\nIn this approach, a given heat-engine will be able to perform the action at stack level as well as resource level via RPC.","commit_id":"2837422b38c1cbf8853fc363d06b3e314e4dc9b9"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"359b1cf9513381001391802471dda6f0bcff107e","unresolved":false,"context_lines":[{"line_number":74,"context_line":"----------"},{"line_number":75,"context_line":""},{"line_number":76,"context_line":"* Create a new process named heat-convergence-engine"},{"line_number":77,"context_line":"* Create a new \"job queue\" style RPC"},{"line_number":78,"context_line":"* Migrate existing resources which do not require refactoring"},{"line_number":79,"context_line":"* Migrate existing resources which do require refactoring"},{"line_number":80,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"1abeadc6_a966e54e","line":77,"updated":"2014-08-22 21:00:34.000000000","message":"Isn\u0027t that called a message queue that messages (the jobs as formatted messages) go into? Whats the benefit of RPC in this case? Doesn\u0027t seem like RPC if it\u0027s really a job queue IMHO. Maybe just drop the RPC part?","commit_id":"2837422b38c1cbf8853fc363d06b3e314e4dc9b9"}]}
