)]}'
{"specs/chronicles-of-a-dlm.rst":[{"author":{"_account_id":6488,"name":"Clint Byrum","email":"clint@fewbar.com","username":"clint-fewbar"},"change_message_id":"8f1587467cdde1deaf486c1b5df4591e5c75d351","unresolved":false,"context_lines":[{"line_number":82,"context_line":""},{"line_number":83,"context_line":"**Problem:**"},{"line_number":84,"context_line":""},{"line_number":85,"context_line":"Multiple engines working on the same stack (or nested stack of)"},{"line_number":86,"context_line":""},{"line_number":87,"context_line":"**Solution:**"},{"line_number":88,"context_line":""}],"source_content_type":"text/x-rst","patch_set":11,"id":"1a4dcd0f_096d63cb","line":85,"updated":"2015-08-06 22:21:16.000000000","message":"Note that with convergence, this is supposed to effectively go away. There won\u0027t be any in-memory-only state, so there\u0027s no need for this lock. Instead, the db will reflect what is, and isn\u0027t done, and threads will only start after having moved a resource from \"needs doing\" to \"in progress\". However, I have not paid attention to how the implementation is solving the stale-in-progress problem. That is definitely a locking problem, as if a resource got stuck in \"in progress\" without an engine, how do you reassign it to a new engine?","commit_id":"53a556e6914ce94dd04d9a9cec9eb4a2930b52cc"},{"author":{"_account_id":6488,"name":"Clint Byrum","email":"clint@fewbar.com","username":"clint-fewbar"},"change_message_id":"8f1587467cdde1deaf486c1b5df4591e5c75d351","unresolved":false,"context_lines":[{"line_number":100,"context_line":""},{"line_number":101,"context_line":"- Database locking requires manual operator manipulation to determine"},{"line_number":102,"context_line":"  when a lock is busted and/or broken. That manipulation varies between"},{"line_number":103,"context_line":"  database vendors/providers (lack of uniformity)."},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"- Lacks introspection capabilities? For example it is not necessarily easy to"},{"line_number":106,"context_line":"  determine who is trying to acquire the lock, how long they have"}],"source_content_type":"text/x-rst","patch_set":11,"id":"1a4dcd0f_be8d83d4","line":103,"updated":"2015-08-06 22:21:16.000000000","message":"Negative. If a lock is busted, the UUID of the instance of heat-engine that has taken control of the lock will not respond on the oslo.messaging topic in time, and another engine is allowed to take over the stack at that point. There is zero operator intervention required. Sometimes the stack will error because its engine failed, but the user can retry an operation and the new engine that picked up the stack will try to break the lock at that point via the messaging ping.","commit_id":"53a556e6914ce94dd04d9a9cec9eb4a2930b52cc"},{"author":{"_account_id":6488,"name":"Clint Byrum","email":"clint@fewbar.com","username":"clint-fewbar"},"change_message_id":"8f1587467cdde1deaf486c1b5df4591e5c75d351","unresolved":false,"context_lines":[{"line_number":104,"context_line":""},{"line_number":105,"context_line":"- Lacks introspection capabilities? For example it is not necessarily easy to"},{"line_number":106,"context_line":"  determine who is trying to acquire the lock, how long they have"},{"line_number":107,"context_line":"  waited and so on."},{"line_number":108,"context_line":""},{"line_number":109,"context_line":"**Code for:**"},{"line_number":110,"context_line":""}],"source_content_type":"text/x-rst","patch_set":11,"id":"1a4dcd0f_3e14134b","line":107,"updated":"2015-08-06 22:21:16.000000000","message":"The UUID is logged by each engine at startup, so one can look there to find out what engine_id \u003d\u003d what host. It would be fairly trivial to add a \u0027hostname\u0027 informational field to the lock table as well to allow quicker introspection. Logs are the only way to see if some other engine wants to try and steal.","commit_id":"53a556e6914ce94dd04d9a9cec9eb4a2930b52cc"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"4163e82eb47280822028fcd29bb1a25d48c5bf0e","unresolved":false,"context_lines":[{"line_number":9,"context_line":"some set of actions in an atomic manner performed by some distributed set of"},{"line_number":10,"context_line":"applications on some set of distributed resources **without** having those"},{"line_number":11,"context_line":"resources end up in some corrupted state due those actions being performed on"},{"line_number":12,"context_line":"them without the traditional concept of `locking`_."},{"line_number":13,"context_line":""},{"line_number":14,"context_line":"A `DLM`_ is one such concept/solution that can help (but not entirely"},{"line_number":15,"context_line":"solve) these types of common resource manipulation patterns in distributed"}],"source_content_type":"text/x-rst","patch_set":14,"id":"1a4dcd0f_f75d1749","line":12,"updated":"2015-08-10 07:02:12.000000000","message":"IIUC, in this spec we want DLM only for the \"control plane\", like attach volume and resize volume requests for Cinder or migrate vs snapshot instance for Nova.\nAnd this is not about DLM as a service for OpenStack workloads.\n\nIs it correct?","commit_id":"272c9c2e7ca50c496a0a40a9c8a3e1e9c0f0c28b"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"6b7368adb830a43175e4cb0a74512e8ed24c9905","unresolved":false,"context_lines":[{"line_number":9,"context_line":"some set of actions in an atomic manner performed by some distributed set of"},{"line_number":10,"context_line":"applications on some set of distributed resources **without** having those"},{"line_number":11,"context_line":"resources end up in some corrupted state due those actions being performed on"},{"line_number":12,"context_line":"them without the traditional concept of `locking`_."},{"line_number":13,"context_line":""},{"line_number":14,"context_line":"A `DLM`_ is one such concept/solution that can help (but not entirely"},{"line_number":15,"context_line":"solve) these types of common resource manipulation patterns in distributed"}],"source_content_type":"text/x-rst","patch_set":14,"id":"1a4dcd0f_34a59446","line":12,"in_reply_to":"1a4dcd0f_f75d1749","updated":"2015-08-10 20:11:31.000000000","message":"Correct.","commit_id":"272c9c2e7ca50c496a0a40a9c8a3e1e9c0f0c28b"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"aa55fa9003a059cefd5f5e078a65c895e2515838","unresolved":false,"context_lines":[{"line_number":135,"context_line":"****"},{"line_number":136,"context_line":""},{"line_number":137,"context_line":"Proposed change"},{"line_number":138,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":139,"context_line":""},{"line_number":140,"context_line":"\u003cWIP\u003e"},{"line_number":141,"context_line":""}],"source_content_type":"text/x-rst","patch_set":14,"id":"1a4dcd0f_1baf9c95","line":138,"updated":"2015-08-09 09:53:13.000000000","message":"It seems clear that Oslo libraries should be the best place to contain the DLM implementation. Could  this be accepted in the spec?","commit_id":"272c9c2e7ca50c496a0a40a9c8a3e1e9c0f0c28b"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"909be1934bce39fecdb9b1229b1b7aa061604b57","unresolved":false,"context_lines":[{"line_number":135,"context_line":"****"},{"line_number":136,"context_line":""},{"line_number":137,"context_line":"Proposed change"},{"line_number":138,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":139,"context_line":""},{"line_number":140,"context_line":"\u003cWIP\u003e"},{"line_number":141,"context_line":""}],"source_content_type":"text/x-rst","patch_set":14,"id":"1a4dcd0f_f9543322","line":138,"in_reply_to":"1a4dcd0f_1baf9c95","updated":"2015-08-10 00:35:25.000000000","message":"I\u0027m really hoping there isn\u0027t our own \u0027DLM implementation\u0027 because that would make me cry :-P","commit_id":"272c9c2e7ca50c496a0a40a9c8a3e1e9c0f0c28b"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"6b7368adb830a43175e4cb0a74512e8ed24c9905","unresolved":false,"context_lines":[{"line_number":135,"context_line":"****"},{"line_number":136,"context_line":""},{"line_number":137,"context_line":"Proposed change"},{"line_number":138,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":139,"context_line":""},{"line_number":140,"context_line":"\u003cWIP\u003e"},{"line_number":141,"context_line":""}],"source_content_type":"text/x-rst","patch_set":14,"id":"1a4dcd0f_bf0bbd23","line":138,"in_reply_to":"1a4dcd0f_d219097b","updated":"2015-08-10 20:11:31.000000000","message":"I\u0027m torn between that, in regards to DLM backend independent abstraction, yes, there exists tooz for this, but as for why we are making all the things pluggable in the first place, especially for a concept like a DLM confuses me. Sometimes we really do need to be opinionated, and I start to feel that by making it pluggable/abstract, and all that that we are actually messing up openstack by doing that.\n\nIMHO I\u0027d rather have 1 opinionated DLM selection and just have everyone consistently use it, vs having 20 variations with none that work very well, wouldn\u0027t u? Sometimes we have to make decisions and pick things, vs being neutral about everything... because in the end the communities inability to make some kind of opinionated decision actually hurts it more than it helps...","commit_id":"272c9c2e7ca50c496a0a40a9c8a3e1e9c0f0c28b"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"4163e82eb47280822028fcd29bb1a25d48c5bf0e","unresolved":false,"context_lines":[{"line_number":135,"context_line":"****"},{"line_number":136,"context_line":""},{"line_number":137,"context_line":"Proposed change"},{"line_number":138,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":139,"context_line":""},{"line_number":140,"context_line":"\u003cWIP\u003e"},{"line_number":141,"context_line":""}],"source_content_type":"text/x-rst","patch_set":14,"id":"1a4dcd0f_d219097b","line":138,"in_reply_to":"1a4dcd0f_f9543322","updated":"2015-08-10 07:02:12.000000000","message":"Whatever DLM backend we accepted, shouldn\u0027t we still have to expose the backend independent abstract OpenStack DLM interface for all of its projects? Wouldn\u0027t we still want it being versioned among with OpenStack releases? If we would, that else but Oslo might come in handy?","commit_id":"272c9c2e7ca50c496a0a40a9c8a3e1e9c0f0c28b"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"aa55fa9003a059cefd5f5e078a65c895e2515838","unresolved":false,"context_lines":[{"line_number":140,"context_line":"\u003cWIP\u003e"},{"line_number":141,"context_line":""},{"line_number":142,"context_line":"Alternatives"},{"line_number":143,"context_line":"------------"},{"line_number":144,"context_line":""},{"line_number":145,"context_line":"\u003cWIP\u003e"},{"line_number":146,"context_line":""}],"source_content_type":"text/x-rst","patch_set":14,"id":"1a4dcd0f_db9ff44e","line":143,"updated":"2015-08-09 09:53:13.000000000","message":"There is an alternative to implement DLM for Oslo.DB without any additional components and technological stacks introduced. And therefore, it seems might be the most benefitial way for the community and operators.","commit_id":"272c9c2e7ca50c496a0a40a9c8a3e1e9c0f0c28b"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"4163e82eb47280822028fcd29bb1a25d48c5bf0e","unresolved":false,"context_lines":[{"line_number":140,"context_line":"\u003cWIP\u003e"},{"line_number":141,"context_line":""},{"line_number":142,"context_line":"Alternatives"},{"line_number":143,"context_line":"------------"},{"line_number":144,"context_line":""},{"line_number":145,"context_line":"\u003cWIP\u003e"},{"line_number":146,"context_line":""}],"source_content_type":"text/x-rst","patch_set":14,"id":"1a4dcd0f_b2023db3","line":143,"in_reply_to":"1a4dcd0f_79a34350","updated":"2015-08-10 07:02:12.000000000","message":"We may want  be creating our own things because:\na) existing  solutions may be a huge overkill for OpenStack control plane needs - multi-Paxos, group membership, replicated logs snapshots, add-another-cool-thing - what for?..\nb) we may never know the trade offs made by a 3rd party solution developers. And AFAIK, all of existing multi-Paxos based solutions diverged from original Paxos standard a *lot*.","commit_id":"272c9c2e7ca50c496a0a40a9c8a3e1e9c0f0c28b"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"6b7368adb830a43175e4cb0a74512e8ed24c9905","unresolved":false,"context_lines":[{"line_number":140,"context_line":"\u003cWIP\u003e"},{"line_number":141,"context_line":""},{"line_number":142,"context_line":"Alternatives"},{"line_number":143,"context_line":"------------"},{"line_number":144,"context_line":""},{"line_number":145,"context_line":"\u003cWIP\u003e"},{"line_number":146,"context_line":""}],"source_content_type":"text/x-rst","patch_set":14,"id":"1a4dcd0f_1f4bd15a","line":143,"in_reply_to":"1a4dcd0f_b2023db3","updated":"2015-08-10 20:11:31.000000000","message":"All these existing solutions are open source projects right? Why would we want to create our own vs just getting involved in said opensource projects, that\u0027s what concerns me about \u0027creating our own things\u0027, it\u0027s not like the other solutions are closed source and can\u0027t be changed/used/tweaked; we don\u0027t have to live in a vacuum if we don\u0027t want to...","commit_id":"272c9c2e7ca50c496a0a40a9c8a3e1e9c0f0c28b"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"a6b7f398613bcbff64759c3740e0c1754dd839eb","unresolved":false,"context_lines":[{"line_number":140,"context_line":"\u003cWIP\u003e"},{"line_number":141,"context_line":""},{"line_number":142,"context_line":"Alternatives"},{"line_number":143,"context_line":"------------"},{"line_number":144,"context_line":""},{"line_number":145,"context_line":"\u003cWIP\u003e"},{"line_number":146,"context_line":""}],"source_content_type":"text/x-rst","patch_set":14,"id":"1a4dcd0f_99940fe9","line":143,"in_reply_to":"1a4dcd0f_d36b5884","updated":"2015-08-10 00:30:23.000000000","message":"Sounds somewhat like tooz ;)\n\nhttps://github.com/openstack/tooz","commit_id":"272c9c2e7ca50c496a0a40a9c8a3e1e9c0f0c28b"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"0a6cd97df69de4eafd2cea309e4d4c7021ba49cb","unresolved":false,"context_lines":[{"line_number":140,"context_line":"\u003cWIP\u003e"},{"line_number":141,"context_line":""},{"line_number":142,"context_line":"Alternatives"},{"line_number":143,"context_line":"------------"},{"line_number":144,"context_line":""},{"line_number":145,"context_line":"\u003cWIP\u003e"},{"line_number":146,"context_line":""}],"source_content_type":"text/x-rst","patch_set":14,"id":"1a4dcd0f_e82b0393","line":143,"in_reply_to":"1a4dcd0f_db9ff44e","updated":"2015-08-09 18:31:19.000000000","message":"Example of such design https://gist.github.com/ryandotsmith/c95fd21fab91b0823328\n\nSo, probably the suggested flow with timeout-wrapped critical sections and TTL-based locks might fit our needs as well:\n\n\"Prune expired lock.\n\nCreate Item unless exists\n\nYield to critical section inside of timeout\n\nDelete Item\"","commit_id":"272c9c2e7ca50c496a0a40a9c8a3e1e9c0f0c28b"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"a6b7f398613bcbff64759c3740e0c1754dd839eb","unresolved":false,"context_lines":[{"line_number":140,"context_line":"\u003cWIP\u003e"},{"line_number":141,"context_line":""},{"line_number":142,"context_line":"Alternatives"},{"line_number":143,"context_line":"------------"},{"line_number":144,"context_line":""},{"line_number":145,"context_line":"\u003cWIP\u003e"},{"line_number":146,"context_line":""}],"source_content_type":"text/x-rst","patch_set":14,"id":"1a4dcd0f_79a34350","line":143,"in_reply_to":"1a4dcd0f_db9ff44e","updated":"2015-08-10 00:30:23.000000000","message":"Except this gets us back into the creating our own things because openstack (as whole)  is unwilling to accept solutions that are theoretically proven to work vs creating our own things. I get the idea that yes it is a \u0027alternative\u0027 in some sense, but why :-/\n\nIt seems like we would just be avoiding the elephant in the room by trying to create our own solution, I\u0027d much rather prefer actually looking at the elephant head on and asking ourselves the tough questions, vs what I see most projects here doing, which is just avoiding that question, and then suffering from that decision.... Things like raft, zookeeper, paxos, are solved things, and why we would go ahead and avoid them IMHO is illogical and confuses me greatly. If this community doesn\u0027t have the \u0027guts\u0027 to make decisions like this and use known solutions that other cloud solutions/projects/other use, then it\u0027s very broken :-/\n\n^^ Is not meant to be harsh, its just the reality that I see and routinely ask myself why we are doing this to ourselves...","commit_id":"272c9c2e7ca50c496a0a40a9c8a3e1e9c0f0c28b"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"1d8dc7c397bc825fcd90e1ea8d923c9e15074591","unresolved":false,"context_lines":[{"line_number":140,"context_line":"\u003cWIP\u003e"},{"line_number":141,"context_line":""},{"line_number":142,"context_line":"Alternatives"},{"line_number":143,"context_line":"------------"},{"line_number":144,"context_line":""},{"line_number":145,"context_line":"\u003cWIP\u003e"},{"line_number":146,"context_line":""}],"source_content_type":"text/x-rst","patch_set":14,"id":"1a4dcd0f_d36b5884","line":143,"in_reply_to":"1a4dcd0f_e82b0393","updated":"2015-08-09 19:21:13.000000000","message":"And there is another obvious alternative exist - don\u0027t implement DLM but instead create a common library for OpenStack distributed consensus, either Paxos, or - probably a better one - Raft based.\n\nWhich could be actually not so bad as it may seemed, considering the recent feature in Keystone - Fernet tokens. This one implies some distributed consensus solution in order to rotate and expire fernet keys *efficiently*... So, why not.","commit_id":"272c9c2e7ca50c496a0a40a9c8a3e1e9c0f0c28b"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"aa55fa9003a059cefd5f5e078a65c895e2515838","unresolved":false,"context_lines":[{"line_number":153,"context_line":"- All the reviewers, code creators, PTL(s) of OpenStack?"},{"line_number":154,"context_line":""},{"line_number":155,"context_line":"Work Items"},{"line_number":156,"context_line":"----------"},{"line_number":157,"context_line":""},{"line_number":158,"context_line":"\u003cWIP\u003e"},{"line_number":159,"context_line":""}],"source_content_type":"text/x-rst","patch_set":14,"id":"1a4dcd0f_1b86bc13","line":156,"updated":"2015-08-09 09:53:13.000000000","message":"The very first item could be to design and document a DLM RESTful API with *clearly* defined user expectations and described behavior, types of supported distributed locks, failure modes and so on.\nThis also seems clear enough to be accepted in the spec.","commit_id":"272c9c2e7ca50c496a0a40a9c8a3e1e9c0f0c28b"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"4163e82eb47280822028fcd29bb1a25d48c5bf0e","unresolved":false,"context_lines":[{"line_number":153,"context_line":"- All the reviewers, code creators, PTL(s) of OpenStack?"},{"line_number":154,"context_line":""},{"line_number":155,"context_line":"Work Items"},{"line_number":156,"context_line":"----------"},{"line_number":157,"context_line":""},{"line_number":158,"context_line":"\u003cWIP\u003e"},{"line_number":159,"context_line":""}],"source_content_type":"text/x-rst","patch_set":14,"id":"1a4dcd0f_d7a6db52","line":156,"in_reply_to":"1a4dcd0f_19023f3f","updated":"2015-08-10 07:02:12.000000000","message":"Even if the hypothetical DLM interface contained only 3 API calls, we should have them documented. Is it TTL based or not? If it is, document timeouts for critical paths as well. Otherwise, document heartbeats to detect lock holder failures and its failure modes, like either to allow to execute its critical path, or interrupt it immediately.","commit_id":"272c9c2e7ca50c496a0a40a9c8a3e1e9c0f0c28b"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"0a6cd97df69de4eafd2cea309e4d4c7021ba49cb","unresolved":false,"context_lines":[{"line_number":153,"context_line":"- All the reviewers, code creators, PTL(s) of OpenStack?"},{"line_number":154,"context_line":""},{"line_number":155,"context_line":"Work Items"},{"line_number":156,"context_line":"----------"},{"line_number":157,"context_line":""},{"line_number":158,"context_line":"\u003cWIP\u003e"},{"line_number":159,"context_line":""}],"source_content_type":"text/x-rst","patch_set":14,"id":"1a4dcd0f_e8546322","line":156,"in_reply_to":"1a4dcd0f_1b86bc13","updated":"2015-08-09 18:31:19.000000000","message":"Fix: or not RESTful API, in case of a library-based implementation","commit_id":"272c9c2e7ca50c496a0a40a9c8a3e1e9c0f0c28b"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"a6b7f398613bcbff64759c3740e0c1754dd839eb","unresolved":false,"context_lines":[{"line_number":153,"context_line":"- All the reviewers, code creators, PTL(s) of OpenStack?"},{"line_number":154,"context_line":""},{"line_number":155,"context_line":"Work Items"},{"line_number":156,"context_line":"----------"},{"line_number":157,"context_line":""},{"line_number":158,"context_line":"\u003cWIP\u003e"},{"line_number":159,"context_line":""}],"source_content_type":"text/x-rst","patch_set":14,"id":"1a4dcd0f_19023f3f","line":156,"in_reply_to":"1a4dcd0f_e8546322","updated":"2015-08-10 00:30:23.000000000","message":"Oh jeez, I\u0027m really hoping not the \u0027design and document a DLM RESTful API\u0027, DLM is a figured out thing, we really shouldn\u0027t be baking another solution \u0027just because\u0027 here. I\u0027m fine with looking at existing solutions, which exist, not really fine in going ahead and building to many of our own things here, this community just doesn\u0027t have the manpower IMHO to do that... And even if it did I\u0027d really question that choice because it\u0027s along the lines of NIH syndrome...","commit_id":"272c9c2e7ca50c496a0a40a9c8a3e1e9c0f0c28b"},{"author":{"_account_id":10239,"name":"Dmitry Tantsur","email":"dtantsur@protonmail.com","username":"dtantsur"},"change_message_id":"37b3d50745054c01951b646d188051bfb4aa81bb","unresolved":false,"context_lines":[{"line_number":102,"context_line":"  a shared lock to an exclusive lock (I\u0027m unsure how this works, usually"},{"line_number":103,"context_line":"  upgrading a shared lock to an exclusive one means kicking out all other"},{"line_number":104,"context_line":"  active readers, what happens if there are multiple readers doing this"},{"line_number":105,"context_line":"  at the same time?)."},{"line_number":106,"context_line":"- Reclaim/take over lock mechanism? (unsure exactly how this works, manual?)"},{"line_number":107,"context_line":""},{"line_number":108,"context_line":"**Code/doc references:**"}],"source_content_type":"text/x-rst","patch_set":18,"id":"1a4dcd0f_d1dc54e6","line":105,"updated":"2015-08-11 14:06:55.000000000","message":"1. ironic shared lock is not a lock actually, 2. upgrading lock means just acquiring a proper lock, and it can fail.","commit_id":"5e9ba083850d46de549917e22b84e4448ebbb706"},{"author":{"_account_id":6618,"name":"Ruby Loo","email":"opensrloo@gmail.com","username":"rloo"},"change_message_id":"eafc17069a81cfe9a1e1f049a12f4e3abfabb6dd","unresolved":false,"context_lines":[{"line_number":102,"context_line":"  a shared lock to an exclusive lock (I\u0027m unsure how this works, usually"},{"line_number":103,"context_line":"  upgrading a shared lock to an exclusive one means kicking out all other"},{"line_number":104,"context_line":"  active readers, what happens if there are multiple readers doing this"},{"line_number":105,"context_line":"  at the same time?)."},{"line_number":106,"context_line":"- Reclaim/take over lock mechanism? (unsure exactly how this works, manual?)"},{"line_number":107,"context_line":""},{"line_number":108,"context_line":"**Code/doc references:**"}],"source_content_type":"text/x-rst","patch_set":18,"id":"1a4dcd0f_ff491732","line":105,"updated":"2015-08-11 14:51:26.000000000","message":"waits for all locks to be released before getting an exclusive lock, or times out if it gets bored (ie waits beyond a configurable value).","commit_id":"5e9ba083850d46de549917e22b84e4448ebbb706"},{"author":{"_account_id":6618,"name":"Ruby Loo","email":"opensrloo@gmail.com","username":"rloo"},"change_message_id":"eafc17069a81cfe9a1e1f049a12f4e3abfabb6dd","unresolved":false,"context_lines":[{"line_number":103,"context_line":"  upgrading a shared lock to an exclusive one means kicking out all other"},{"line_number":104,"context_line":"  active readers, what happens if there are multiple readers doing this"},{"line_number":105,"context_line":"  at the same time?)."},{"line_number":106,"context_line":"- Reclaim/take over lock mechanism? (unsure exactly how this works, manual?)"},{"line_number":107,"context_line":""},{"line_number":108,"context_line":"**Code/doc references:**"},{"line_number":109,"context_line":""}],"source_content_type":"text/x-rst","patch_set":18,"id":"1a4dcd0f_a5bb820e","line":106,"updated":"2015-08-11 14:51:26.000000000","message":"It isn\u0027t a take over \"lock mechanism\" (if I understand this). There is a take-over mechanism in ironic, to handle the case where a conductor that is associated with an active node (instance deployed) goes on holiday.\n\nWe have a periodic task [1] that checks for nodes associated with a conductor. If any of those nodes were made active by another conductor, the current conductor \u0027takes over\u0027. Which typically means setting up local state on the current conductor so that if the node has to be rebuilt or whatever, the current conductor has the info needed to do that. \n\n[1] https://github.com/openstack/ironic/blob/aa0808fc976327f22c28c6d0e30c429293e72c73/ironic/conductor/manager.py#L1260","commit_id":"5e9ba083850d46de549917e22b84e4448ebbb706"},{"author":{"_account_id":10239,"name":"Dmitry Tantsur","email":"dtantsur@protonmail.com","username":"dtantsur"},"change_message_id":"37b3d50745054c01951b646d188051bfb4aa81bb","unresolved":false,"context_lines":[{"line_number":103,"context_line":"  upgrading a shared lock to an exclusive one means kicking out all other"},{"line_number":104,"context_line":"  active readers, what happens if there are multiple readers doing this"},{"line_number":105,"context_line":"  at the same time?)."},{"line_number":106,"context_line":"- Reclaim/take over lock mechanism? (unsure exactly how this works, manual?)"},{"line_number":107,"context_line":""},{"line_number":108,"context_line":"**Code/doc references:**"},{"line_number":109,"context_line":""}],"source_content_type":"text/x-rst","patch_set":18,"id":"1a4dcd0f_519604f1","line":106,"updated":"2015-08-11 14:06:55.000000000","message":"if conductor is inactive for some time (does not record heartbeat in database), its nodes are overtaken by another conductor.","commit_id":"5e9ba083850d46de549917e22b84e4448ebbb706"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"052c5665debfd04f299e1b7dffb4f81f2b8004dd","unresolved":false,"context_lines":[{"line_number":103,"context_line":"  upgrading a shared lock to an exclusive one means kicking out all other"},{"line_number":104,"context_line":"  active readers, what happens if there are multiple readers doing this"},{"line_number":105,"context_line":"  at the same time?)."},{"line_number":106,"context_line":"- Reclaim/take over lock mechanism? (unsure exactly how this works, manual?)"},{"line_number":107,"context_line":""},{"line_number":108,"context_line":"**Code/doc references:**"},{"line_number":109,"context_line":""}],"source_content_type":"text/x-rst","patch_set":18,"id":"1a4dcd0f_ed5f2c3f","line":106,"in_reply_to":"1a4dcd0f_519604f1","updated":"2015-08-11 16:12:43.000000000","message":"Ok, so inactivity is detected by this heartbeat, does the local conductor that is really slow/inactive kill itself if it stops heartbeating to the database? Without that kind of conductor suicide both conductors could be doing the same work right (and therefore conflicting?).","commit_id":"5e9ba083850d46de549917e22b84e4448ebbb706"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"052c5665debfd04f299e1b7dffb4f81f2b8004dd","unresolved":false,"context_lines":[{"line_number":103,"context_line":"  upgrading a shared lock to an exclusive one means kicking out all other"},{"line_number":104,"context_line":"  active readers, what happens if there are multiple readers doing this"},{"line_number":105,"context_line":"  at the same time?)."},{"line_number":106,"context_line":"- Reclaim/take over lock mechanism? (unsure exactly how this works, manual?)"},{"line_number":107,"context_line":""},{"line_number":108,"context_line":"**Code/doc references:**"},{"line_number":109,"context_line":""}],"source_content_type":"text/x-rst","patch_set":18,"id":"1a4dcd0f_ad9044ee","line":106,"in_reply_to":"1a4dcd0f_a5bb820e","updated":"2015-08-11 16:12:43.000000000","message":"Ok, so \u0027going on holiday\u0027 is determined by timeout? Wondering how its determined that a conductor is on holiday, say its just a really slow conductor? If it\u0027s say really really slow, then it may actually still be doing stuff when a take-over happens right? In that case will both conductors be doing stuff on the same nodes? If the slow conductor speeds up after it\u0027s work gets taken over, any idea what happens?","commit_id":"5e9ba083850d46de549917e22b84e4448ebbb706"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"902b0268130f60e09305961676685cf0ef052d20","unresolved":false,"context_lines":[{"line_number":184,"context_line":"  on ping/pongs and database updating (each operation in this"},{"line_number":185,"context_line":"  complex \u0027stealing dance\u0027 may fail or be problematic, and therefore is not"},{"line_number":186,"context_line":"  especially simple)."},{"line_number":187,"context_line":""},{"line_number":188,"context_line":"**Code/doc references:**"},{"line_number":189,"context_line":""},{"line_number":190,"context_line":"- http://docs.openstack.org/developer/heat/_modules/heat/engine/stack_lock.html"}],"source_content_type":"text/x-rst","patch_set":22,"id":"9a1a9d01_6a073582","line":187,"updated":"2015-09-30 14:08:44.000000000","message":"I think Ceilometer and its Tooz use should be also evaluated on this spec.","commit_id":"5e331be6c564925ed8d9271e461aa199601f7925"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"7eeb3d25c63b181e1a8cfbe29f9d25b60373cc20","unresolved":false,"context_lines":[{"line_number":184,"context_line":"  on ping/pongs and database updating (each operation in this"},{"line_number":185,"context_line":"  complex \u0027stealing dance\u0027 may fail or be problematic, and therefore is not"},{"line_number":186,"context_line":"  especially simple)."},{"line_number":187,"context_line":""},{"line_number":188,"context_line":"**Code/doc references:**"},{"line_number":189,"context_line":""},{"line_number":190,"context_line":"- http://docs.openstack.org/developer/heat/_modules/heat/engine/stack_lock.html"}],"source_content_type":"text/x-rst","patch_set":22,"id":"9a1a9d01_f6c80c57","line":187,"in_reply_to":"9a1a9d01_1657c0ca","updated":"2015-09-30 15:06:08.000000000","message":"Random thought: do u want to add some more project analysis/deep-dives??? ;)","commit_id":"5e331be6c564925ed8d9271e461aa199601f7925"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"9f3eab0f001219547d22794bf38a4b74d3b88042","unresolved":false,"context_lines":[{"line_number":184,"context_line":"  on ping/pongs and database updating (each operation in this"},{"line_number":185,"context_line":"  complex \u0027stealing dance\u0027 may fail or be problematic, and therefore is not"},{"line_number":186,"context_line":"  especially simple)."},{"line_number":187,"context_line":""},{"line_number":188,"context_line":"**Code/doc references:**"},{"line_number":189,"context_line":""},{"line_number":190,"context_line":"- http://docs.openstack.org/developer/heat/_modules/heat/engine/stack_lock.html"}],"source_content_type":"text/x-rst","patch_set":22,"id":"9a1a9d01_1657c0ca","line":187,"in_reply_to":"9a1a9d01_6a073582","updated":"2015-09-30 15:03:50.000000000","message":"Possibly, I didnt want to have to dive into each project (of the now ever growing project list); but I\u0027m fine with adding a few more. Could likely spend my entire day/month looking at what each project has done...","commit_id":"5e331be6c564925ed8d9271e461aa199601f7925"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"902b0268130f60e09305961676685cf0ef052d20","unresolved":false,"context_lines":[{"line_number":209,"context_line":"  analysis done at https://aphyr.com/posts/291-call-me-maybe-zookeeper)"},{"line_number":210,"context_line":"* `consul`_ (community respected analysis done at"},{"line_number":211,"context_line":"  https://aphyr.com/posts/316-call-me-maybe-etcd-and-consul)"},{"line_number":212,"context_line":"* `etc.d`_ (community respected analysis done at"},{"line_number":213,"context_line":"  https://aphyr.com/posts/316-call-me-maybe-etcd-and-consul)"},{"line_number":214,"context_line":""},{"line_number":215,"context_line":"From a biased point of view, I am going to suggest using `zookeeper`_ as it"}],"source_content_type":"text/x-rst","patch_set":22,"id":"9a1a9d01_6aa415b9","line":212,"updated":"2015-09-30 14:08:44.000000000","message":"Why aren\u0027t we evaluating also Redis (RedLock) here? I feel like it\u0027s simple to configure and already used in OpenStack by Ceilometer. Decent Python client is also in the picture.","commit_id":"5e331be6c564925ed8d9271e461aa199601f7925"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"9f3eab0f001219547d22794bf38a4b74d3b88042","unresolved":false,"context_lines":[{"line_number":209,"context_line":"  analysis done at https://aphyr.com/posts/291-call-me-maybe-zookeeper)"},{"line_number":210,"context_line":"* `consul`_ (community respected analysis done at"},{"line_number":211,"context_line":"  https://aphyr.com/posts/316-call-me-maybe-etcd-and-consul)"},{"line_number":212,"context_line":"* `etc.d`_ (community respected analysis done at"},{"line_number":213,"context_line":"  https://aphyr.com/posts/316-call-me-maybe-etcd-and-consul)"},{"line_number":214,"context_line":""},{"line_number":215,"context_line":"From a biased point of view, I am going to suggest using `zookeeper`_ as it"}],"source_content_type":"text/x-rst","patch_set":22,"id":"9a1a9d01_b6fdb4b2","line":212,"in_reply_to":"9a1a9d01_6aa415b9","updated":"2015-09-30 15:03:50.000000000","message":"So tooz redis drivers uses the redlock algorithm. I\u0027ll mention it here.\n\nAlthough I\u0027m of the opinion that for long term sustainability we don\u0027t want to just be using redis here (because these other solutions have other features as well, that redis just doesn\u0027t have); redis just for short-term locking, fine, for longer-term capabilities, meh, not so much...","commit_id":"5e331be6c564925ed8d9271e461aa199601f7925"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"902b0268130f60e09305961676685cf0ef052d20","unresolved":false,"context_lines":[{"line_number":215,"context_line":"From a biased point of view, I am going to suggest using `zookeeper`_ as it"},{"line_number":216,"context_line":"has the best developed python client library and patterns (via `kazoo`_), most"},{"line_number":217,"context_line":"robust support for upgrades (via rolling restarts), scale out features such"},{"line_number":218,"context_line":"as read-only servers), documentation and is already used in a variety of"},{"line_number":219,"context_line":"existing opensouce projects (mesos, neo4j, hbase, mapreduce"},{"line_number":220,"context_line":"and `more`_). It also is one of the few services that actually was proven to"},{"line_number":221,"context_line":"work as expected by the jepsen testing framework. It is **not** a perfect"}],"source_content_type":"text/x-rst","patch_set":22,"id":"9a1a9d01_4a111103","line":218,"updated":"2015-09-30 14:08:44.000000000","message":"You\u0027re missing a paired opening (.","commit_id":"5e331be6c564925ed8d9271e461aa199601f7925"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"902b0268130f60e09305961676685cf0ef052d20","unresolved":false,"context_lines":[{"line_number":216,"context_line":"has the best developed python client library and patterns (via `kazoo`_), most"},{"line_number":217,"context_line":"robust support for upgrades (via rolling restarts), scale out features such"},{"line_number":218,"context_line":"as read-only servers), documentation and is already used in a variety of"},{"line_number":219,"context_line":"existing opensouce projects (mesos, neo4j, hbase, mapreduce"},{"line_number":220,"context_line":"and `more`_). It also is one of the few services that actually was proven to"},{"line_number":221,"context_line":"work as expected by the jepsen testing framework. It is **not** a perfect"},{"line_number":222,"context_line":"solution (none of the above will be) but it can be (and IMHO should be) a"}],"source_content_type":"text/x-rst","patch_set":22,"id":"9a1a9d01_ea234554","line":219,"updated":"2015-09-30 14:08:44.000000000","message":"s/opensouce/opensource","commit_id":"5e331be6c564925ed8d9271e461aa199601f7925"},{"author":{"_account_id":6159,"name":"Flavio Percoco Premoli","display_name":"flaper87","email":"flavio.percoco@flyrlabs.com","username":"flaper87"},"change_message_id":"cc2fde53c6c35ffdaf98f1831ed4f29bf5230f61","unresolved":false,"context_lines":[{"line_number":4,"context_line":""},{"line_number":5,"context_line":"No blueprint, this is intended as a reference/consensus document."},{"line_number":6,"context_line":""},{"line_number":7,"context_line":"The various OpenStack projects (and most distributed system projects that"},{"line_number":8,"context_line":"do meaningful things) have an ongoing requirement to perform"},{"line_number":9,"context_line":"some set of actions in an atomic manner performed by some distributed set of"},{"line_number":10,"context_line":"applications on some set of distributed resources **without** having those"}],"source_content_type":"text/x-rst","patch_set":29,"id":"7a2fa921_be1684a5","line":7,"updated":"2015-10-13 08:45:52.000000000","message":"could you remove these parentheses? I don\u0027t think that\u0027s useful for the spec.","commit_id":"ff04f7d317336b2fd8baa92c210455f751eb6c50"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"6375f7d265be738248fe42dda553869e4056b681","unresolved":false,"context_lines":[{"line_number":4,"context_line":""},{"line_number":5,"context_line":"No blueprint, this is intended as a reference/consensus document."},{"line_number":6,"context_line":""},{"line_number":7,"context_line":"The various OpenStack projects (and most distributed system projects that"},{"line_number":8,"context_line":"do meaningful things) have an ongoing requirement to perform"},{"line_number":9,"context_line":"some set of actions in an atomic manner performed by some distributed set of"},{"line_number":10,"context_line":"applications on some set of distributed resources **without** having those"}],"source_content_type":"text/x-rst","patch_set":29,"id":"5a2ca52d_45f8d1e5","line":7,"in_reply_to":"7a2fa921_be1684a5","updated":"2015-10-14 00:29:45.000000000","message":"Done","commit_id":"ff04f7d317336b2fd8baa92c210455f751eb6c50"},{"author":{"_account_id":6159,"name":"Flavio Percoco Premoli","display_name":"flaper87","email":"flavio.percoco@flyrlabs.com","username":"flaper87"},"change_message_id":"cc2fde53c6c35ffdaf98f1831ed4f29bf5230f61","unresolved":false,"context_lines":[{"line_number":73,"context_line":"---------------------"},{"line_number":74,"context_line":""},{"line_number":75,"context_line":"To show the current diversity let\u0027s dive slightly into what *some* of the"},{"line_number":76,"context_line":"projects (in no particular order, and if some are missing or the"},{"line_number":77,"context_line":"information is not accurate please do let me know) have created and/or used"},{"line_number":78,"context_line":"to resolve the problems mentioned above."},{"line_number":79,"context_line":""}],"source_content_type":"text/x-rst","patch_set":29,"id":"7a2fa921_5e0bb07d","line":76,"updated":"2015-10-13 08:45:52.000000000","message":"these parentheses is not really needed. It\u0027ll appear in the final spec. Lets let people use reviews to comment and mention missing cases.","commit_id":"ff04f7d317336b2fd8baa92c210455f751eb6c50"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"a5346af64c2cbf185a4ee99fc64595cf9732fb57","unresolved":false,"context_lines":[{"line_number":73,"context_line":"---------------------"},{"line_number":74,"context_line":""},{"line_number":75,"context_line":"To show the current diversity let\u0027s dive slightly into what *some* of the"},{"line_number":76,"context_line":"projects (in no particular order, and if some are missing or the"},{"line_number":77,"context_line":"information is not accurate please do let me know) have created and/or used"},{"line_number":78,"context_line":"to resolve the problems mentioned above."},{"line_number":79,"context_line":""}],"source_content_type":"text/x-rst","patch_set":29,"id":"5a2ca52d_22826b01","line":76,"in_reply_to":"7a2fa921_5e0bb07d","updated":"2015-10-14 00:05:18.000000000","message":"Okie dokie.","commit_id":"ff04f7d317336b2fd8baa92c210455f751eb6c50"},{"author":{"_account_id":6786,"name":"Sergey Lukjanov","email":"me@slukjanov.name","username":"slukjanov"},"change_message_id":"3b435a541c6dc22034cebca8e6eeec5f3ef09d30","unresolved":false,"context_lines":[{"line_number":188,"context_line":"**Code/doc references:**"},{"line_number":189,"context_line":""},{"line_number":190,"context_line":"- http://docs.openstack.org/developer/heat/_modules/heat/engine/stack_lock.html"},{"line_number":191,"context_line":"- https://github.com/openstack/heat/blob/master/heat/engine/resource.py#L1307"},{"line_number":192,"context_line":""},{"line_number":193,"context_line":"Solution analysis"},{"line_number":194,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":29,"id":"5a2ca52d_5568acba","line":191,"updated":"2015-10-13 09:48:42.000000000","message":"Current and potential usages in other OpenStack projects:\n\n* Ceilometer - distributing tasks across central agents (token ring based on tooz)\n* Sahara - the same as Ceilometer (need to run multiple periodic checks and etc.) and Heat (need to work with clusters in parallel from different engines)","commit_id":"ff04f7d317336b2fd8baa92c210455f751eb6c50"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"a5346af64c2cbf185a4ee99fc64595cf9732fb57","unresolved":false,"context_lines":[{"line_number":188,"context_line":"**Code/doc references:**"},{"line_number":189,"context_line":""},{"line_number":190,"context_line":"- http://docs.openstack.org/developer/heat/_modules/heat/engine/stack_lock.html"},{"line_number":191,"context_line":"- https://github.com/openstack/heat/blob/master/heat/engine/resource.py#L1307"},{"line_number":192,"context_line":""},{"line_number":193,"context_line":"Solution analysis"},{"line_number":194,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":29,"id":"5a2ca52d_a2767b3c","line":191,"in_reply_to":"5a2ca52d_5568acba","updated":"2015-10-14 00:05:18.000000000","message":"Cool, thx, will include.","commit_id":"ff04f7d317336b2fd8baa92c210455f751eb6c50"},{"author":{"_account_id":6159,"name":"Flavio Percoco Premoli","display_name":"flaper87","email":"flavio.percoco@flyrlabs.com","username":"flaper87"},"change_message_id":"cc2fde53c6c35ffdaf98f1831ed4f29bf5230f61","unresolved":false,"context_lines":[{"line_number":223,"context_line":"| Consul    |         |     |         |                        |             |"},{"line_number":224,"context_line":"+-----------+---------+-----+---------+------------------------+-------------+"},{"line_number":225,"context_line":"| Redis     |         |     |         |                        |             |"},{"line_number":226,"context_line":"+-----------+---------+-----+---------+------------------------+-------------+"},{"line_number":227,"context_line":""},{"line_number":228,"context_line":"Proposed change"},{"line_number":229,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":29,"id":"7a2fa921_7e10eca8","line":226,"updated":"2015-10-13 08:45:52.000000000","message":"I think this table could use an OPs-friendly column","commit_id":"ff04f7d317336b2fd8baa92c210455f751eb6c50"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"5c2eebd73dde27a23c13c0de5a998cce9363b17c","unresolved":false,"context_lines":[{"line_number":223,"context_line":"| Consul    |         |     |         |                        |             |"},{"line_number":224,"context_line":"+-----------+---------+-----+---------+------------------------+-------------+"},{"line_number":225,"context_line":"| Redis     |         |     |         |                        |             |"},{"line_number":226,"context_line":"+-----------+---------+-----+---------+------------------------+-------------+"},{"line_number":227,"context_line":""},{"line_number":228,"context_line":"Proposed change"},{"line_number":229,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":29,"id":"5a2ca52d_6212a3b6","line":226,"in_reply_to":"5a2ca52d_82949f37","updated":"2015-10-14 00:07:35.000000000","message":"I already think this table is gonna be hard to get right, so I might get rid of it, hard to fit anything useful in a table.","commit_id":"ff04f7d317336b2fd8baa92c210455f751eb6c50"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"a5346af64c2cbf185a4ee99fc64595cf9732fb57","unresolved":false,"context_lines":[{"line_number":223,"context_line":"| Consul    |         |     |         |                        |             |"},{"line_number":224,"context_line":"+-----------+---------+-----+---------+------------------------+-------------+"},{"line_number":225,"context_line":"| Redis     |         |     |         |                        |             |"},{"line_number":226,"context_line":"+-----------+---------+-----+---------+------------------------+-------------+"},{"line_number":227,"context_line":""},{"line_number":228,"context_line":"Proposed change"},{"line_number":229,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":29,"id":"5a2ca52d_82949f37","line":226,"in_reply_to":"7a2fa921_7e10eca8","updated":"2015-10-14 00:05:18.000000000","message":"How does one fill in a value for that column? Seems like that might be somewhat arbitrary :-P","commit_id":"ff04f7d317336b2fd8baa92c210455f751eb6c50"},{"author":{"_account_id":6786,"name":"Sergey Lukjanov","email":"me@slukjanov.name","username":"slukjanov"},"change_message_id":"3b435a541c6dc22034cebca8e6eeec5f3ef09d30","unresolved":false,"context_lines":[{"line_number":233,"context_line":"the author of this specification is a core member on), most"},{"line_number":234,"context_line":"robust support for upgrades (via rolling restarts), scale out features such"},{"line_number":235,"context_line":"as read-only servers), documentation and is already used in a variety of"},{"line_number":236,"context_line":"existing opensouce projects (mesos, neo4j, hbase, mapreduce"},{"line_number":237,"context_line":"and `more`_). It also is one of the few services that actually was proven to"},{"line_number":238,"context_line":"work as expected by the jepsen testing framework. It is **not** a perfect"},{"line_number":239,"context_line":"solution (none of the above will be) but it can be (and IMHO should be) a"}],"source_content_type":"text/x-rst","patch_set":29,"id":"7a2fa921_34fff335","line":236,"updated":"2015-10-13 09:48:42.000000000","message":"I would say there are few other interesting usages - Solr, Kafka. It\u0027s better to say Hadoop MarReduce.","commit_id":"ff04f7d317336b2fd8baa92c210455f751eb6c50"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"a5346af64c2cbf185a4ee99fc64595cf9732fb57","unresolved":false,"context_lines":[{"line_number":233,"context_line":"the author of this specification is a core member on), most"},{"line_number":234,"context_line":"robust support for upgrades (via rolling restarts), scale out features such"},{"line_number":235,"context_line":"as read-only servers), documentation and is already used in a variety of"},{"line_number":236,"context_line":"existing opensouce projects (mesos, neo4j, hbase, mapreduce"},{"line_number":237,"context_line":"and `more`_). It also is one of the few services that actually was proven to"},{"line_number":238,"context_line":"work as expected by the jepsen testing framework. It is **not** a perfect"},{"line_number":239,"context_line":"solution (none of the above will be) but it can be (and IMHO should be) a"}],"source_content_type":"text/x-rst","patch_set":29,"id":"5a2ca52d_c2aa9770","line":236,"in_reply_to":"7a2fa921_34fff335","updated":"2015-10-14 00:05:18.000000000","message":"Will add and change.","commit_id":"ff04f7d317336b2fd8baa92c210455f751eb6c50"},{"author":{"_account_id":6786,"name":"Sergey Lukjanov","email":"me@slukjanov.name","username":"slukjanov"},"change_message_id":"3b435a541c6dc22034cebca8e6eeec5f3ef09d30","unresolved":false,"context_lines":[{"line_number":246,"context_line":""},{"line_number":247,"context_line":"A few other reasons why its likely to prefer zookeeper here as well:"},{"line_number":248,"context_line":""},{"line_number":249,"context_line":"#. `Sahara`_ already needs/uses(?) zookeeper"},{"line_number":250,"context_line":"   for `namenode high availability`_."},{"line_number":251,"context_line":"#. Service groups in nova already have a zookeeper driver; although"},{"line_number":252,"context_line":"   work is underway to refactor this (see `servicegroup-refactoring`_ and"}],"source_content_type":"text/x-rst","patch_set":29,"id":"7a2fa921_f4a65b52","line":249,"updated":"2015-10-13 09:48:42.000000000","message":"In Sahara we\u0027re deploying Zookeeper using Cloudera Manager or Ambari (Hadoop stack provisioning \u0026 cluster managers). It\u0027s needed for Hadoop HA implementation. In addition to it it\u0027s used by this this managers too.","commit_id":"ff04f7d317336b2fd8baa92c210455f751eb6c50"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"a5346af64c2cbf185a4ee99fc64595cf9732fb57","unresolved":false,"context_lines":[{"line_number":246,"context_line":""},{"line_number":247,"context_line":"A few other reasons why its likely to prefer zookeeper here as well:"},{"line_number":248,"context_line":""},{"line_number":249,"context_line":"#. `Sahara`_ already needs/uses(?) zookeeper"},{"line_number":250,"context_line":"   for `namenode high availability`_."},{"line_number":251,"context_line":"#. Service groups in nova already have a zookeeper driver; although"},{"line_number":252,"context_line":"   work is underway to refactor this (see `servicegroup-refactoring`_ and"}],"source_content_type":"text/x-rst","patch_set":29,"id":"5a2ca52d_e2ca334f","line":249,"in_reply_to":"7a2fa921_f4a65b52","updated":"2015-10-14 00:05:18.000000000","message":"Cool, will update, thought it would have been needed just wasn\u0027t 100% sure.","commit_id":"ff04f7d317336b2fd8baa92c210455f751eb6c50"},{"author":{"_account_id":6159,"name":"Flavio Percoco Premoli","display_name":"flaper87","email":"flavio.percoco@flyrlabs.com","username":"flaper87"},"change_message_id":"cc2fde53c6c35ffdaf98f1831ed4f29bf5230f61","unresolved":false,"context_lines":[{"line_number":292,"context_line":".. _namenode high availability: https://hadoop.apache.org/docs/current/\\"},{"line_number":293,"context_line":"                                hadoop-project-dist/hadoop-hdfs/\\"},{"line_number":294,"context_line":"                                HDFSHighAvailabilityWithNFS.html"},{"line_number":295,"context_line":""},{"line_number":296,"context_line":"Alternatives"},{"line_number":297,"context_line":"------------"},{"line_number":298,"context_line":""}],"source_content_type":"text/x-rst","patch_set":29,"id":"7a2fa921_1e269816","line":295,"updated":"2015-10-13 08:45:52.000000000","message":"All great points, thanks!","commit_id":"ff04f7d317336b2fd8baa92c210455f751eb6c50"},{"author":{"_account_id":6786,"name":"Sergey Lukjanov","email":"me@slukjanov.name","username":"slukjanov"},"change_message_id":"3b435a541c6dc22034cebca8e6eeec5f3ef09d30","unresolved":false,"context_lines":[{"line_number":296,"context_line":"Alternatives"},{"line_number":297,"context_line":"------------"},{"line_number":298,"context_line":""},{"line_number":299,"context_line":"Place all functionality behind `tooz`_ (as much as possible) and let the"},{"line_number":300,"context_line":"operator choose which implementation to use (do note that functionality that is"},{"line_number":301,"context_line":"not possible in all backends will not be able to be exposed through a tooz"},{"line_number":302,"context_line":"API, so this may limit the developer using `tooz`_ to implement some"}],"source_content_type":"text/x-rst","patch_set":29,"id":"7a2fa921_7e18e975","line":299,"updated":"2015-10-13 09:48:42.000000000","message":"IMO the DLM solution shouldn\u0027t be pluggable to use all benefits of the chosen system. Any upper layer libs will make the generic part smaller and merge all cons of the different solutions capabilities.","commit_id":"ff04f7d317336b2fd8baa92c210455f751eb6c50"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"a5346af64c2cbf185a4ee99fc64595cf9732fb57","unresolved":false,"context_lines":[{"line_number":296,"context_line":"Alternatives"},{"line_number":297,"context_line":"------------"},{"line_number":298,"context_line":""},{"line_number":299,"context_line":"Place all functionality behind `tooz`_ (as much as possible) and let the"},{"line_number":300,"context_line":"operator choose which implementation to use (do note that functionality that is"},{"line_number":301,"context_line":"not possible in all backends will not be able to be exposed through a tooz"},{"line_number":302,"context_line":"API, so this may limit the developer using `tooz`_ to implement some"}],"source_content_type":"text/x-rst","patch_set":29,"id":"5a2ca52d_82e25fd4","line":299,"in_reply_to":"7a2fa921_7e18e975","updated":"2015-10-14 00:05:18.000000000","message":"Yup, this is the abstraction *problem* in a nutshell.","commit_id":"ff04f7d317336b2fd8baa92c210455f751eb6c50"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"1ddc2b8f1886dce94bad005f6c1ad04e43a37a0d","unresolved":false,"context_lines":[{"line_number":102,"context_line":""},{"line_number":103,"context_line":"**Problem:**"},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"Avoid multiple conductors from manipulating the same bare-metal"},{"line_number":106,"context_line":"instances and/or nodes at the same time while still being scalable and"},{"line_number":107,"context_line":"highly available."},{"line_number":108,"context_line":""}],"source_content_type":"text/x-rst","patch_set":36,"id":"3a29b11f_9fce1f82","line":105,"updated":"2015-10-27 05:57:29.000000000","message":"couple more things that might be worth mentioning:\n\n* track what services are running, supporting what drivers, and rebalance work when service state changes\n* sync state of temporary agents instead of polling or heartbeats","commit_id":"d94d979c1e19ee1dc5a18bf699230b4725293b24"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"23b0e590b35eb9d35711e435636400ba62fd662a","unresolved":false,"context_lines":[{"line_number":102,"context_line":""},{"line_number":103,"context_line":"**Problem:**"},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"Avoid multiple conductors from manipulating the same bare-metal"},{"line_number":106,"context_line":"instances and/or nodes at the same time while still being scalable and"},{"line_number":107,"context_line":"highly available."},{"line_number":108,"context_line":""}],"source_content_type":"text/x-rst","patch_set":36,"id":"1a26ad4f_3339d2a4","line":105,"in_reply_to":"3a29b11f_9fce1f82","updated":"2015-11-03 01:40:41.000000000","message":"can u explain \u0027sync state of temporary agents instead of polling or heartbeats\u0027\n\nI\u0027m unsure exactly what that means? :)","commit_id":"d94d979c1e19ee1dc5a18bf699230b4725293b24"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"1ddc2b8f1886dce94bad005f6c1ad04e43a37a0d","unresolved":false,"context_lines":[{"line_number":118,"context_line":""},{"line_number":119,"context_line":"**Notes:**"},{"line_number":120,"context_line":""},{"line_number":121,"context_line":"- Has logic for shared and exclusive locks and provisions for upgrading"},{"line_number":122,"context_line":"  a shared lock to an exclusive lock (I\u0027m unsure how this works, usually"},{"line_number":123,"context_line":"  upgrading a shared lock to an exclusive one means kicking out all other"},{"line_number":124,"context_line":"  active readers, what happens if there are multiple readers doing this"}],"source_content_type":"text/x-rst","patch_set":36,"id":"3a29b11f_9f95bf41","line":121,"updated":"2015-10-27 05:57:29.000000000","message":"shared and exclusive locks can co-exist; only one exclusive lock allowed at a time\n\nupgrading a shared to exclusive lock fails if another exclusive lock exists","commit_id":"d94d979c1e19ee1dc5a18bf699230b4725293b24"},{"author":{"_account_id":10239,"name":"Dmitry Tantsur","email":"dtantsur@protonmail.com","username":"dtantsur"},"change_message_id":"82f843f2f46a8a59179b45e4f6840d20eb274deb","unresolved":false,"context_lines":[{"line_number":118,"context_line":""},{"line_number":119,"context_line":"**Notes:**"},{"line_number":120,"context_line":""},{"line_number":121,"context_line":"- Has logic for shared and exclusive locks and provisions for upgrading"},{"line_number":122,"context_line":"  a shared lock to an exclusive lock (I\u0027m unsure how this works, usually"},{"line_number":123,"context_line":"  upgrading a shared lock to an exclusive one means kicking out all other"},{"line_number":124,"context_line":"  active readers, what happens if there are multiple readers doing this"}],"source_content_type":"text/x-rst","patch_set":36,"id":"da85f559_34f43559","line":121,"in_reply_to":"1a26ad4f_7332eaf3","updated":"2015-11-12 13:44:30.000000000","message":"\u003e if a shared (equivalent to a reader lock?) lock exists on key X, how can a exclusive lock also exist on key X\n\nin our implementation - it can, but it\u0027s wrong IMO","commit_id":"d94d979c1e19ee1dc5a18bf699230b4725293b24"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"fd5b5b74b592fe2973eb96d9a4b1eadb94a2d123","unresolved":false,"context_lines":[{"line_number":118,"context_line":""},{"line_number":119,"context_line":"**Notes:**"},{"line_number":120,"context_line":""},{"line_number":121,"context_line":"- Has logic for shared and exclusive locks and provisions for upgrading"},{"line_number":122,"context_line":"  a shared lock to an exclusive lock (I\u0027m unsure how this works, usually"},{"line_number":123,"context_line":"  upgrading a shared lock to an exclusive one means kicking out all other"},{"line_number":124,"context_line":"  active readers, what happens if there are multiple readers doing this"}],"source_content_type":"text/x-rst","patch_set":36,"id":"1a26ad4f_7332eaf3","line":121,"in_reply_to":"3a29b11f_9f95bf41","updated":"2015-11-03 01:32:53.000000000","message":"really? that seems sorta odd,\n\nif a shared (equivalent to a reader lock?) lock exists on key X, how can a exclusive lock also exist on key X? unless I\u0027m confused there shouldn\u0027t be a shared lock and an exclusive lock co-existing on the same key at the same time (isn\u0027t that the basic tenant of shared/read-writer locks? aka that multiple readers can exist but not multiple writers and no multiple readers while a writer is active)","commit_id":"d94d979c1e19ee1dc5a18bf699230b4725293b24"},{"author":{"_account_id":6488,"name":"Clint Byrum","email":"clint@fewbar.com","username":"clint-fewbar"},"change_message_id":"fc70c1bb197b3357c2ef037ffe4dcc65849afc16","unresolved":false,"context_lines":[{"line_number":118,"context_line":""},{"line_number":119,"context_line":"**Notes:**"},{"line_number":120,"context_line":""},{"line_number":121,"context_line":"- Has logic for shared and exclusive locks and provisions for upgrading"},{"line_number":122,"context_line":"  a shared lock to an exclusive lock (I\u0027m unsure how this works, usually"},{"line_number":123,"context_line":"  upgrading a shared lock to an exclusive one means kicking out all other"},{"line_number":124,"context_line":"  active readers, what happens if there are multiple readers doing this"}],"source_content_type":"text/x-rst","patch_set":36,"id":"da85f559_b6a6d70c","line":121,"in_reply_to":"da85f559_34f43559","updated":"2015-11-12 18:43:14.000000000","message":"The model Devananda describes, which I trust he knows is the model implemented in Ironic, is most like the SQL READ COMMITTED isolation model. You never get data that changes during a transaction, because you have a shared lock (many readers on that version of the database), but new transactions that change things will upgrade to an exclusive lock to change the data and create a new version without affecting existing readers. New transactions trying to acquire the shared lock will block until the exclusive lock is released. Using this model requires that code accepts MVCC, including rollbacks. This is fine for leader election as long as it\u0027s ok for members to read a stale leader and they don\u0027t ever trust a shared lock to self-identify as leader, but it may not be so good for something else in OpenStack, since it may not be possible for us to rollback something like a volume delete from a SAN (this is a big reason why SQL is not a great choice for a backend btw).\n\nSERIALIZABLE describes more like what Josh suggests. If there are shared locks at all, then the new exclusive lock will block until they are released. This way if you ran the same sequence of reads and writes again, they\u0027d result in the same external view. This is more like what I think of when I say shared/exclusive locks too, but both are valid since the terms are so generic. Note that this mode still doesn\u0027t sidestep MVCC issues, especially if you try to use Galera in multi-active-master mode.\n\nHowever, _NEITHER_ mode \"kicks out\" shared readers that already have the lock. That\u0027s simply not possible. The exclusive upgrader must wait for all shared locks to release, or use MVCC.","commit_id":"d94d979c1e19ee1dc5a18bf699230b4725293b24"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"1a14455bc1fc10fe2df28c712f27d0ba13a520e2","unresolved":false,"context_lines":[{"line_number":118,"context_line":""},{"line_number":119,"context_line":"**Notes:**"},{"line_number":120,"context_line":""},{"line_number":121,"context_line":"- Has logic for shared and exclusive locks and provisions for upgrading"},{"line_number":122,"context_line":"  a shared lock to an exclusive lock (I\u0027m unsure how this works, usually"},{"line_number":123,"context_line":"  upgrading a shared lock to an exclusive one means kicking out all other"},{"line_number":124,"context_line":"  active readers, what happens if there are multiple readers doing this"}],"source_content_type":"text/x-rst","patch_set":36,"id":"da85f559_dada6217","line":121,"in_reply_to":"da85f559_b6a6d70c","updated":"2015-11-13 19:13:54.000000000","message":"Thanks for the explanation, thats a great description of the different meanings that confused me here, much appreciated :)\n\nTakeaway from this is that I start to feel we should prefer the classic description (SERIALIZABLE) and go with that, because afaik that\u0027s what https://github.com/python-zk/kazoo/pull/306 was aiming for (I don\u0027t think such a concept/pattern even exists yet in consul/others)... because unless we back locks by a capable MVCC system it doesn\u0027t seem like the other description is even possible (?)\n\nSERIALIZABLE is also more what https://fasteners.readthedocs.org/en/latest/api/lock.html#fasteners.lock.ReaderWriterLock does as well, and various projects in openstack use that (indirectly via oslo.concurrency or directly through that library)","commit_id":"d94d979c1e19ee1dc5a18bf699230b4725293b24"},{"author":{"_account_id":2889,"name":"Aeva Black","email":"aeva.online@gmail.com","username":"tenbrae"},"change_message_id":"1ddc2b8f1886dce94bad005f6c1ad04e43a37a0d","unresolved":false,"context_lines":[{"line_number":123,"context_line":"  upgrading a shared lock to an exclusive one means kicking out all other"},{"line_number":124,"context_line":"  active readers, what happens if there are multiple readers doing this"},{"line_number":125,"context_line":"  at the same time?)."},{"line_number":126,"context_line":"- Reclaim/take over lock mechanism? (unsure exactly how this works, manual?)"},{"line_number":127,"context_line":""},{"line_number":128,"context_line":"**Code/doc references:**"},{"line_number":129,"context_line":""}],"source_content_type":"text/x-rst","patch_set":36,"id":"3a29b11f_5f76473f","line":126,"updated":"2015-10-27 05:57:29.000000000","message":"service heartbeats are used to identify dead locks, but releasing those is manual and clunky","commit_id":"d94d979c1e19ee1dc5a18bf699230b4725293b24"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"98c9e9843fdaeb5a0a9c62f700327d449faaeaad","unresolved":false,"context_lines":[{"line_number":25,"context_line":".. _locking: https://en.wikipedia.org/wiki/Lock_%28computer_science%29"},{"line_number":26,"context_line":""},{"line_number":27,"context_line":"Problem description"},{"line_number":28,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":29,"context_line":""},{"line_number":30,"context_line":"Building distributed systems is **hard**. It is especially hard when the"},{"line_number":31,"context_line":"distributed system (and the applications ``[X, Y, Z...]`` that compose the"}],"source_content_type":"text/x-rst","patch_set":37,"id":"1a26ad4f_ab956133","line":28,"updated":"2015-11-03 09:35:32.000000000","message":"There is another huge problem. System tests for distributed components and clearly documented failure modes in API dev docs. Oslo.messaging is a great anti-example here: RPC failure modes are not documented; there is no HA testing for existing failure modes and consistency claims.\n\nWe should address that issue for this DLM initiative from the very beginning. And picking a testing framework like jepsen or https://github.com/dcm-oss/blockade or anything else is should be done in this scope as well.","commit_id":"5f3c085241d34e40e9296b92e6e169602519da78"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"1bcde5ecb811fba3eccb87257ced2f065f3a4aac","unresolved":false,"context_lines":[{"line_number":25,"context_line":".. _locking: https://en.wikipedia.org/wiki/Lock_%28computer_science%29"},{"line_number":26,"context_line":""},{"line_number":27,"context_line":"Problem description"},{"line_number":28,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":29,"context_line":""},{"line_number":30,"context_line":"Building distributed systems is **hard**. It is especially hard when the"},{"line_number":31,"context_line":"distributed system (and the applications ``[X, Y, Z...]`` that compose the"}],"source_content_type":"text/x-rst","patch_set":37,"id":"fa80f949_bb4b022b","line":28,"in_reply_to":"1a26ad4f_ab956133","updated":"2015-11-03 19:42:48.000000000","message":"Understood, I think we do need to do better here and try to define a clear contract of what the drivers in tooz should and must do (and a way to verify that they do work as expected).\n\nThe issue I have is that there is varying levels of what is acceptable, it starts to enter the CAP theroum area, and we as a community likely don\u0027t need 100% consistent data stores (we may be ok with eventually consistent for certain scenarios) so starting off with contract that isn\u0027t what people actually want (and one that reduces availability or partition tolerance) may actually be more problematic than good.\n\nBut yes, I agree overall that we need to be very clear about the contract, although I\"m not sure this document should try to define that (it is already big enough); but perhaps a follow up document should... Want to work on that?","commit_id":"5f3c085241d34e40e9296b92e6e169602519da78"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"df2a1036fd149f4c354a44fbc7a4058d978e1b43","unresolved":false,"context_lines":[{"line_number":25,"context_line":".. _locking: https://en.wikipedia.org/wiki/Lock_%28computer_science%29"},{"line_number":26,"context_line":""},{"line_number":27,"context_line":"Problem description"},{"line_number":28,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":29,"context_line":""},{"line_number":30,"context_line":"Building distributed systems is **hard**. It is especially hard when the"},{"line_number":31,"context_line":"distributed system (and the applications ``[X, Y, Z...]`` that compose the"}],"source_content_type":"text/x-rst","patch_set":37,"id":"fa80f949_dc7eebe3","line":28,"in_reply_to":"fa80f949_bb4b022b","updated":"2015-11-04 09:37:41.000000000","message":"Accepting this spec in development without the contract would make yet another Oslo.messaging UX, where RPC sessions might likely get disrupted in the case of an  AMQP failover in the middle. Would we want to have the same for the DLM solution? IMO, the contract is a must have item here before we could start with the dev driven by this specs.\n\nThe contract may be as simple as possible. Like just to describe claims of the DLM devs vs the DLM API users\u0027 expectations in OpenStack projects. As an API calls, timeouts, and returned results, nothing more.\n\nWhile integration testing and the framework to be used for that, may be put in the separate spec as well, indeed.","commit_id":"5f3c085241d34e40e9296b92e6e169602519da78"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"abee406cddbe2f86f91b7f7dc0230e5c9104896a","unresolved":false,"context_lines":[{"line_number":25,"context_line":".. _locking: https://en.wikipedia.org/wiki/Lock_%28computer_science%29"},{"line_number":26,"context_line":""},{"line_number":27,"context_line":"Problem description"},{"line_number":28,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":29,"context_line":""},{"line_number":30,"context_line":"Building distributed systems is **hard**. It is especially hard when the"},{"line_number":31,"context_line":"distributed system (and the applications ``[X, Y, Z...]`` that compose the"}],"source_content_type":"text/x-rst","patch_set":37,"id":"da85f559_6b963d96","line":28,"in_reply_to":"fa80f949_dc7eebe3","updated":"2015-11-07 00:49:01.000000000","message":"I think https://review.openstack.org/#/c/240645/ exists to help here.","commit_id":"5f3c085241d34e40e9296b92e6e169602519da78"}]}