)]}'
{"specs/mitaka/at-least-once-guarantee.rst":[{"author":{"_account_id":13290,"name":"Oleksii Zamiatin","email":"ozamiatin@mirantis.com","username":"ozamiatin"},"change_message_id":"ea1b624b556262140d0758f28733964d18b37303","unresolved":false,"context_lines":[{"line_number":25,"context_line":"duplicated replies or messages being processed twice, which is highly unlikely"},{"line_number":26,"context_line":"for rpc pattern."},{"line_number":27,"context_line":""},{"line_number":28,"context_line":".. [#] Note that the guarantees are not strong. Duplicates are still possible."},{"line_number":29,"context_line":""},{"line_number":30,"context_line":"Proposed change"},{"line_number":31,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":1,"id":"7a740942_24b3069c","line":28,"updated":"2015-12-11 11:26:08.000000000","message":"As far as we have \u0027retry\u0027 parameter in oslo.messaging configured for infinite retries by default, we can\u0027t claim it as pure at most once delivery.\n\nThere was also a summit discussion that we need to provide some spec where to clarify existing patterns and their level of reliability and what they guarantee and what they don\u0027t.","commit_id":"bbe4b28b5b31e34c3d5fe4fafab46abecf32acab"},{"author":{"_account_id":2813,"name":"Mehdi Abaakouk","email":"sileht@sileht.net","username":"sileht"},"change_message_id":"78b42fe6303d89ff0902f9ade665640b17a9c2bc","unresolved":false,"context_lines":[{"line_number":25,"context_line":"duplicated replies or messages being processed twice, which is highly unlikely"},{"line_number":26,"context_line":"for rpc pattern."},{"line_number":27,"context_line":""},{"line_number":28,"context_line":".. [#] Note that the guarantees are not strong. Duplicates are still possible."},{"line_number":29,"context_line":""},{"line_number":30,"context_line":"Proposed change"},{"line_number":31,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":1,"id":"7a740942_a8a104fa","line":28,"in_reply_to":"7a740942_24b3069c","updated":"2015-12-11 13:54:37.000000000","message":"This retry parameter was introduced for a Ceilometer use case to be able to move it to oslo.messaging, but now Ceilometer doesn\u0027t use that anymore and have move to the notification system.\n\nSo perhaps we can\u0027t deprecated it. Anyways this is unrelated to this specs.","commit_id":"bbe4b28b5b31e34c3d5fe4fafab46abecf32acab"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"133b1ba52830217c81affee284f18f383d3ccae4","unresolved":false,"context_lines":[{"line_number":25,"context_line":"duplicated replies or messages being processed twice, which is highly unlikely"},{"line_number":26,"context_line":"for rpc pattern."},{"line_number":27,"context_line":""},{"line_number":28,"context_line":".. [#] Note that the guarantees are not strong. Duplicates are still possible."},{"line_number":29,"context_line":""},{"line_number":30,"context_line":"Proposed change"},{"line_number":31,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":1,"id":"7a740942_f414c5e9","line":28,"in_reply_to":"7a740942_39256808","updated":"2015-12-11 14:37:10.000000000","message":"Yes, a bit out of scope but I\u0027d like to keep this comment here for less numbers of eyebrow-raised and oh-really? readers","commit_id":"bbe4b28b5b31e34c3d5fe4fafab46abecf32acab"},{"author":{"_account_id":8601,"name":"Dmitriy Ukhlov","email":"dukhlov@mirantis.com","username":"dukhlov"},"change_message_id":"fd03ad2da2e4a9bdfe98456fcf2812aa5922041c","unresolved":false,"context_lines":[{"line_number":25,"context_line":"duplicated replies or messages being processed twice, which is highly unlikely"},{"line_number":26,"context_line":"for rpc pattern."},{"line_number":27,"context_line":""},{"line_number":28,"context_line":".. [#] Note that the guarantees are not strong. Duplicates are still possible."},{"line_number":29,"context_line":""},{"line_number":30,"context_line":"Proposed change"},{"line_number":31,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":1,"id":"7a740942_39256808","line":28,"in_reply_to":"7a740942_a8a104fa","updated":"2015-12-11 14:27:56.000000000","message":"I think that just deprecate retry parameter is not good idea because in this case don\u0027t have at-most-one logic. But it is important for RPC. I think could change retry parameter by idempotent\u003dTrue or False. Because it is usefull to perform retries for idempotent requests and is potentially unsafe for non idempotent request. Also default behavior should be non idempotent for safety and if client logic knows that this request is idempotent it could optimize it and set idempotent\u003dTrue. To enable retrying. I understand that it changes behavior of RPC messaging, and probably could be done only with new API version and migration period. Guys, please comment this. What are your thoughts? Does it make sense for you?","commit_id":"bbe4b28b5b31e34c3d5fe4fafab46abecf32acab"},{"author":{"_account_id":2813,"name":"Mehdi Abaakouk","email":"sileht@sileht.net","username":"sileht"},"change_message_id":"78b42fe6303d89ff0902f9ade665640b17a9c2bc","unresolved":false,"context_lines":[{"line_number":30,"context_line":"Proposed change"},{"line_number":31,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":32,"context_line":""},{"line_number":33,"context_line":"Given that said, the solution is to add new API for a work queue. There should"},{"line_number":34,"context_line":"be new method ``ensure()`` or ``distribute()``, which resembles the"},{"line_number":35,"context_line":"``_dispatch_and_reply`` of the ``RPCDispatcher`` class. But new method should"},{"line_number":36,"context_line":"provide the \"at-least-once\" delivery model and may normally return nothing to"}],"source_content_type":"text/x-rst","patch_set":1,"id":"7a740942_db453c43","line":33,"updated":"2015-12-11 13:54:37.000000000","message":"work queues pattern already exists in oslo.messaging you just have to send a message with server unset in the Target . \n\nIf I get it correctly, the goals here is to allow the application to put message back in the queues when something goes wrong.","commit_id":"bbe4b28b5b31e34c3d5fe4fafab46abecf32acab"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"133b1ba52830217c81affee284f18f383d3ccae4","unresolved":false,"context_lines":[{"line_number":30,"context_line":"Proposed change"},{"line_number":31,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":32,"context_line":""},{"line_number":33,"context_line":"Given that said, the solution is to add new API for a work queue. There should"},{"line_number":34,"context_line":"be new method ``ensure()`` or ``distribute()``, which resembles the"},{"line_number":35,"context_line":"``_dispatch_and_reply`` of the ``RPCDispatcher`` class. But new method should"},{"line_number":36,"context_line":"provide the \"at-least-once\" delivery model and may normally return nothing to"}],"source_content_type":"text/x-rst","patch_set":1,"id":"7a740942_54791981","line":33,"in_reply_to":"7a740942_db453c43","updated":"2015-12-11 14:37:10.000000000","message":"thank you, will fix wording","commit_id":"bbe4b28b5b31e34c3d5fe4fafab46abecf32acab"},{"author":{"_account_id":2813,"name":"Mehdi Abaakouk","email":"sileht@sileht.net","username":"sileht"},"change_message_id":"78b42fe6303d89ff0902f9ade665640b17a9c2bc","unresolved":false,"context_lines":[{"line_number":31,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":32,"context_line":""},{"line_number":33,"context_line":"Given that said, the solution is to add new API for a work queue. There should"},{"line_number":34,"context_line":"be new method ``ensure()`` or ``distribute()``, which resembles the"},{"line_number":35,"context_line":"``_dispatch_and_reply`` of the ``RPCDispatcher`` class. But new method should"},{"line_number":36,"context_line":"provide the \"at-least-once\" delivery model and may normally return nothing to"},{"line_number":37,"context_line":"the caller side. It may also return an error or event that says \"did not"}],"source_content_type":"text/x-rst","patch_set":1,"id":"7a740942_25c3fd8a","line":34,"updated":"2015-12-11 13:54:37.000000000","message":"I like distribute()  Bit I don\u0027t get where this method will live ? in the rpc client ?","commit_id":"bbe4b28b5b31e34c3d5fe4fafab46abecf32acab"},{"author":{"_account_id":13290,"name":"Oleksii Zamiatin","email":"ozamiatin@mirantis.com","username":"ozamiatin"},"change_message_id":"ea1b624b556262140d0758f28733964d18b37303","unresolved":false,"context_lines":[{"line_number":43,"context_line":"arbitrary order. Therefore, the new method must guarantee serializable [#]_"},{"line_number":44,"context_line":"\"at-least-once\" delivery for messages to be distributed."},{"line_number":45,"context_line":""},{"line_number":46,"context_line":"Besides that, the Oslo messaging library should not handle duplicated or lost"},{"line_number":47,"context_line":"or delivered out of order messages to a work queue. That should be the app side"},{"line_number":48,"context_line":"responsibility instead. The only way to handle those properly is to handle at"},{"line_number":49,"context_line":"the app server level in an atomic way  - meaning the app must have a safe way"}],"source_content_type":"text/x-rst","patch_set":1,"id":"7a740942_64890e26","line":46,"updated":"2015-12-11 11:26:08.000000000","message":"Now we have the dispatcher in Oslo messaging which runs method execution. Does this mean that we are going to move dispatching logic to the app-side in the new pattern?","commit_id":"bbe4b28b5b31e34c3d5fe4fafab46abecf32acab"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"133b1ba52830217c81affee284f18f383d3ccae4","unresolved":false,"context_lines":[{"line_number":43,"context_line":"arbitrary order. Therefore, the new method must guarantee serializable [#]_"},{"line_number":44,"context_line":"\"at-least-once\" delivery for messages to be distributed."},{"line_number":45,"context_line":""},{"line_number":46,"context_line":"Besides that, the Oslo messaging library should not handle duplicated or lost"},{"line_number":47,"context_line":"or delivered out of order messages to a work queue. That should be the app side"},{"line_number":48,"context_line":"responsibility instead. The only way to handle those properly is to handle at"},{"line_number":49,"context_line":"the app server level in an atomic way  - meaning the app must have a safe way"}],"source_content_type":"text/x-rst","patch_set":1,"id":"7a740942_59fb947b","line":46,"in_reply_to":"7a740942_64890e26","updated":"2015-12-11 14:37:10.000000000","message":"This is only related to a work queue. Bad wording perhaps? Should I rephrase?","commit_id":"bbe4b28b5b31e34c3d5fe4fafab46abecf32acab"},{"author":{"_account_id":2813,"name":"Mehdi Abaakouk","email":"sileht@sileht.net","username":"sileht"},"change_message_id":"78b42fe6303d89ff0902f9ade665640b17a9c2bc","unresolved":false,"context_lines":[{"line_number":49,"context_line":"the app server level in an atomic way  - meaning the app must have a safe way"},{"line_number":50,"context_line":"to distinguish duplicates for those requests that are not idempotent and nobody"},{"line_number":51,"context_line":"else. The application has to bite the bullet and implement request idempotence"},{"line_number":52,"context_line":"and atomicity properly on the server side."},{"line_number":53,"context_line":""},{"line_number":54,"context_line":".. [#] In distributed systems world, serializable means that the history of"},{"line_number":55,"context_line":"   all operations is equivalent to one that took place in some single atomic"}],"source_content_type":"text/x-rst","patch_set":1,"id":"7a740942_05228152","line":52,"updated":"2015-12-11 13:54:37.000000000","message":"Nice, this paragraph could directly go to the distribute() docstring :)","commit_id":"bbe4b28b5b31e34c3d5fe4fafab46abecf32acab"},{"author":{"_account_id":13290,"name":"Oleksii Zamiatin","email":"ozamiatin@mirantis.com","username":"ozamiatin"},"change_message_id":"ea1b624b556262140d0758f28733964d18b37303","unresolved":false,"context_lines":[{"line_number":68,"context_line":"-----------------------"},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"The API for rpc shall be either reworked to support a work queue pattern,"},{"line_number":71,"context_line":"if the alternatives accepted, or unchanged but extended to provide a separate"},{"line_number":72,"context_line":"API for a work queue by new method ``ensure()`` or ``distribute()``."},{"line_number":73,"context_line":""},{"line_number":74,"context_line":"Security impact"}],"source_content_type":"text/x-rst","patch_set":1,"id":"7a740942_840afa88","line":71,"updated":"2015-12-11 11:26:08.000000000","message":"+1 for extending the API and implementing the new pattern (not mixing it with existing RPC)","commit_id":"bbe4b28b5b31e34c3d5fe4fafab46abecf32acab"},{"author":{"_account_id":2813,"name":"Mehdi Abaakouk","email":"sileht@sileht.net","username":"sileht"},"change_message_id":"78b42fe6303d89ff0902f9ade665640b17a9c2bc","unresolved":false,"context_lines":[{"line_number":113,"context_line":"-----------"},{"line_number":114,"context_line":""},{"line_number":115,"context_line":"Primary assignee:"},{"line_number":116,"context_line":"  \u003clooking-for-volunteers!\u003e"},{"line_number":117,"context_line":""},{"line_number":118,"context_line":"Milestones"},{"line_number":119,"context_line":"----------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"7a740942_db8a1c64","line":116,"updated":"2015-12-11 13:54:37.000000000","message":"Really nobody will work on this ?","commit_id":"bbe4b28b5b31e34c3d5fe4fafab46abecf32acab"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"133b1ba52830217c81affee284f18f383d3ccae4","unresolved":false,"context_lines":[{"line_number":113,"context_line":"-----------"},{"line_number":114,"context_line":""},{"line_number":115,"context_line":"Primary assignee:"},{"line_number":116,"context_line":"  \u003clooking-for-volunteers!\u003e"},{"line_number":117,"context_line":""},{"line_number":118,"context_line":"Milestones"},{"line_number":119,"context_line":"----------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"7a740942_590f1439","line":116,"in_reply_to":"7a740942_db8a1c64","updated":"2015-12-11 14:37:10.000000000","message":"I cannot suggest a primary person, I\u0027d like to but I\u0027m not a python dev :) So I could add myself only as a secondary assignee","commit_id":"bbe4b28b5b31e34c3d5fe4fafab46abecf32acab"},{"author":{"_account_id":11744,"name":"Alec Hothan","email":"ahothan@cisco.com","username":"ahothan"},"change_message_id":"d17f514bb498d56af35b8fd208da2c1e476603a3","unresolved":false,"context_lines":[{"line_number":6,"context_line":""},{"line_number":7,"context_line":"OpenStack projects like Mistral have identified a work queue pattern as a"},{"line_number":8,"context_line":"use case to be supported in Oslo messaging. Historically, Oslo messaging rpc"},{"line_number":9,"context_line":"delivery model is an \"at-most-once\". In order to support a work queue"},{"line_number":10,"context_line":"pattern, it should as well provide an \"at-least-once\" one and expose new API,"},{"line_number":11,"context_line":"which has nothing to the existing API for rpc."},{"line_number":12,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_c8955732","line":9,"updated":"2015-12-11 21:33:38.000000000","message":"ok not really the purpose of this review but even an \u0027at-most-once\u0027 denomination is a stretch because oslo messaging cannot guarantee that under all conditions. I guess this is why the situation is so dire today and perhaps efforts such as this one to add new services that are (hopefully) much better documented would be the only practical and safe way out of this hole.","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"62400892f9ccc263eafa603a18fff9b9676289cb","unresolved":false,"context_lines":[{"line_number":15,"context_line":""},{"line_number":16,"context_line":"At most once delivery ensures the \"ack\" to be send before the processing"},{"line_number":17,"context_line":"(dispatching) of a message and guarantees there would be no duplicates [#]_."},{"line_number":18,"context_line":"At most once delivery model allows messages to disappear, which is highly"},{"line_number":19,"context_line":"unlikely for the work queue pattern there each message is a task for workers."},{"line_number":20,"context_line":"That means that existing API for dispatching rpc can not fit needs of a work"},{"line_number":21,"context_line":"queue."}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_62a7a2a7","line":18,"updated":"2015-12-11 20:25:25.000000000","message":"Highly unlikely or highly likely?","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"0ab9961b9891ea8e1f151028c767ff31011df01d","unresolved":false,"context_lines":[{"line_number":15,"context_line":""},{"line_number":16,"context_line":"At most once delivery ensures the \"ack\" to be send before the processing"},{"line_number":17,"context_line":"(dispatching) of a message and guarantees there would be no duplicates [#]_."},{"line_number":18,"context_line":"At most once delivery model allows messages to disappear, which is highly"},{"line_number":19,"context_line":"unlikely for the work queue pattern there each message is a task for workers."},{"line_number":20,"context_line":"That means that existing API for dispatching rpc can not fit needs of a work"},{"line_number":21,"context_line":"queue."}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_970d2a40","line":18,"in_reply_to":"7a740942_057a68f1","updated":"2015-12-11 23:36:11.000000000","message":"+1 ;)","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"77a2de9fceebbd4ce224287980d9d61edc9366b3","unresolved":false,"context_lines":[{"line_number":15,"context_line":""},{"line_number":16,"context_line":"At most once delivery ensures the \"ack\" to be send before the processing"},{"line_number":17,"context_line":"(dispatching) of a message and guarantees there would be no duplicates [#]_."},{"line_number":18,"context_line":"At most once delivery model allows messages to disappear, which is highly"},{"line_number":19,"context_line":"unlikely for the work queue pattern there each message is a task for workers."},{"line_number":20,"context_line":"That means that existing API for dispatching rpc can not fit needs of a work"},{"line_number":21,"context_line":"queue."}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_057a68f1","line":18,"in_reply_to":"7a740942_62a7a2a7","updated":"2015-12-11 20:55:25.000000000","message":"Argh... I should have put it as highly annoying!","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":11744,"name":"Alec Hothan","email":"ahothan@cisco.com","username":"ahothan"},"change_message_id":"d17f514bb498d56af35b8fd208da2c1e476603a3","unresolved":false,"context_lines":[{"line_number":18,"context_line":"At most once delivery model allows messages to disappear, which is highly"},{"line_number":19,"context_line":"unlikely for the work queue pattern there each message is a task for workers."},{"line_number":20,"context_line":"That means that existing API for dispatching rpc can not fit needs of a work"},{"line_number":21,"context_line":"queue."},{"line_number":22,"context_line":""},{"line_number":23,"context_line":"From the other side, existing API for rpc can not be changed to the"},{"line_number":24,"context_line":"\"at-least-once\" delivery model as that might render rpc workers to return"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_cb78f180","line":21,"updated":"2015-12-11 21:33:38.000000000","message":"The wording is a bit confusing, I\u0027d suggest to not talk about the work queue here and only describe issues with the current rpc service (so perhaps remove the \"which is unlikely for the work queue...\").\nRegarding the \"ack before processing\" and \"at most once\", these are 2 different level, one is an implementation detail while the other is a feature description.\nOn the \"at most once\" paradigm, the only practical use case where that attribute would be desirable is when a request does not have to be executed at all cost when it is sent - with the intent that it does not matter if the message happens to not make it (presumably this could be an optional action or something that the sender can resend later in any shape or form).\nNothing bad about this \"qos\" per se, the main gripe is that many of the usages of such API by openstack users  have not accounted for such semantic and this can cause serious malfunction when things start to unravel (well it is no longer speculative because it is relatively easy to \"break\" openstack with simple failure conditions so we know there is a lot of code using oslo messaging that is not implemented correctly wrt failure conditions).\nRegarding the implementation, an ACK that is sent before processing has pretty much little value for the sender, because it just means the request was received but does not guarantee execution (WTH can the sender do with this ACK anyway than throw it away).","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"62400892f9ccc263eafa603a18fff9b9676289cb","unresolved":false,"context_lines":[{"line_number":22,"context_line":""},{"line_number":23,"context_line":"From the other side, existing API for rpc can not be changed to the"},{"line_number":24,"context_line":"\"at-least-once\" delivery model as that might render rpc workers to return"},{"line_number":25,"context_line":"duplicated replies or messages being processed twice, which is highly unlikely"},{"line_number":26,"context_line":"for rpc pattern."},{"line_number":27,"context_line":""},{"line_number":28,"context_line":"At least once delivery is implemented for notification pattern already but"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_c2e3f6d6","line":25,"updated":"2015-12-11 20:25:25.000000000","message":"Perhaps u mean \u0027highly undesirable\u0027 here?","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"77a2de9fceebbd4ce224287980d9d61edc9366b3","unresolved":false,"context_lines":[{"line_number":22,"context_line":""},{"line_number":23,"context_line":"From the other side, existing API for rpc can not be changed to the"},{"line_number":24,"context_line":"\"at-least-once\" delivery model as that might render rpc workers to return"},{"line_number":25,"context_line":"duplicated replies or messages being processed twice, which is highly unlikely"},{"line_number":26,"context_line":"for rpc pattern."},{"line_number":27,"context_line":""},{"line_number":28,"context_line":"At least once delivery is implemented for notification pattern already but"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_c57f80dd","line":25,"in_reply_to":"7a740942_c2e3f6d6","updated":"2015-12-11 20:55:25.000000000","message":"yes :)","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":11744,"name":"Alec Hothan","email":"ahothan@cisco.com","username":"ahothan"},"change_message_id":"d17f514bb498d56af35b8fd208da2c1e476603a3","unresolved":false,"context_lines":[{"line_number":23,"context_line":"From the other side, existing API for rpc can not be changed to the"},{"line_number":24,"context_line":"\"at-least-once\" delivery model as that might render rpc workers to return"},{"line_number":25,"context_line":"duplicated replies or messages being processed twice, which is highly unlikely"},{"line_number":26,"context_line":"for rpc pattern."},{"line_number":27,"context_line":""},{"line_number":28,"context_line":"At least once delivery is implemented for notification pattern already but"},{"line_number":29,"context_line":"Notification staff can not be used for work queue pattern as is."}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_0ba279ae","line":26,"updated":"2015-12-11 21:33:38.000000000","message":"highly undesirable... still are you even sure that what you paint here as undesirable is actually not possible with the current implementation?\nI would not bet the farm on it.","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"0ab9961b9891ea8e1f151028c767ff31011df01d","unresolved":false,"context_lines":[{"line_number":23,"context_line":"From the other side, existing API for rpc can not be changed to the"},{"line_number":24,"context_line":"\"at-least-once\" delivery model as that might render rpc workers to return"},{"line_number":25,"context_line":"duplicated replies or messages being processed twice, which is highly unlikely"},{"line_number":26,"context_line":"for rpc pattern."},{"line_number":27,"context_line":""},{"line_number":28,"context_line":"At least once delivery is implemented for notification pattern already but"},{"line_number":29,"context_line":"Notification staff can not be used for work queue pattern as is."}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_77299e9a","line":26,"in_reply_to":"7a740942_0ba279ae","updated":"2015-12-11 23:36:11.000000000","message":"Highly undesireable means its not \u0027desired to happen\u0027, I would think that would be the case with RPC (whether or not it actually happens alot is a different question).","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"62400892f9ccc263eafa603a18fff9b9676289cb","unresolved":false,"context_lines":[{"line_number":26,"context_line":"for rpc pattern."},{"line_number":27,"context_line":""},{"line_number":28,"context_line":"At least once delivery is implemented for notification pattern already but"},{"line_number":29,"context_line":"Notification staff can not be used for work queue pattern as is."},{"line_number":30,"context_line":""},{"line_number":31,"context_line":"The work queue pattern may be used in Oslo messaging by sending a message"},{"line_number":32,"context_line":"with the server value unset in the ``Target``. But it doesn\u0027t fit the"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_42d8661e","line":29,"updated":"2015-12-11 20:25:25.000000000","message":"Staff?","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"0ab9961b9891ea8e1f151028c767ff31011df01d","unresolved":false,"context_lines":[{"line_number":26,"context_line":"for rpc pattern."},{"line_number":27,"context_line":""},{"line_number":28,"context_line":"At least once delivery is implemented for notification pattern already but"},{"line_number":29,"context_line":"Notification staff can not be used for work queue pattern as is."},{"line_number":30,"context_line":""},{"line_number":31,"context_line":"The work queue pattern may be used in Oslo messaging by sending a message"},{"line_number":32,"context_line":"with the server value unset in the ``Target``. But it doesn\u0027t fit the"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_e9fff2fe","line":29,"in_reply_to":"7a740942_25668451","updated":"2015-12-11 23:36:11.000000000","message":"Ya, staff seems odd, was it supposed to be stuff, idk","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"77a2de9fceebbd4ce224287980d9d61edc9366b3","unresolved":false,"context_lines":[{"line_number":26,"context_line":"for rpc pattern."},{"line_number":27,"context_line":""},{"line_number":28,"context_line":"At least once delivery is implemented for notification pattern already but"},{"line_number":29,"context_line":"Notification staff can not be used for work queue pattern as is."},{"line_number":30,"context_line":""},{"line_number":31,"context_line":"The work queue pattern may be used in Oslo messaging by sending a message"},{"line_number":32,"context_line":"with the server value unset in the ``Target``. But it doesn\u0027t fit the"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_25668451","line":29,"in_reply_to":"7a740942_42d8661e","updated":"2015-12-11 20:55:25.000000000","message":"should change it to notifications pattern maybe?","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":8601,"name":"Dmitriy Ukhlov","email":"dukhlov@mirantis.com","username":"dukhlov"},"change_message_id":"2d14c39819c21389d05de37c2f0d1f2301861b71","unresolved":false,"context_lines":[{"line_number":39,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":40,"context_line":""},{"line_number":41,"context_line":"Given that said, the solution is to add new API for a work queue. There should"},{"line_number":42,"context_line":"be new method ``ensure()`` or ``distribute()``, which resembles the"},{"line_number":43,"context_line":"``_dispatch_and_reply`` of the ``RPCDispatcher`` class. But new method should"},{"line_number":44,"context_line":"provide the \"at-least-once\" delivery model and may normally return nothing to"},{"line_number":45,"context_line":"the caller side. It may also return an error or event that says \"did not"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_32049a48","line":42,"updated":"2015-12-11 15:54:55.000000000","message":"Could you provide more concrete and formal description of proposed change? It is not clear now what are these ``ensure()`` or ``distribute()`` methods for? but please start from user\u0027s API like:\n1) send_work_item(Target, WorkItem) - sends work item to target, returns immediately after delivery confirmation, if this method doesn\u0027t raise any exception, processing of this work item should be guaranteed at least once\n2) ???\n\nits is my current understanding what are you proposing. But I see some mentions about returned result to clients about work item status\n\nplease provide more details to clarify these points.","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"77a2de9fceebbd4ce224287980d9d61edc9366b3","unresolved":false,"context_lines":[{"line_number":39,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":40,"context_line":""},{"line_number":41,"context_line":"Given that said, the solution is to add new API for a work queue. There should"},{"line_number":42,"context_line":"be new method ``ensure()`` or ``distribute()``, which resembles the"},{"line_number":43,"context_line":"``_dispatch_and_reply`` of the ``RPCDispatcher`` class. But new method should"},{"line_number":44,"context_line":"provide the \"at-least-once\" delivery model and may normally return nothing to"},{"line_number":45,"context_line":"the caller side. It may also return an error or event that says \"did not"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_c540e061","line":42,"in_reply_to":"7a740942_32049a48","updated":"2015-12-11 20:55:25.000000000","message":"Let\u0027s think of it altogether please. Perhaps we want only push queues semantics? Maybe this https://cloud.google.com/appengine/docs/python/taskqueue/overview-push can bring us more ideas on that topic.","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"fd19992926e8bd367a6cf265e8c79f4621053e76","unresolved":false,"context_lines":[{"line_number":39,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":40,"context_line":""},{"line_number":41,"context_line":"Given that said, the solution is to add new API for a work queue. There should"},{"line_number":42,"context_line":"be new method ``ensure()`` or ``distribute()``, which resembles the"},{"line_number":43,"context_line":"``_dispatch_and_reply`` of the ``RPCDispatcher`` class. But new method should"},{"line_number":44,"context_line":"provide the \"at-least-once\" delivery model and may normally return nothing to"},{"line_number":45,"context_line":"the caller side. It may also return an error or event that says \"did not"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_0b327910","line":42,"in_reply_to":"7a740942_c540e061","updated":"2015-12-11 21:46:26.000000000","message":"Maybe the workqueue_task_{create,distribute,wait,delete}() is that we want for the user API?","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"39293629aa08158718a3102e52024f476c5130a2","unresolved":false,"context_lines":[{"line_number":62,"context_line":".. [#] In distributed systems world, serializable means that the history of"},{"line_number":63,"context_line":"   all operations is equivalent to one that took place in some single atomic"},{"line_number":64,"context_line":"   order but with unknown invocation and completion times - it places no bounds"},{"line_number":65,"context_line":"   on time or order."},{"line_number":66,"context_line":""},{"line_number":67,"context_line":"In fact there are two layers - messaging driver API and Oslo messaging user API."},{"line_number":68,"context_line":"The second one should have patterns to solve the work queue use case. On the"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_e5b5dcec","line":65,"updated":"2015-12-11 20:36:27.000000000","message":"Confused here a little, one statement says \u0027single atomic order\u0027 then later is says \u0027no bounds on time or order\u0027 ;)","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":11744,"name":"Alec Hothan","email":"ahothan@cisco.com","username":"ahothan"},"change_message_id":"29d162a7f50e99912008307e182f464adfa73b5c","unresolved":false,"context_lines":[{"line_number":62,"context_line":".. [#] In distributed systems world, serializable means that the history of"},{"line_number":63,"context_line":"   all operations is equivalent to one that took place in some single atomic"},{"line_number":64,"context_line":"   order but with unknown invocation and completion times - it places no bounds"},{"line_number":65,"context_line":"   on time or order."},{"line_number":66,"context_line":""},{"line_number":67,"context_line":"In fact there are two layers - messaging driver API and Oslo messaging user API."},{"line_number":68,"context_line":"The second one should have patterns to solve the work queue use case. On the"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_f19a7e61","line":65,"in_reply_to":"7a740942_68178345","updated":"2015-12-11 22:43:49.000000000","message":"Honestly not sure what you want to say here?\nSerialization can mean a lot of things at different layers. Do you mean that the timing of the sequence of messages (spacing between message arrival) and their order of delivery is  not guaranteed and thus the user must design its code accordingly?\nIn the above section, you mention \"no strong guarantee\", possible \"loss\", \"duplicate\" or out of order messages, and \"therefore the new method must guarantee serializable at least once delivery\", really don\u0027t see how these (what is before and after the therefore) are related in any way.\nIn general serialization guarantee would mean keeping things in order, so the choice of wording may not be best.","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"77a2de9fceebbd4ce224287980d9d61edc9366b3","unresolved":false,"context_lines":[{"line_number":62,"context_line":".. [#] In distributed systems world, serializable means that the history of"},{"line_number":63,"context_line":"   all operations is equivalent to one that took place in some single atomic"},{"line_number":64,"context_line":"   order but with unknown invocation and completion times - it places no bounds"},{"line_number":65,"context_line":"   on time or order."},{"line_number":66,"context_line":""},{"line_number":67,"context_line":"In fact there are two layers - messaging driver API and Oslo messaging user API."},{"line_number":68,"context_line":"The second one should have patterns to solve the work queue use case. On the"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_68178345","line":65,"in_reply_to":"7a740942_e5b5dcec","updated":"2015-12-11 20:55:25.000000000","message":"perhaps I should rephrase that this is about *expected* order of the operations. While the single atomic operation defines the *actual* order?","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"e53c0222ba7b013209569f7b75041c44db50e229","unresolved":false,"context_lines":[{"line_number":62,"context_line":".. [#] In distributed systems world, serializable means that the history of"},{"line_number":63,"context_line":"   all operations is equivalent to one that took place in some single atomic"},{"line_number":64,"context_line":"   order but with unknown invocation and completion times - it places no bounds"},{"line_number":65,"context_line":"   on time or order."},{"line_number":66,"context_line":""},{"line_number":67,"context_line":"In fact there are two layers - messaging driver API and Oslo messaging user API."},{"line_number":68,"context_line":"The second one should have patterns to solve the work queue use case. On the"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_ce0d5490","line":65,"in_reply_to":"7a740942_f19a7e61","updated":"2015-12-12 09:02:17.000000000","message":"I followed the terminology described here https://aphyr.com/posts/313-strong-consistency-models\nI cannot find any references to existing academic publications, though I believe the serializable means exactly that is described - no ordering  constraints, only start/stop events of each operation does matter.","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":2813,"name":"Mehdi Abaakouk","email":"sileht@sileht.net","username":"sileht"},"change_message_id":"9c9088eba722682f67b85f677ca24927b462b3e4","unresolved":false,"context_lines":[{"line_number":67,"context_line":"In fact there are two layers - messaging driver API and Oslo messaging user API."},{"line_number":68,"context_line":"The second one should have patterns to solve the work queue use case. On the"},{"line_number":69,"context_line":"driver layer shall be low level patterns for \"at-least-once\" and \"at-most-once\"."},{"line_number":70,"context_line":""},{"line_number":71,"context_line":"Alternatives"},{"line_number":72,"context_line":"------------"},{"line_number":73,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_f3d93158","line":70,"updated":"2015-12-11 17:35:22.000000000","message":"Something is missing in this specs:\n\nImagine you send a message to a worker, the message trigger a bug into the worker, an unexpected exception is raised.\n\nWhat should we do with the message ? ack or requeue ?\n\nAnother scenario if the unexpected message crash your worker.\n\nThe message will be requeued automatically.\n\nIn these cases if the crash is due to how the worker handles the message, your queues will grow until your fix the bug and put the fix in production.\n\nHow do you plan to handle such case ?","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"ed9befb0a7b3f502d8b6d9f8b1fcd14f7973d33e","unresolved":false,"context_lines":[{"line_number":67,"context_line":"In fact there are two layers - messaging driver API and Oslo messaging user API."},{"line_number":68,"context_line":"The second one should have patterns to solve the work queue use case. On the"},{"line_number":69,"context_line":"driver layer shall be low level patterns for \"at-least-once\" and \"at-most-once\"."},{"line_number":70,"context_line":""},{"line_number":71,"context_line":"Alternatives"},{"line_number":72,"context_line":"------------"},{"line_number":73,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_88cf4f7e","line":70,"in_reply_to":"7a740942_85d2d8fe","updated":"2015-12-11 21:01:12.000000000","message":"If we don\u0027t want retries, we could use configurable timeouts passed both in the driver and user APIs as well.","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":11744,"name":"Alec Hothan","email":"ahothan@cisco.com","username":"ahothan"},"change_message_id":"29d162a7f50e99912008307e182f464adfa73b5c","unresolved":false,"context_lines":[{"line_number":67,"context_line":"In fact there are two layers - messaging driver API and Oslo messaging user API."},{"line_number":68,"context_line":"The second one should have patterns to solve the work queue use case. On the"},{"line_number":69,"context_line":"driver layer shall be low level patterns for \"at-least-once\" and \"at-most-once\"."},{"line_number":70,"context_line":""},{"line_number":71,"context_line":"Alternatives"},{"line_number":72,"context_line":"------------"},{"line_number":73,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_d1b2e295","line":70,"in_reply_to":"7a740942_88cf4f7e","updated":"2015-12-11 22:43:49.000000000","message":"Above you say \"the Oslo messaging library should not handle duplicated or lost\nor delivered out of order messages to a work queue. That should be the app side\nresponsibility instead.\"\nHere you seem to say the opposite since you suggest to pass a timeout value to the user API?","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"e53c0222ba7b013209569f7b75041c44db50e229","unresolved":false,"context_lines":[{"line_number":67,"context_line":"In fact there are two layers - messaging driver API and Oslo messaging user API."},{"line_number":68,"context_line":"The second one should have patterns to solve the work queue use case. On the"},{"line_number":69,"context_line":"driver layer shall be low level patterns for \"at-least-once\" and \"at-most-once\"."},{"line_number":70,"context_line":""},{"line_number":71,"context_line":"Alternatives"},{"line_number":72,"context_line":"------------"},{"line_number":73,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_ee7930fd","line":70,"in_reply_to":"7a740942_d1b2e295","updated":"2015-12-12 09:02:17.000000000","message":"I\u0027m not sure how timeouts, which are just the execution contract for a request, are in opposite here. User only asks to follow the contract deadline and nothing more, abort if violated.","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":11744,"name":"Alec Hothan","email":"ahothan@cisco.com","username":"ahothan"},"change_message_id":"008d7fc110e28846c1eeb3a005a63638831f2be6","unresolved":false,"context_lines":[{"line_number":67,"context_line":"In fact there are two layers - messaging driver API and Oslo messaging user API."},{"line_number":68,"context_line":"The second one should have patterns to solve the work queue use case. On the"},{"line_number":69,"context_line":"driver layer shall be low level patterns for \"at-least-once\" and \"at-most-once\"."},{"line_number":70,"context_line":""},{"line_number":71,"context_line":"Alternatives"},{"line_number":72,"context_line":"------------"},{"line_number":73,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_368780b6","line":70,"in_reply_to":"7a740942_ee7930fd","updated":"2015-12-13 18:55:05.000000000","message":"In the proposed work queue service, can you describe explicitly how ACKS are used and what is their semantic for the sender side.\nFor example do you limit to ACK or do you also support NACK, does the sender gets notified asynchronously about an ACK/NACK?\nClearly delimitate what happens at each layer and the interface between these layers (and ultimately how does that map on the transport layer: rabbitMQ, ZMQ...).","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"facfe8f53bfa97061ce2a6548f90dc90c556eedf","unresolved":false,"context_lines":[{"line_number":67,"context_line":"In fact there are two layers - messaging driver API and Oslo messaging user API."},{"line_number":68,"context_line":"The second one should have patterns to solve the work queue use case. On the"},{"line_number":69,"context_line":"driver layer shall be low level patterns for \"at-least-once\" and \"at-most-once\"."},{"line_number":70,"context_line":""},{"line_number":71,"context_line":"Alternatives"},{"line_number":72,"context_line":"------------"},{"line_number":73,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_85d2d8fe","line":70,"in_reply_to":"7a740942_f3d93158","updated":"2015-12-11 20:40:02.000000000","message":"I believe on the driver API level, the worker would have to nack the message back on to the queue if it can\u0027t handle it for a given timeout or if it raised an exception. On the user API level, the requested task - which cannot be executed due to exceptions ping-ponging the task across workers - must fail after a given amount of retries done by the task distributor.","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"62400892f9ccc263eafa603a18fff9b9676289cb","unresolved":false,"context_lines":[{"line_number":71,"context_line":"Alternatives"},{"line_number":72,"context_line":"------------"},{"line_number":73,"context_line":""},{"line_number":74,"context_line":"* Ensure stronger delivery claims, like sequential."},{"line_number":75,"context_line":"* Provide duplicates handling on the Oslo.messaging side"},{"line_number":76,"context_line":"* Rework existing API for rpc to support a work queue with at-least-once"},{"line_number":77,"context_line":"  delivery as well as rpc with at-most-once one."}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_a2526a65","line":74,"updated":"2015-12-11 20:25:25.000000000","message":"I wonder how this would work, seems like not so easy...","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"ed9befb0a7b3f502d8b6d9f8b1fcd14f7973d33e","unresolved":false,"context_lines":[{"line_number":71,"context_line":"Alternatives"},{"line_number":72,"context_line":"------------"},{"line_number":73,"context_line":""},{"line_number":74,"context_line":"* Ensure stronger delivery claims, like sequential."},{"line_number":75,"context_line":"* Provide duplicates handling on the Oslo.messaging side"},{"line_number":76,"context_line":"* Rework existing API for rpc to support a work queue with at-least-once"},{"line_number":77,"context_line":"  delivery as well as rpc with at-most-once one."}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_e827f360","line":74,"in_reply_to":"7a740942_287adb88","updated":"2015-12-11 21:01:12.000000000","message":"Opps, I mean the serializable case. And stronger guarantees would require much more efforts for us, indeed.","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"77a2de9fceebbd4ce224287980d9d61edc9366b3","unresolved":false,"context_lines":[{"line_number":71,"context_line":"Alternatives"},{"line_number":72,"context_line":"------------"},{"line_number":73,"context_line":""},{"line_number":74,"context_line":"* Ensure stronger delivery claims, like sequential."},{"line_number":75,"context_line":"* Provide duplicates handling on the Oslo.messaging side"},{"line_number":76,"context_line":"* Rework existing API for rpc to support a work queue with at-least-once"},{"line_number":77,"context_line":"  delivery as well as rpc with at-most-once one."}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_287adb88","line":74,"in_reply_to":"7a740942_a2526a65","updated":"2015-12-11 20:55:25.000000000","message":"That would work for commutative and idempotent tasks only perhaps.","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":2813,"name":"Mehdi Abaakouk","email":"sileht@sileht.net","username":"sileht"},"change_message_id":"d3646039bcd20c0b36ee9bd5a0d7d84a898cd987","unresolved":false,"context_lines":[{"line_number":127,"context_line":"-----------"},{"line_number":128,"context_line":""},{"line_number":129,"context_line":"Primary assignee:"},{"line_number":130,"context_line":"  \u003clooking-for-volunteers!\u003e"},{"line_number":131,"context_line":""},{"line_number":132,"context_line":"Milestones"},{"line_number":133,"context_line":"----------"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_1bd0991e","line":130,"updated":"2015-12-11 16:39:13.000000000","message":"This is a serious issue if nobody take the lead of this. And when I said lead, I\u0027m not talking about just writing the code today, but also maintains that into the next years...","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":8601,"name":"Dmitriy Ukhlov","email":"dukhlov@mirantis.com","username":"dukhlov"},"change_message_id":"f1addb98257a24e78718428aabc8ddbb58486338","unresolved":false,"context_lines":[{"line_number":127,"context_line":"-----------"},{"line_number":128,"context_line":""},{"line_number":129,"context_line":"Primary assignee:"},{"line_number":130,"context_line":"  \u003clooking-for-volunteers!\u003e"},{"line_number":131,"context_line":""},{"line_number":132,"context_line":"Milestones"},{"line_number":133,"context_line":"----------"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_d30675cd","line":130,"in_reply_to":"7a740942_1bd0991e","updated":"2015-12-11 17:35:14.000000000","message":"I could take it when i finish pika driver work, If no objections","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"77a2de9fceebbd4ce224287980d9d61edc9366b3","unresolved":false,"context_lines":[{"line_number":127,"context_line":"-----------"},{"line_number":128,"context_line":""},{"line_number":129,"context_line":"Primary assignee:"},{"line_number":130,"context_line":"  \u003clooking-for-volunteers!\u003e"},{"line_number":131,"context_line":""},{"line_number":132,"context_line":"Milestones"},{"line_number":133,"context_line":"----------"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a740942_48b6473f","line":130,"in_reply_to":"7a740942_d30675cd","updated":"2015-12-11 20:55:25.000000000","message":"thank you Dmitry! I will update this then.","commit_id":"0945a49b9c06d1414ccc4f3bc80f08cf42f5c12a"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"904d83970b7592242797a1f55e4dfb324013cbf1","unresolved":false,"context_lines":[{"line_number":39,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":40,"context_line":""},{"line_number":41,"context_line":"Given that said, the solution is to add new API for a work queue. There should"},{"line_number":42,"context_line":"be new classess WQClient with create/submit/wait and WQDispatcher and its"},{"line_number":43,"context_line":"methods like ``distribute()`` [#]_, which resembles the ``_dispatch_and_reply``"},{"line_number":44,"context_line":"of the RPCDispatcher class. But new method should provide the \"at-least-once\""},{"line_number":45,"context_line":"delivery model and may normally return nothing. It may also return an error or"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_0440ba55","line":42,"updated":"2015-12-15 00:38:03.000000000","message":"So just a reference point, the closest api that I know of that looks like this (but isn\u0027t this is):\n\nhttp://docs.openstack.org/developer/taskflow/jobs.html#taskflow.jobs.base.JobBoard\n\nBasically that API is the following:\n\n    iterjobs()\n    \"\"\"Returns an iterator of jobs that are currently on this board.\"\"\"\n\n    wait()\n    \"\"\"Waits a given amount of time for jobs to be posted.\"\"\"\n\n    consume(job, who)\n    \"\"\"Permanently (and atomically) removes a job from the jobboard.\"\"\"\n\n    post(name, book\u003dNone, details\u003dNone)\n    \"\"Atomically creates and posts a job to the jobboard.\"\"\n\n    claim(job, who)\n    \"\"Atomically attempts to claim the provided job.\"\"\n\n    abandon(job, who)\n    \"\"\"Atomically attempts to abandon the provided job.\"\"\"\n\nThe above is not based on a MQ underlying implementation (instead zookeeper is preferred for these \u0027jobs\u0027); something to think about/reference when building whatever this new API is...","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":8601,"name":"Dmitriy Ukhlov","email":"dukhlov@mirantis.com","username":"dukhlov"},"change_message_id":"63c71583f1406c73213c6c858e52b15fef14d543","unresolved":false,"context_lines":[{"line_number":39,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":40,"context_line":""},{"line_number":41,"context_line":"Given that said, the solution is to add new API for a work queue. There should"},{"line_number":42,"context_line":"be new classess WQClient with create/submit/wait and WQDispatcher and its"},{"line_number":43,"context_line":"methods like ``distribute()`` [#]_, which resembles the ``_dispatch_and_reply``"},{"line_number":44,"context_line":"of the RPCDispatcher class. But new method should provide the \"at-least-once\""},{"line_number":45,"context_line":"delivery model and may normally return nothing. It may also return an error or"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_da7efdad","line":42,"in_reply_to":"5a710552_0440ba55","updated":"2015-12-15 09:12:46.000000000","message":"Joshua, it is another usecase - it is task flow, when each task has a current state and you can perform some action for task and change its state. For this pattern we need DB for storing state at least and it is not connected with oslo.messaging I guess. This specification about sheduling some async tast execution. I think its API should looks like:\n  \n   send_job(target, job_params, timeout)\n   \"\"\"Send job_params message to one of job workers defined by target,\n   Timeout for sending our job and getting ack that it was accepted\n   \"\"\"\n\nthat\u0027s it. \n\nAlso methods for getting job status and job cancelation are usefull but we can not implement them without some synchronized storage, like SQL DB. So If we want to have such methods I suggest to implement it on high level library, based on messaging and db libraries.","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"e0df6d0c86b2961360e09e40f3f6773d4c3e12ea","unresolved":false,"context_lines":[{"line_number":39,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":40,"context_line":""},{"line_number":41,"context_line":"Given that said, the solution is to add new API for a work queue. There should"},{"line_number":42,"context_line":"be new classess WQClient with create/submit/wait and WQDispatcher and its"},{"line_number":43,"context_line":"methods like ``distribute()`` [#]_, which resembles the ``_dispatch_and_reply``"},{"line_number":44,"context_line":"of the RPCDispatcher class. But new method should provide the \"at-least-once\""},{"line_number":45,"context_line":"delivery model and may normally return nothing. It may also return an error or"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_918fd7cb","line":42,"in_reply_to":"5a710552_663ad038","updated":"2015-12-15 19:12:55.000000000","message":"Those are just minuscule details imho, the key components of it are iterjobs() [or itertasks or whatever u want to call it] and post() and claim() and consume() [abandon() is nice also]. \n\nSo just thinking out loud, but is what mistral/others want really more this API, or do they really need the lower level send_job() api (which is basically the post() + a wait until completed API from the jobboard stuffs)....","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":8601,"name":"Dmitriy Ukhlov","email":"dukhlov@mirantis.com","username":"dukhlov"},"change_message_id":"36ce343d600d29937ffa9f0bddfd113992e0d6b8","unresolved":false,"context_lines":[{"line_number":39,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":40,"context_line":""},{"line_number":41,"context_line":"Given that said, the solution is to add new API for a work queue. There should"},{"line_number":42,"context_line":"be new classess WQClient with create/submit/wait and WQDispatcher and its"},{"line_number":43,"context_line":"methods like ``distribute()`` [#]_, which resembles the ``_dispatch_and_reply``"},{"line_number":44,"context_line":"of the RPCDispatcher class. But new method should provide the \"at-least-once\""},{"line_number":45,"context_line":"delivery model and may normally return nothing. It may also return an error or"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_663ad038","line":42,"in_reply_to":"5a710552_da7efdad","updated":"2015-12-15 09:47:08.000000000","message":"Sorry, I have not understand Joshua\u0027s API first. Yes it is WorkQueue, but it still needs some storage for direct access by task name to get status or remove job. I guess we need to remove such possibility and implement it via messaging. Or implement full Joshua\u0027s API but in another project and use Database for storage","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"904d83970b7592242797a1f55e4dfb324013cbf1","unresolved":false,"context_lines":[{"line_number":48,"context_line":"by the server and processed by a worker."},{"line_number":49,"context_line":""},{"line_number":50,"context_line":"API shall give no strong guarantees for messages delivery. Messages"},{"line_number":51,"context_line":"may be lost in distributed systems. As well as duplicated or delivered in an"},{"line_number":52,"context_line":"arbitrary order. Therefore, the new method must guarantee serializable [#]_"},{"line_number":53,"context_line":"\"at-least-once\" delivery for messages to be distributed."},{"line_number":54,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_64065e22","line":51,"updated":"2015-12-15 00:38:03.000000000","message":"The lack of ordering will be interesting, at least zookeeper has some kind of sequencing ability, I hope its not really needed...","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":8601,"name":"Dmitriy Ukhlov","email":"dukhlov@mirantis.com","username":"dukhlov"},"change_message_id":"63c71583f1406c73213c6c858e52b15fef14d543","unresolved":false,"context_lines":[{"line_number":48,"context_line":"by the server and processed by a worker."},{"line_number":49,"context_line":""},{"line_number":50,"context_line":"API shall give no strong guarantees for messages delivery. Messages"},{"line_number":51,"context_line":"may be lost in distributed systems. As well as duplicated or delivered in an"},{"line_number":52,"context_line":"arbitrary order. Therefore, the new method must guarantee serializable [#]_"},{"line_number":53,"context_line":"\"at-least-once\" delivery for messages to be distributed."},{"line_number":54,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_85a1f6ab","line":51,"in_reply_to":"5a710552_64065e22","updated":"2015-12-15 09:12:46.000000000","message":"I think API should provide strong guarantees for messages delivery at-lease-once. Sure mesasge could be lost in distributed system, but if we say that API does not provide strong guarantees, it means that application logic should take care about message delivery, because for work queue it is important to process all scheduled jobs. I think that use case of WorkQueue pattern is to shedule async task execution and be sure that it will be executed sometime. So yes it is possible to lose the message but API implementation should do all possible to avoid it, and in case of message lost, recovery might be required manually\n\nI agree that lack of ordering is not a problem for this pattern","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"b8522a87ee4e1ca82a6e7dfb4fe22978258271ba","unresolved":false,"context_lines":[{"line_number":48,"context_line":"by the server and processed by a worker."},{"line_number":49,"context_line":""},{"line_number":50,"context_line":"API shall give no strong guarantees for messages delivery. Messages"},{"line_number":51,"context_line":"may be lost in distributed systems. As well as duplicated or delivered in an"},{"line_number":52,"context_line":"arbitrary order. Therefore, the new method must guarantee serializable [#]_"},{"line_number":53,"context_line":"\"at-least-once\" delivery for messages to be distributed."},{"line_number":54,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_8531f60d","line":51,"in_reply_to":"5a710552_64065e22","updated":"2015-12-15 09:01:25.000000000","message":"Then workers failed, unprocessed messages would be returned to queues out-of-order. I see no possible ways to retain the original order for such cases.","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"e0df6d0c86b2961360e09e40f3f6773d4c3e12ea","unresolved":false,"context_lines":[{"line_number":48,"context_line":"by the server and processed by a worker."},{"line_number":49,"context_line":""},{"line_number":50,"context_line":"API shall give no strong guarantees for messages delivery. Messages"},{"line_number":51,"context_line":"may be lost in distributed systems. As well as duplicated or delivered in an"},{"line_number":52,"context_line":"arbitrary order. Therefore, the new method must guarantee serializable [#]_"},{"line_number":53,"context_line":"\"at-least-once\" delivery for messages to be distributed."},{"line_number":54,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_b1b1b391","line":51,"in_reply_to":"5a710552_8531f60d","updated":"2015-12-15 19:12:55.000000000","message":"http://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#Sequence+Nodes+--+Unique+Naming (one such way).","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"e7065c307240ed2998234ca776b4a9d82564ab76","unresolved":false,"context_lines":[{"line_number":48,"context_line":"by the server and processed by a worker."},{"line_number":49,"context_line":""},{"line_number":50,"context_line":"API shall give no strong guarantees for messages delivery. Messages"},{"line_number":51,"context_line":"may be lost in distributed systems. As well as duplicated or delivered in an"},{"line_number":52,"context_line":"arbitrary order. Therefore, the new method must guarantee serializable [#]_"},{"line_number":53,"context_line":"\"at-least-once\" delivery for messages to be distributed."},{"line_number":54,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_9d814a92","line":51,"in_reply_to":"5a710552_85a1f6ab","updated":"2015-12-15 12:10:24.000000000","message":"\u003e I think API should provide strong guarantees for messages delivery at-lease-once.\n\nI kind of agree with this - we may want at timeout that signals that the task _may_ have been lost and it\u0027s up to the app to try again, but in the general case we want to provide delivery some time in the future.","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"e7065c307240ed2998234ca776b4a9d82564ab76","unresolved":false,"context_lines":[{"line_number":53,"context_line":"\"at-least-once\" delivery for messages to be distributed."},{"line_number":54,"context_line":""},{"line_number":55,"context_line":"Oslo messaging library must abide execution contracts but it should not handle"},{"line_number":56,"context_line":"duplicated or lost or delivered out of order messages to a work queue. That"},{"line_number":57,"context_line":"should be the app side responsibility instead. The only way to handle those"},{"line_number":58,"context_line":"properly is to handle at the app server level in an atomic way - meaning the"},{"line_number":59,"context_line":"app must have a safe way to distinguish duplicates for those requests that are"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_502669bb","line":56,"range":{"start_line":56,"start_character":0,"end_line":56,"end_character":53},"updated":"2015-12-15 12:10:24.000000000","message":"Duplicate or ooo is fine, but lost? As discussed above - the whole point is to have \"at least once\" no?","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":11744,"name":"Alec Hothan","email":"ahothan@cisco.com","username":"ahothan"},"change_message_id":"38fca4322195d67f94ed3c69cb4391f9c596841e","unresolved":false,"context_lines":[{"line_number":53,"context_line":"\"at-least-once\" delivery for messages to be distributed."},{"line_number":54,"context_line":""},{"line_number":55,"context_line":"Oslo messaging library must abide execution contracts but it should not handle"},{"line_number":56,"context_line":"duplicated or lost or delivered out of order messages to a work queue. That"},{"line_number":57,"context_line":"should be the app side responsibility instead. The only way to handle those"},{"line_number":58,"context_line":"properly is to handle at the app server level in an atomic way - meaning the"},{"line_number":59,"context_line":"app must have a safe way to distinguish duplicates for those requests that are"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_cd4e2ac1","line":56,"in_reply_to":"5a710552_22bcaa04","updated":"2015-12-15 16:25:14.000000000","message":"There seems to be a mix of terms used in this document, can we standardize? \"message\" and \"task\" seem to be used interchangeably.","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"f1a9ed4dd7d17fec11bbdee58caad22de5896200","unresolved":false,"context_lines":[{"line_number":53,"context_line":"\"at-least-once\" delivery for messages to be distributed."},{"line_number":54,"context_line":""},{"line_number":55,"context_line":"Oslo messaging library must abide execution contracts but it should not handle"},{"line_number":56,"context_line":"duplicated or lost or delivered out of order messages to a work queue. That"},{"line_number":57,"context_line":"should be the app side responsibility instead. The only way to handle those"},{"line_number":58,"context_line":"properly is to handle at the app server level in an atomic way - meaning the"},{"line_number":59,"context_line":"app must have a safe way to distinguish duplicates for those requests that are"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_22bcaa04","line":56,"in_reply_to":"5a710552_502669bb","updated":"2015-12-15 12:54:10.000000000","message":"In the face of a crash-stop failure, a task may be lost undetected by the library logic for WQ, generally speaking. But otherwise, processing contracts shall guarantee at least detection of this case, at most - no lost tasks at all.","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":11744,"name":"Alec Hothan","email":"ahothan@cisco.com","username":"ahothan"},"change_message_id":"38fca4322195d67f94ed3c69cb4391f9c596841e","unresolved":false,"context_lines":[{"line_number":54,"context_line":""},{"line_number":55,"context_line":"Oslo messaging library must abide execution contracts but it should not handle"},{"line_number":56,"context_line":"duplicated or lost or delivered out of order messages to a work queue. That"},{"line_number":57,"context_line":"should be the app side responsibility instead. The only way to handle those"},{"line_number":58,"context_line":"properly is to handle at the app server level in an atomic way - meaning the"},{"line_number":59,"context_line":"app must have a safe way to distinguish duplicates for those requests that are"},{"line_number":60,"context_line":"not idempotent and nobody else. The application has to bite the bullet and"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_cd814a91","line":57,"updated":"2015-12-15 16:25:14.000000000","message":"guarantee of detection of message loss... presumably the client side library would use a timer for ACK reception and this is true as long as the client process does not crash? (otherwise you would need to persist across restarts which is much more difficult).","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"e7065c307240ed2998234ca776b4a9d82564ab76","unresolved":false,"context_lines":[{"line_number":64,"context_line":"   was omitted here. Both client and server are using IPC while workers are"},{"line_number":65,"context_line":"   communicating to the server via AMQP broker."},{"line_number":66,"context_line":""},{"line_number":67,"context_line":".. [#] In distributed systems world, serializable means that the history of"},{"line_number":68,"context_line":"   all operations is equivalent to one that took place in some single atomic"},{"line_number":69,"context_line":"   order but with unknown invocation and completion times - it places no bounds"},{"line_number":70,"context_line":"   on time or order."},{"line_number":71,"context_line":""},{"line_number":72,"context_line":"In fact there are two layers - messaging driver API and Oslo messaging user API."},{"line_number":73,"context_line":"The second one should have patterns to solve the work queue use case."}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_c9e4f03c","line":70,"range":{"start_line":67,"start_character":7,"end_line":70,"end_character":20},"updated":"2015-12-15 12:10:24.000000000","message":"Usually serializable is used when talking about transactions (batches of operation) and means they don\u0027t interleave but can happen in any order.\n\nRegardless of that semantic quirk - \"at least once\" immediately means it\u0027s not serializable. An operation that was issued once actually may have happened twice - that does not mean only arbitrary order but arbitrary number.","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"f1a9ed4dd7d17fec11bbdee58caad22de5896200","unresolved":false,"context_lines":[{"line_number":67,"context_line":".. [#] In distributed systems world, serializable means that the history of"},{"line_number":68,"context_line":"   all operations is equivalent to one that took place in some single atomic"},{"line_number":69,"context_line":"   order but with unknown invocation and completion times - it places no bounds"},{"line_number":70,"context_line":"   on time or order."},{"line_number":71,"context_line":""},{"line_number":72,"context_line":"In fact there are two layers - messaging driver API and Oslo messaging user API."},{"line_number":73,"context_line":"The second one should have patterns to solve the work queue use case."}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_02040ebe","line":70,"in_reply_to":"5a710552_c9e4f03c","updated":"2015-12-15 12:54:10.000000000","message":"AFAIK, it applies as well to any generic operations. good point with duplicates though. Any ideas which consistency claim that would give us then?","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"b8522a87ee4e1ca82a6e7dfb4fe22978258271ba","unresolved":false,"context_lines":[{"line_number":106,"context_line":"  f |  Client.delete_task("},{"line_number":107,"context_line":"  l |      task, timeout,    Server.wait                   Worker.wait"},{"line_number":108,"context_line":"  o |      callback)"},{"line_number":109,"context_line":"  w v"},{"line_number":110,"context_line":""},{"line_number":111,"context_line":"Failure modes"},{"line_number":112,"context_line":"-------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_a0874060","line":109,"updated":"2015-12-15 09:01:25.000000000","message":"This is WIP and doesn\u0027t reflect any end-state workflow. Perhaps this spec should end up with only task_cast user API and distribute_cast method. The call semantics seems too complex, at least for me to understand and describe it... fast enough... BUT would be nice to have it described as well.","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":11744,"name":"Alec Hothan","email":"ahothan@cisco.com","username":"ahothan"},"change_message_id":"38fca4322195d67f94ed3c69cb4391f9c596841e","unresolved":false,"context_lines":[{"line_number":107,"context_line":"  l |      task, timeout,    Server.wait                   Worker.wait"},{"line_number":108,"context_line":"  o |      callback)"},{"line_number":109,"context_line":"  w v"},{"line_number":110,"context_line":""},{"line_number":111,"context_line":"Failure modes"},{"line_number":112,"context_line":"-------------"},{"line_number":113,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_658f0ecd","line":110,"updated":"2015-12-15 16:25:14.000000000","message":"Not clear how many processes are represented in this diagram? I see the client process, followed by an IPC arrow (is that the IPC API or is it an IPC?) then a Worker queue server (is this another background process or a thread in the client library?), followed by driver API arrow, then \"AMQP broker\" (is this the broker server itself or the AMQP broker client library?), then a driver API basic consume arrow, followed by a worker process (is this another process?)\n\nMaybe need some clarification to delimitate clearly API library boundary from process boundary. Because of all these unknowns, it is hard to comment further down as  to the exact location of all these modules and queues.\n\nI know it is not obvious using plain text, I wish one could paste png diagrams in all openstack specs...","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"9766a109daace29fb55e75e01e6ba87b1d28c99d","unresolved":false,"context_lines":[{"line_number":107,"context_line":"  l |      task, timeout,    Server.wait                   Worker.wait"},{"line_number":108,"context_line":"  o |      callback)"},{"line_number":109,"context_line":"  w v"},{"line_number":110,"context_line":""},{"line_number":111,"context_line":"Failure modes"},{"line_number":112,"context_line":"-------------"},{"line_number":113,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_14ef5a88","line":110,"in_reply_to":"5a710552_658f0ecd","updated":"2015-12-16 07:19:36.000000000","message":"I put IPC as an example of inter-process communication of caller to server. While it is AMQP for server to workers. We may want to implement AMQP-for-all though.","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":8601,"name":"Dmitriy Ukhlov","email":"dukhlov@mirantis.com","username":"dukhlov"},"change_message_id":"63c71583f1406c73213c6c858e52b15fef14d543","unresolved":false,"context_lines":[{"line_number":108,"context_line":"  o |      callback)"},{"line_number":109,"context_line":"  w v"},{"line_number":110,"context_line":""},{"line_number":111,"context_line":"Failure modes"},{"line_number":112,"context_line":"-------------"},{"line_number":113,"context_line":""},{"line_number":114,"context_line":"Processing contracts (assumes none crash-stop failures):"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_80eec489","line":111,"updated":"2015-12-15 09:12:46.000000000","message":"You described async vervion of RPC with at-least-once processing.\nIt is not WorkingQueue in my opinion\nLets clarify usecase first. What we want to achive and where  we want to use it\n\nBTW in suggested solution I dont understand why do we need WorkQueue server. It looks like it logic could be implemented on client side. And also it not clear how delete_task will work because task is already in queue and we will process it anyway","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"0b4bd3169789fad7ec26492e200f056a64222c4d","unresolved":false,"context_lines":[{"line_number":108,"context_line":"  o |      callback)"},{"line_number":109,"context_line":"  w v"},{"line_number":110,"context_line":""},{"line_number":111,"context_line":"Failure modes"},{"line_number":112,"context_line":"-------------"},{"line_number":113,"context_line":""},{"line_number":114,"context_line":"Processing contracts (assumes none crash-stop failures):"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_615b5588","line":111,"in_reply_to":"5a710552_22efcae9","updated":"2015-12-16 10:39:36.000000000","message":"@Nikola, another use case for apps is making a cross-clusters geo-distributed topology like \"client -\u003e Server -\u003e Manager(server 2) -\u003e [multi-task workers]\", see http://www.cse.nd.edu/Reports/2013/TR-2013-05.pdf for details. although they describe completely another type of work queue, but the concept is valid here as well. What is why the lib shall have well implemented server API to allow apps to do that, eventually, in some distant future :)","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"2fd7074c38538a0d9636735decec1599d4584cd9","unresolved":false,"context_lines":[{"line_number":108,"context_line":"  o |      callback)"},{"line_number":109,"context_line":"  w v"},{"line_number":110,"context_line":""},{"line_number":111,"context_line":"Failure modes"},{"line_number":112,"context_line":"-------------"},{"line_number":113,"context_line":""},{"line_number":114,"context_line":"Processing contracts (assumes none crash-stop failures):"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_006fe62b","line":111,"in_reply_to":"5a710552_615b5588","updated":"2015-12-16 12:40:23.000000000","message":"Ok having a well defined API is (always?) a good idea, as long as we are clear on what the difference is when you run it as a thread that is part of the same process, or as a separate process on a possibly remote box.\n\nI assume that the first implementation will not allow for running it separately.","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"e7065c307240ed2998234ca776b4a9d82564ab76","unresolved":false,"context_lines":[{"line_number":108,"context_line":"  o |      callback)"},{"line_number":109,"context_line":"  w v"},{"line_number":110,"context_line":""},{"line_number":111,"context_line":"Failure modes"},{"line_number":112,"context_line":"-------------"},{"line_number":113,"context_line":""},{"line_number":114,"context_line":"Processing contracts (assumes none crash-stop failures):"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_e968ccbc","line":111,"in_reply_to":"5a710552_80eec489","updated":"2015-12-15 12:10:24.000000000","message":"\u003e BTW in suggested solution I dont understand why do we need WorkQueue server. It looks like it logic could be implemented on client side. And also it not clear how delete_task will work because task is already in queue and we will process it anyway\n\nI had the same concerns - A lot of what WorkQueue server is supposed to do above can be done by the broker (ack the received message) or worker (requeue failed message, or ask for redelivery periodically).\n\nDelete is also not something we can really do without jumping through hoops. If we want delete - we\u0027ll want something closer to what Joshua describes above.","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"f1a9ed4dd7d17fec11bbdee58caad22de5896200","unresolved":false,"context_lines":[{"line_number":108,"context_line":"  o |      callback)"},{"line_number":109,"context_line":"  w v"},{"line_number":110,"context_line":""},{"line_number":111,"context_line":"Failure modes"},{"line_number":112,"context_line":"-------------"},{"line_number":113,"context_line":""},{"line_number":114,"context_line":"Processing contracts (assumes none crash-stop failures):"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_22efcae9","line":111,"in_reply_to":"5a710552_e968ccbc","updated":"2015-12-15 12:54:10.000000000","message":"There are (maight be?) broker-less cases. We should not decide for users and rely on the broker instead of the master\n\nAFAICT, create/delete should prepare and teardown the data required for a task entity. Like cache files and encapsulate data into the task message body, for example.\nDo we want external files as inputs? Some DSL or raw data inside perhaps?..","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"e7065c307240ed2998234ca776b4a9d82564ab76","unresolved":false,"context_lines":[{"line_number":123,"context_line":"  ping-ponging it across workers or driver errors or a crash-stop failures at"},{"line_number":124,"context_line":"  any side."},{"line_number":125,"context_line":""},{"line_number":126,"context_line":"* On the user API level, the requested task shall fail after the given timeout,"},{"line_number":127,"context_line":"  if either failed processing by the timeout or due to distribution or other"},{"line_number":128,"context_line":"  processing issues down the way."},{"line_number":129,"context_line":"  "},{"line_number":130,"context_line":"* A callback shall be executed by the expected timeout w/o errors or with no"},{"line_number":131,"context_line":"  time bounds but with the returned error passed from the server."}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_c44617bf","line":128,"range":{"start_line":126,"start_character":2,"end_line":128,"end_character":33},"updated":"2015-12-15 12:10:24.000000000","message":"So this is at odds with what we said above - that we can\u0027t guarantee that the message _has not_ been delivered. So this should really be - \"may have failed\" not \"fail\".\n\nI understand that you tried to split this between crash and non-crash failures, but I am not sure we want to make the distinction in the API, since it may not be ultimately useful. How is a crash failure different from another failure from the client POV?","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"f1a9ed4dd7d17fec11bbdee58caad22de5896200","unresolved":false,"context_lines":[{"line_number":125,"context_line":""},{"line_number":126,"context_line":"* On the user API level, the requested task shall fail after the given timeout,"},{"line_number":127,"context_line":"  if either failed processing by the timeout or due to distribution or other"},{"line_number":128,"context_line":"  processing issues down the way."},{"line_number":129,"context_line":"  "},{"line_number":130,"context_line":"* A callback shall be executed by the expected timeout w/o errors or with no"},{"line_number":131,"context_line":"  time bounds but with the returned error passed from the server."}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_02fdee39","line":128,"in_reply_to":"5a710552_c44617bf","updated":"2015-12-15 12:54:10.000000000","message":"A crash-stop failure might end up not reporting back valid results for the API \"call\" -at any stage!- as the failure may not always be detected. It may also render false FAILURE or false SUCCESS being reported. While an exception or an error leaves some detectable details one can trust.\nFor an API \"cast\" there probably would be no difference tho.","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"e7065c307240ed2998234ca776b4a9d82564ab76","unresolved":false,"context_lines":[{"line_number":138,"context_line":"  ack-then-process or process-then-ack modes. The only difference is that the"},{"line_number":139,"context_line":"  latter has stronger task execution guarantees then the former one."},{"line_number":140,"context_line":""},{"line_number":141,"context_line":".. [#] A callback may be used only then both the client and server are using"},{"line_number":142,"context_line":"   the same host and communitacting via IPC."},{"line_number":143,"context_line":""},{"line_number":144,"context_line":"TODO(bogdando) describe non-IPC-all case "},{"line_number":145,"context_line":"  "}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_1f0f8ae2","line":142,"range":{"start_line":141,"start_character":7,"end_line":142,"end_character":44},"updated":"2015-12-15 12:10:24.000000000","message":"Again this tells me that server logic really should be split between the client and the worker (And the queue)","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"f1a9ed4dd7d17fec11bbdee58caad22de5896200","unresolved":false,"context_lines":[{"line_number":139,"context_line":"  latter has stronger task execution guarantees then the former one."},{"line_number":140,"context_line":""},{"line_number":141,"context_line":".. [#] A callback may be used only then both the client and server are using"},{"line_number":142,"context_line":"   the same host and communitacting via IPC."},{"line_number":143,"context_line":""},{"line_number":144,"context_line":"TODO(bogdando) describe non-IPC-all case "},{"line_number":145,"context_line":"  "}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_22910af1","line":142,"in_reply_to":"5a710552_1f0f8ae2","updated":"2015-12-15 12:54:10.000000000","message":"A good point. But would it be possible without making assumptions for a broker existence on the driver layer?","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"2fd7074c38538a0d9636735decec1599d4584cd9","unresolved":false,"context_lines":[{"line_number":139,"context_line":"  latter has stronger task execution guarantees then the former one."},{"line_number":140,"context_line":""},{"line_number":141,"context_line":".. [#] A callback may be used only then both the client and server are using"},{"line_number":142,"context_line":"   the same host and communitacting via IPC."},{"line_number":143,"context_line":""},{"line_number":144,"context_line":"TODO(bogdando) describe non-IPC-all case "},{"line_number":145,"context_line":"  "}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_e02bba92","line":142,"in_reply_to":"5a710552_22910af1","updated":"2015-12-16 12:40:23.000000000","message":"actually this basically means - we want the server \"abstraction\" (i.e. a well defined API) but the only implementation we\u0027ll have at firs is running it as a thread of the same process. That is fine.","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"e7065c307240ed2998234ca776b4a9d82564ab76","unresolved":false,"context_lines":[{"line_number":150,"context_line":"  know the status of workers and there are no contracts between them. It\u0027s up"},{"line_number":151,"context_line":"  to the worker app how to recover and is out of the scope of Oslo messaging."},{"line_number":152,"context_line":""},{"line_number":153,"context_line":"* Then AMQP broker crashes or fails to process publish/consume/(n)ack/reject"},{"line_number":154,"context_line":"  operations to work queues, it shall do the best to recover its state."},{"line_number":155,"context_line":"  But there is no contracts between the broker and workers and a work queue"},{"line_number":156,"context_line":"  server. It\u0027s up to the broker how to recover and is out of the scope of Oslo"},{"line_number":157,"context_line":"  messaging."},{"line_number":158,"context_line":""},{"line_number":159,"context_line":"* Then the work queue server crashes of fails to distribute a task by the given"},{"line_number":160,"context_line":"  timeout, it shall do the best to signal the caller about the failure."}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_9ff13ad0","line":157,"range":{"start_line":153,"start_character":2,"end_line":157,"end_character":12},"updated":"2015-12-15 12:10:24.000000000","message":"While this is probably the best approach to take - we want to be extremely clear about this.","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"2fd7074c38538a0d9636735decec1599d4584cd9","unresolved":false,"context_lines":[{"line_number":154,"context_line":"  operations to work queues, it shall do the best to recover its state."},{"line_number":155,"context_line":"  But there is no contracts between the broker and workers and a work queue"},{"line_number":156,"context_line":"  server. It\u0027s up to the broker how to recover and is out of the scope of Oslo"},{"line_number":157,"context_line":"  messaging."},{"line_number":158,"context_line":""},{"line_number":159,"context_line":"* Then the work queue server crashes of fails to distribute a task by the given"},{"line_number":160,"context_line":"  timeout, it shall do the best to signal the caller about the failure."}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_4bf74ff5","line":157,"in_reply_to":"5a710552_227fcaab","updated":"2015-12-16 12:40:23.000000000","message":"Well there are ways to make brokers more reliable, for example acking messages to the client  only after they\u0027ve been persisted to disk.\n\nWhat I was saying is that we should be clear that we are not providing end2end acks, but something weaker, which you can make \"less weak\" depending on the underlying driver.","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"f1a9ed4dd7d17fec11bbdee58caad22de5896200","unresolved":false,"context_lines":[{"line_number":154,"context_line":"  operations to work queues, it shall do the best to recover its state."},{"line_number":155,"context_line":"  But there is no contracts between the broker and workers and a work queue"},{"line_number":156,"context_line":"  server. It\u0027s up to the broker how to recover and is out of the scope of Oslo"},{"line_number":157,"context_line":"  messaging."},{"line_number":158,"context_line":""},{"line_number":159,"context_line":"* Then the work queue server crashes of fails to distribute a task by the given"},{"line_number":160,"context_line":"  timeout, it shall do the best to signal the caller about the failure."}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_227fcaab","line":157,"in_reply_to":"5a710552_9ff13ad0","updated":"2015-12-15 12:54:10.000000000","message":"But we can do nothing to a broker. It is 100% out of the library scope...","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"b8522a87ee4e1ca82a6e7dfb4fe22978258271ba","unresolved":false,"context_lines":[{"line_number":172,"context_line":"* A crash-stop failure leaves no guarantees to neither a callback to be"},{"line_number":173,"context_line":"  called, nor reported failure ensures no task was executed, nor reported"},{"line_number":174,"context_line":"  success ensures a task to be or was executed. It may cause a task to"},{"line_number":175,"context_line":"  be lost or duplicated as well."},{"line_number":176,"context_line":""},{"line_number":177,"context_line":"Alternatives"},{"line_number":178,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_00973491","line":175,"updated":"2015-12-15 09:01:25.000000000","message":"ditto, WIP.","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"904d83970b7592242797a1f55e4dfb324013cbf1","unresolved":false,"context_lines":[{"line_number":183,"context_line":"  delivery as well as rpc with at-most-once one."},{"line_number":184,"context_line":"* Rework existing \"at-least-once\" pattern implemented for notifications"},{"line_number":185,"context_line":"  to support a work queue"},{"line_number":186,"context_line":"* Use external task queue solutions like celery. Do nothing for Oslo."},{"line_number":187,"context_line":""},{"line_number":188,"context_line":"Impact on Existing APIs"},{"line_number":189,"context_line":"-----------------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_8f6843bc","line":186,"updated":"2015-12-15 00:38:03.000000000","message":"Or put the jobboard (or similar) concept in tooz and refine it there, leave oslo.messaging for RPC. I start to wonder about that option imho. Because really the jobboard is the task queue, celery is a lot more built ontop of similarish stuff, taskflow also builds concepts ontop of that jobboard.... Worth thinking about at least...","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"e0df6d0c86b2961360e09e40f3f6773d4c3e12ea","unresolved":false,"context_lines":[{"line_number":183,"context_line":"  delivery as well as rpc with at-most-once one."},{"line_number":184,"context_line":"* Rework existing \"at-least-once\" pattern implemented for notifications"},{"line_number":185,"context_line":"  to support a work queue"},{"line_number":186,"context_line":"* Use external task queue solutions like celery. Do nothing for Oslo."},{"line_number":187,"context_line":""},{"line_number":188,"context_line":"Impact on Existing APIs"},{"line_number":189,"context_line":"-----------------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_040de336","line":186,"in_reply_to":"5a710552_3f3a8637","updated":"2015-12-15 19:12:55.000000000","message":"Right, its a architectural decision in the end... IMHO I prefer picking a solution that is better (tooz and zookeeper for example) and pushing stronger implementations vs pushing weaker ones (doing it because it is more lightweight, although appealing doesn\u0027t seem like the best path forward, at least it wouldn\u0027t be the way I would want to do it...).","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"b8522a87ee4e1ca82a6e7dfb4fe22978258271ba","unresolved":false,"context_lines":[{"line_number":183,"context_line":"  delivery as well as rpc with at-most-once one."},{"line_number":184,"context_line":"* Rework existing \"at-least-once\" pattern implemented for notifications"},{"line_number":185,"context_line":"  to support a work queue"},{"line_number":186,"context_line":"* Use external task queue solutions like celery. Do nothing for Oslo."},{"line_number":187,"context_line":""},{"line_number":188,"context_line":"Impact on Existing APIs"},{"line_number":189,"context_line":"-----------------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_c0e7ec3e","line":186,"in_reply_to":"5a710552_8f6843bc","updated":"2015-12-15 09:01:25.000000000","message":"A good point, thank you!","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"e7065c307240ed2998234ca776b4a9d82564ab76","unresolved":false,"context_lines":[{"line_number":183,"context_line":"  delivery as well as rpc with at-most-once one."},{"line_number":184,"context_line":"* Rework existing \"at-least-once\" pattern implemented for notifications"},{"line_number":185,"context_line":"  to support a work queue"},{"line_number":186,"context_line":"* Use external task queue solutions like celery. Do nothing for Oslo."},{"line_number":187,"context_line":""},{"line_number":188,"context_line":"Impact on Existing APIs"},{"line_number":189,"context_line":"-----------------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"5a710552_3f3a8637","line":186,"in_reply_to":"5a710552_c0e7ec3e","updated":"2015-12-15 12:10:24.000000000","message":"\u003e refine it there, leave oslo.messaging for RPC\n\nThis is definitely an appealing option IMHO. They are not equivalent though.\n\nDoing this using only rabbitmq gives us less power (we can\u0027t easily revoke tasks etc, we have to rely on acks) but also makes it more lightweight for apps that can be happy with this simpler but ultimately weaker service.","commit_id":"1ef72258efd42c894f5894ffdd9de8d11e03480e"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"d43f6c60813e047ae47f486f6a641bd240f49a01","unresolved":false,"context_lines":[{"line_number":38,"context_line":"Proposed change"},{"line_number":39,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":40,"context_line":""},{"line_number":41,"context_line":"Given that said, the solution is to add new API for a work queue. There should"},{"line_number":42,"context_line":"be new classess WQClient with create/submit/wait/delete and WQDispatcher and"},{"line_number":43,"context_line":"methods like ``distribute()`` [#]_, which resembles the ``_dispatch_and_reply``"},{"line_number":44,"context_line":"of the RPCDispatcher class. But new method shall provide the \"at-least-once\""}],"source_content_type":"text/x-rst","patch_set":4,"id":"5a710552_815be12d","line":41,"updated":"2015-12-16 10:30:53.000000000","message":"todo add the brief description like: Propose a lightweight async WQ cast/call flow with called results returned as futures to track the progress. The contract for the futures covers only the task distribution (accepted for processing). And does not covers processing itself. Later, WQ may encapsulate and replace existing RPC by unified workers capable do rpc-as-a-task as well.","commit_id":"486c241583901e8611ced9c7eba166e85548ee5a"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"5624f7b05e11704fa165f8f42b3494e7b12df327","unresolved":false,"context_lines":[{"line_number":40,"context_line":""},{"line_number":41,"context_line":"Given that said, the solution is to add new API for a work queue. There should"},{"line_number":42,"context_line":"be new classess WQClient with create/submit/wait/delete and WQDispatcher and"},{"line_number":43,"context_line":"methods like ``distribute()`` [#]_, which resembles the ``_dispatch_and_reply``"},{"line_number":44,"context_line":"of the RPCDispatcher class. But new method shall provide the \"at-least-once\""},{"line_number":45,"context_line":"delivery model and may normally return nothing, if being casted. If called,"},{"line_number":46,"context_line":"it may also return an error or event that says \"failed to distribute the task X"},{"line_number":47,"context_line":"in Y msec\" but even so it cannot guarantee that the corresponding request was"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5a710552_a0912d9b","line":44,"range":{"start_line":43,"start_character":36,"end_line":44,"end_character":20},"updated":"2015-12-16 13:43:15.000000000","message":"So this part has been confusing me from the previous patch-set.\n\n_dispatch_and_reply is really an internal implementation detail of the \"RPC server\" i.e, it\u0027s main task is to map a message name to a python instance method name and call that method. But it happens after message is consumed.\n\nWhat you re talking about here with distribute() will be run in the context of a messaging \"client\" (even though it may be a separate process from the client in the sense of the API we try to design here) and eventually publish messages to the broker, so will result in a basic.publish.\n\nI don\u0027t think they resemble eachother - I feel like I\u0027m missing something here...","commit_id":"486c241583901e8611ced9c7eba166e85548ee5a"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"f6dee974f8f8a1383270b81e7cdba14b8f6a0004","unresolved":false,"context_lines":[{"line_number":41,"context_line":"Given that said, the solution is to add new API for a work queue. There should"},{"line_number":42,"context_line":"be new classess WQClient with create/submit/wait/delete and WQDispatcher and"},{"line_number":43,"context_line":"methods like ``distribute()`` [#]_, which resembles the ``_dispatch_and_reply``"},{"line_number":44,"context_line":"of the RPCDispatcher class. But new method shall provide the \"at-least-once\""},{"line_number":45,"context_line":"delivery model and may normally return nothing, if being casted. If called,"},{"line_number":46,"context_line":"it may also return an error or event that says \"failed to distribute the task X"},{"line_number":47,"context_line":"in Y msec\" but even so it cannot guarantee that the corresponding request was"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5a710552_236f87c6","line":44,"in_reply_to":"5a710552_a0912d9b","updated":"2015-12-16 14:01:16.000000000","message":"I will remove this then","commit_id":"486c241583901e8611ced9c7eba166e85548ee5a"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"7343f08293a542452c706d16fae4efe73f1a38da","unresolved":false,"context_lines":[{"line_number":80,"context_line":"   on time or order of operations. Here it is relaxed as there may be"},{"line_number":81,"context_line":"   duplicating operations."},{"line_number":82,"context_line":""},{"line_number":83,"context_line":".. [#] In the face of a crash-stop failure, a task may be lost undetected by the"},{"line_number":84,"context_line":"   library logic for WQ. But otherwise, processing contracts shall guarantee at"},{"line_number":85,"context_line":"   least detection of this case, at most - no lost tasks at all."},{"line_number":86,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"5a710552_9dee06b3","line":83,"updated":"2015-12-15 20:36:29.000000000","message":"Eck, that sucks. Lost tasks on crash/stops?","commit_id":"486c241583901e8611ced9c7eba166e85548ee5a"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"5624f7b05e11704fa165f8f42b3494e7b12df327","unresolved":false,"context_lines":[{"line_number":80,"context_line":"   on time or order of operations. Here it is relaxed as there may be"},{"line_number":81,"context_line":"   duplicating operations."},{"line_number":82,"context_line":""},{"line_number":83,"context_line":".. [#] In the face of a crash-stop failure, a task may be lost undetected by the"},{"line_number":84,"context_line":"   library logic for WQ. But otherwise, processing contracts shall guarantee at"},{"line_number":85,"context_line":"   least detection of this case, at most - no lost tasks at all."},{"line_number":86,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"5a710552_cca9157d","line":83,"in_reply_to":"5a710552_12bbce0f","updated":"2015-12-16 13:43:15.000000000","message":"\u003e If *stateless* WQ server crashes inside of its timer loop for a task being distributed and yet to be accepted by a worker, it would be lost undetected as well. That is things work like (and suxx).\n\nSo running the WQ server logic as part of the client would eliminate that failure mode right? I totally agree that we may want to make it\u0027s API clean so that eventually it could be it\u0027s own thing.\n\n(Sorry for beating my own drum from a previous review)","commit_id":"486c241583901e8611ced9c7eba166e85548ee5a"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"b92ef7c68a72c143fdbdc68be86e30efd2de5447","unresolved":false,"context_lines":[{"line_number":80,"context_line":"   on time or order of operations. Here it is relaxed as there may be"},{"line_number":81,"context_line":"   duplicating operations."},{"line_number":82,"context_line":""},{"line_number":83,"context_line":".. [#] In the face of a crash-stop failure, a task may be lost undetected by the"},{"line_number":84,"context_line":"   library logic for WQ. But otherwise, processing contracts shall guarantee at"},{"line_number":85,"context_line":"   least detection of this case, at most - no lost tasks at all."},{"line_number":86,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"5a710552_12bbce0f","line":83,"in_reply_to":"5a710552_9dee06b3","updated":"2015-12-16 07:36:12.000000000","message":"If messages -encapsulating tasks- are not persistent, those would be lost on the broker crash-stop. Lost undetected by the library. If *stateless* WQ server crashes inside of its timer loop for a task being distributed and yet to be accepted by a worker, it would be lost undetected as well. That is things work like (and suxx).","commit_id":"486c241583901e8611ced9c7eba166e85548ee5a"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"f6dee974f8f8a1383270b81e7cdba14b8f6a0004","unresolved":false,"context_lines":[{"line_number":80,"context_line":"   on time or order of operations. Here it is relaxed as there may be"},{"line_number":81,"context_line":"   duplicating operations."},{"line_number":82,"context_line":""},{"line_number":83,"context_line":".. [#] In the face of a crash-stop failure, a task may be lost undetected by the"},{"line_number":84,"context_line":"   library logic for WQ. But otherwise, processing contracts shall guarantee at"},{"line_number":85,"context_line":"   least detection of this case, at most - no lost tasks at all."},{"line_number":86,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"5a710552_a335b7da","line":83,"in_reply_to":"5a710552_cca9157d","updated":"2015-12-16 14:01:16.000000000","message":"I believe all of the depicted below components: clients, server, broker, workers shall be able running on separate nodes, if the app dev wants this. We cannot eliminate this failure mode. I regret I made an IPC example, shall redo to AMQP-for-all","commit_id":"486c241583901e8611ced9c7eba166e85548ee5a"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"9b68c6c1bfb6e634169530e2ac1e21f11e9e80a1","unresolved":false,"context_lines":[{"line_number":166,"context_line":"   event shall be used."},{"line_number":167,"context_line":"   (TODO)bogdando describe AMQP-for-all case as well"},{"line_number":168,"context_line":""},{"line_number":169,"context_line":".. [#] Workers may ping-pong the message back to the queue due to exceptions"},{"line_number":170,"context_line":"   raised for the particular message with each worker. These must be limited"},{"line_number":171,"context_line":"   in time and retries as well."},{"line_number":172,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"5a710552_b450c65f","line":169,"updated":"2015-12-15 17:52:19.000000000","message":"@Mehdi, I hope I answered your case with exceptions in workers here.","commit_id":"486c241583901e8611ced9c7eba166e85548ee5a"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"7343f08293a542452c706d16fae4efe73f1a38da","unresolved":false,"context_lines":[{"line_number":174,"context_line":""},{"line_number":175,"context_line":"Crash-stop failure contracts:"},{"line_number":176,"context_line":""},{"line_number":177,"context_line":"* When a worker crashes or violates task processing timeout [#]_, it shall do"},{"line_number":178,"context_line":"  the best to resubmit its task back to the queue. The worker may track its"},{"line_number":179,"context_line":"  local state of accepted and not processed tasks to try to recover after a"},{"line_number":180,"context_line":"  crash-stop failure. The acceptance report message shall be atomic with the"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5a710552_1d5e9676","line":177,"updated":"2015-12-15 20:36:29.000000000","message":"This will be interesting... I wonder how a worker that crashes does its best at anything :-P","commit_id":"486c241583901e8611ced9c7eba166e85548ee5a"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"d8cbdfd3853c14098ff2666282a794903d8181ba","unresolved":false,"context_lines":[{"line_number":174,"context_line":""},{"line_number":175,"context_line":"Crash-stop failure contracts:"},{"line_number":176,"context_line":""},{"line_number":177,"context_line":"* When a worker crashes or violates task processing timeout [#]_, it shall do"},{"line_number":178,"context_line":"  the best to resubmit its task back to the queue. The worker may track its"},{"line_number":179,"context_line":"  local state of accepted and not processed tasks to try to recover after a"},{"line_number":180,"context_line":"  crash-stop failure. The acceptance report message shall be atomic with the"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5a710552_b89659ee","line":177,"in_reply_to":"5a710552_12174eee","updated":"2015-12-16 08:23:07.000000000","message":"Yup.","commit_id":"486c241583901e8611ced9c7eba166e85548ee5a"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"b92ef7c68a72c143fdbdc68be86e30efd2de5447","unresolved":false,"context_lines":[{"line_number":174,"context_line":""},{"line_number":175,"context_line":"Crash-stop failure contracts:"},{"line_number":176,"context_line":""},{"line_number":177,"context_line":"* When a worker crashes or violates task processing timeout [#]_, it shall do"},{"line_number":178,"context_line":"  the best to resubmit its task back to the queue. The worker may track its"},{"line_number":179,"context_line":"  local state of accepted and not processed tasks to try to recover after a"},{"line_number":180,"context_line":"  crash-stop failure. The acceptance report message shall be atomic with the"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5a710552_12174eee","line":177,"in_reply_to":"5a710552_1d5e9676","updated":"2015-12-16 07:36:12.000000000","message":"Example worker app is a daemon with the respawn option. Its state is kept as a local file and reread on restart. Makes sense?","commit_id":"486c241583901e8611ced9c7eba166e85548ee5a"},{"author":{"_account_id":5511,"name":"Nikola Dipanov","email":"ndipanov@redhat.com","username":"ndipanov"},"change_message_id":"5624f7b05e11704fa165f8f42b3494e7b12df327","unresolved":false,"context_lines":[{"line_number":175,"context_line":"Crash-stop failure contracts:"},{"line_number":176,"context_line":""},{"line_number":177,"context_line":"* When a worker crashes or violates task processing timeout [#]_, it shall do"},{"line_number":178,"context_line":"  the best to resubmit its task back to the queue. The worker may track its"},{"line_number":179,"context_line":"  local state of accepted and not processed tasks to try to recover after a"},{"line_number":180,"context_line":"  crash-stop failure. The acceptance report message shall be atomic with the"},{"line_number":181,"context_line":"  local state commit. It\u0027s up to the worker app how to recover and is out of"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5a710552_2ce9b980","line":178,"range":{"start_line":178,"start_character":27,"end_line":178,"end_character":50},"updated":"2015-12-16 13:43:15.000000000","message":"So this can be done by asking for redelivery in case of rabbit. this could be hidden behind the Worker API maybe?","commit_id":"486c241583901e8611ced9c7eba166e85548ee5a"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"f6dee974f8f8a1383270b81e7cdba14b8f6a0004","unresolved":false,"context_lines":[{"line_number":175,"context_line":"Crash-stop failure contracts:"},{"line_number":176,"context_line":""},{"line_number":177,"context_line":"* When a worker crashes or violates task processing timeout [#]_, it shall do"},{"line_number":178,"context_line":"  the best to resubmit its task back to the queue. The worker may track its"},{"line_number":179,"context_line":"  local state of accepted and not processed tasks to try to recover after a"},{"line_number":180,"context_line":"  crash-stop failure. The acceptance report message shall be atomic with the"},{"line_number":181,"context_line":"  local state commit. It\u0027s up to the worker app how to recover and is out of"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5a710552_237d476e","line":178,"in_reply_to":"5a710552_2ce9b980","updated":"2015-12-16 14:01:16.000000000","message":"I don\u0027t want to be specific to the driver backend here. Why do you think AMQP basic.reject (or nack for a batch) should not be used to requeue message back?","commit_id":"486c241583901e8611ced9c7eba166e85548ee5a"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"b92ef7c68a72c143fdbdc68be86e30efd2de5447","unresolved":false,"context_lines":[{"line_number":179,"context_line":"  local state of accepted and not processed tasks to try to recover after a"},{"line_number":180,"context_line":"  crash-stop failure. The acceptance report message shall be atomic with the"},{"line_number":181,"context_line":"  local state commit. It\u0027s up to the worker app how to recover and is out of"},{"line_number":182,"context_line":"  the scope of Oslo messaging."},{"line_number":183,"context_line":""},{"line_number":184,"context_line":"* Then AMQP broker crashes or fails to process publish/consume/(n)ack/reject"},{"line_number":185,"context_line":"  operations to work queues, it shall do the best to recover its state."}],"source_content_type":"text/x-rst","patch_set":4,"id":"5a710552_d2f2266f","line":182,"updated":"2015-12-16 07:36:12.000000000","message":"The worker shall ack and drop the message on the floor, if it sees its processing contract is expired. It is expired then the distribution SLA T plus submission timestamp from the server (both encapsulated in the message along with the task itself) is less when the current worker local time.","commit_id":"486c241583901e8611ced9c7eba166e85548ee5a"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"0f7ef9d676f81b7a9ff3e264440cb2e868370358","unresolved":false,"context_lines":[{"line_number":223,"context_line":"  delivery as well as rpc with at-most-once one."},{"line_number":224,"context_line":"* Rework existing \"at-least-once\" pattern implemented for notifications"},{"line_number":225,"context_line":"  to support a work queue"},{"line_number":226,"context_line":"* Use external task queue solutions like celery/taskflow. Do nothing for Oslo."},{"line_number":227,"context_line":"* Or put the jobboard (or similar) concept in tooz and refine it there, leave"},{"line_number":228,"context_line":"  Oslo.messaging for RPC. Because really the jobboard is the task queue, celery"},{"line_number":229,"context_line":"  is a lot more built ontop of similarish stuff, taskflow also builds concepts"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5a710552_2727c1b7","line":226,"updated":"2015-12-15 19:38:05.000000000","message":"So I\u0027m not sure celery belongs here, celery is really just using kombu itself (kombu came out/was developed side-by-side of the celery project btw), and is the framework on-top of it. Taskflow isn\u0027t really a task queue solution either (but the jobboard concept from it can be), both celery and taskflow build on those foundational concepts to add things, but they are not the solution them-self so probably not worth mentioning them as such here.","commit_id":"486c241583901e8611ced9c7eba166e85548ee5a"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"f6dee974f8f8a1383270b81e7cdba14b8f6a0004","unresolved":false,"context_lines":[{"line_number":223,"context_line":"  delivery as well as rpc with at-most-once one."},{"line_number":224,"context_line":"* Rework existing \"at-least-once\" pattern implemented for notifications"},{"line_number":225,"context_line":"  to support a work queue"},{"line_number":226,"context_line":"* Use external task queue solutions like celery/taskflow. Do nothing for Oslo."},{"line_number":227,"context_line":"* Or put the jobboard (or similar) concept in tooz and refine it there, leave"},{"line_number":228,"context_line":"  Oslo.messaging for RPC. Because really the jobboard is the task queue, celery"},{"line_number":229,"context_line":"  is a lot more built ontop of similarish stuff, taskflow also builds concepts"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5a710552_03da8b78","line":226,"in_reply_to":"5a710552_2727c1b7","updated":"2015-12-16 14:01:16.000000000","message":"I will remove these names","commit_id":"486c241583901e8611ced9c7eba166e85548ee5a"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"0f7ef9d676f81b7a9ff3e264440cb2e868370358","unresolved":false,"context_lines":[{"line_number":225,"context_line":"  to support a work queue"},{"line_number":226,"context_line":"* Use external task queue solutions like celery/taskflow. Do nothing for Oslo."},{"line_number":227,"context_line":"* Or put the jobboard (or similar) concept in tooz and refine it there, leave"},{"line_number":228,"context_line":"  Oslo.messaging for RPC. Because really the jobboard is the task queue, celery"},{"line_number":229,"context_line":"  is a lot more built ontop of similarish stuff, taskflow also builds concepts"},{"line_number":230,"context_line":"  ontop of that jobboard."},{"line_number":231,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"5a710552_a4264fcc","line":228,"updated":"2015-12-15 19:38:05.000000000","message":"+1\n\nI sorta like this (but I\u0027m obviously biased).\n\nI somewhat predict that all the above changes to oslo.messaging will eventually just rebuild something *like* what taskflow already has/provides (but in a way that seems initially more lightweight, but will imho actually be more complex to get right and will have many more edge cases [see above cases...] because of that complexity), so why go through all that work in the first place...\n\nJust something to consider.","commit_id":"486c241583901e8611ced9c7eba166e85548ee5a"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"91f615446914c3583ecb346d1775816422054212","unresolved":false,"context_lines":[{"line_number":296,"context_line":" * Write the ``WorkQueueDispatcher`` class with"},{"line_number":297,"context_line":"   at least ``ensure()`` or ``distribute()`` method"},{"line_number":298,"context_line":" * Design test cases"},{"line_number":299,"context_line":" * Update work queue use cases documentaion for developers"},{"line_number":300,"context_line":""},{"line_number":301,"context_line":"Documentation Impact"},{"line_number":302,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5a710552_4810be01","line":299,"updated":"2015-12-16 09:43:31.000000000","message":"wip.\n\nThe step 2 should be: impliment only task_submit, distribute for castable tasks as PoC\n\nThe step 3: implement the same for callable tasks (MVP)\n...","commit_id":"486c241583901e8611ced9c7eba166e85548ee5a"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"d2a373bc77783fbc7f98800c349aa4527db43cbc","unresolved":false,"context_lines":[{"line_number":296,"context_line":" * Write the ``WorkQueueDispatcher`` class with"},{"line_number":297,"context_line":"   at least ``ensure()`` or ``distribute()`` method"},{"line_number":298,"context_line":" * Design test cases"},{"line_number":299,"context_line":" * Update work queue use cases documentaion for developers"},{"line_number":300,"context_line":""},{"line_number":301,"context_line":"Documentation Impact"},{"line_number":302,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5a710552_28ce925a","line":299,"in_reply_to":"5a710552_4810be01","updated":"2015-12-16 09:48:38.000000000","message":"The MVP coverage should be for:\nmessaging.WQClient,\n.WQDispatcher.get_wq_server,\nClient.submit_task (AMQP, not IPC),\nServer.start,\nClient.create_worker,\nWorker.start,\n.WQDispatcher.distribute\n\nplus the step one as is","commit_id":"486c241583901e8611ced9c7eba166e85548ee5a"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"3ff9a2a1e7b66f256c3d31f49da15f1cf23e1b1b","unresolved":false,"context_lines":[{"line_number":287,"context_line":"  workers while being processed and *only then* acked to be removed from the"},{"line_number":288,"context_line":"  queue by the AMQP broker. The task exclusive owning logic shall be"},{"line_number":289,"context_line":"  implemented at the server and worker apps side and out of the library scope."},{"line_number":290,"context_line":"  For example, the server may ensure the task to be picked."},{"line_number":291,"context_line":""},{"line_number":292,"context_line":"Alternatives"},{"line_number":293,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":6,"id":"3a7e1126_999c9c10","line":290,"updated":"2015-12-18 15:08:15.000000000","message":"Actually I have no idea how to ensure tasks to be executed as singletons: by not more than one worker for every moment of time.","commit_id":"69f8ae2069f9bb8d69f81a2bd57fa87e9820666b"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"cad5c96dde896c9fa892aeba5713579877eea487","unresolved":false,"context_lines":[{"line_number":287,"context_line":"  workers while being processed and *only then* acked to be removed from the"},{"line_number":288,"context_line":"  queue by the AMQP broker. The task exclusive owning logic shall be"},{"line_number":289,"context_line":"  implemented at the server and worker apps side and out of the library scope."},{"line_number":290,"context_line":"  For example, the server may ensure the task to be picked."},{"line_number":291,"context_line":""},{"line_number":292,"context_line":"Alternatives"},{"line_number":293,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":6,"id":"3a7e1126_c89d6c67","line":290,"in_reply_to":"3a7e1126_999c9c10","updated":"2015-12-18 16:18:43.000000000","message":"Sounds like u need a lock somewhere.","commit_id":"69f8ae2069f9bb8d69f81a2bd57fa87e9820666b"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"cad5c96dde896c9fa892aeba5713579877eea487","unresolved":false,"context_lines":[{"line_number":302,"context_line":"* Rework existing \"at-least-once\" pattern implemented for notifications"},{"line_number":303,"context_line":"  to support a work queue"},{"line_number":304,"context_line":"* Use external task queue solutions. Leave Oslo.messaging for RPC."},{"line_number":305,"context_line":"* Put the jobboard (or similar) concept in tooz and refine it there, leave"},{"line_number":306,"context_line":"  Oslo.messaging for RPC. Because really the jobboard is the task queue. And"},{"line_number":307,"context_line":"  for example celery and taskflow is a lot more built ontop of that jobboard"},{"line_number":308,"context_line":"  or similarish."}],"source_content_type":"text/x-rst","patch_set":6,"id":"3a7e1126_e8714845","line":305,"updated":"2015-12-18 16:18:43.000000000","message":"Lol this is somewhat funny (in how its worded),\n\nPerhaps at least link to http://docs.openstack.org/developer/taskflow/jobs.html","commit_id":"69f8ae2069f9bb8d69f81a2bd57fa87e9820666b"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"77434232963d4e23dd0b652774c71f53475fa6ea","unresolved":false,"context_lines":[{"line_number":302,"context_line":"* Rework existing \"at-least-once\" pattern implemented for notifications"},{"line_number":303,"context_line":"  to support a work queue"},{"line_number":304,"context_line":"* Use external task queue solutions. Leave Oslo.messaging for RPC."},{"line_number":305,"context_line":"* Put the jobboard (or similar) concept in tooz and refine it there, leave"},{"line_number":306,"context_line":"  Oslo.messaging for RPC. Because really the jobboard is the task queue. And"},{"line_number":307,"context_line":"  for example celery and taskflow is a lot more built ontop of that jobboard"},{"line_number":308,"context_line":"  or similarish."}],"source_content_type":"text/x-rst","patch_set":6,"id":"3a7e1126_c847acba","line":305,"in_reply_to":"3a7e1126_e8714845","updated":"2015-12-18 16:19:51.000000000","message":"Api is at:\n\nhttp://docs.openstack.org/developer/taskflow/jobs.html#taskflow.jobs.base.JobBoard\n\nhttp://docs.openstack.org/developer/taskflow/jobs.html#taskflow.jobs.base.Job","commit_id":"69f8ae2069f9bb8d69f81a2bd57fa87e9820666b"},{"author":{"_account_id":8601,"name":"Dmitriy Ukhlov","email":"dukhlov@mirantis.com","username":"dukhlov"},"change_message_id":"5d3779ed6e1a78319066701de8d47115fb5048d3","unresolved":false,"context_lines":[{"line_number":23,"context_line":"From the other side, existing API for rpc can not be changed to the"},{"line_number":24,"context_line":"\"at-least-once\" delivery model as that might render rpc workers to return"},{"line_number":25,"context_line":"duplicated replies or messages being processed twice, which is highly"},{"line_number":26,"context_line":"undesirable for rpc pattern."},{"line_number":27,"context_line":""},{"line_number":28,"context_line":"At least once delivery is implemented for notification pattern already but"},{"line_number":29,"context_line":"Notification pattern can not be used for work queue pattern as is."}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a7e1126_30676bea","line":26,"range":{"start_line":26,"start_character":16,"end_line":26,"end_character":19},"updated":"2015-12-21 18:16:32.000000000","message":"It is not correct a bit. oslo messaging RPC it is low level API which allows to send request to server and receive reply from them. On top of oslo messaging RPC, aplication specific rpc clients could be constructed.\n\nat-least-once or at-most-once logic is not change RPC API, it is defined by RPC server, and could be different for each RPC server/client pair, because only application logic know which behavior is desireble.\n\nI don\u0027t see any problem to make RpcServer configurable to provide ability to end user to construct what they need","commit_id":"c1d7c45e61adf35ed74178ba37c26bea7262b5bc"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"f287538322439cb8e6a2daeac62a9cc92003fd0b","unresolved":false,"context_lines":[{"line_number":23,"context_line":"From the other side, existing API for rpc can not be changed to the"},{"line_number":24,"context_line":"\"at-least-once\" delivery model as that might render rpc workers to return"},{"line_number":25,"context_line":"duplicated replies or messages being processed twice, which is highly"},{"line_number":26,"context_line":"undesirable for rpc pattern."},{"line_number":27,"context_line":""},{"line_number":28,"context_line":"At least once delivery is implemented for notification pattern already but"},{"line_number":29,"context_line":"Notification pattern can not be used for work queue pattern as is."}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a7e1126_3298cb69","line":26,"range":{"start_line":26,"start_character":16,"end_line":26,"end_character":19},"in_reply_to":"3a7e1126_30676bea","updated":"2015-12-21 20:08:01.000000000","message":"These stanzas were consolidated from all of the comments given to the related patch for the ack-then-process mode, see below. I\u0027d prefer to keep that as is to preserve original meaning.","commit_id":"c1d7c45e61adf35ed74178ba37c26bea7262b5bc"},{"author":{"_account_id":8601,"name":"Dmitriy Ukhlov","email":"dukhlov@mirantis.com","username":"dukhlov"},"change_message_id":"5d3779ed6e1a78319066701de8d47115fb5048d3","unresolved":false,"context_lines":[{"line_number":137,"context_line":"  l |  Client.check_task(    Server.stop                   Worker.stop"},{"line_number":138,"context_line":"  o |      futures)"},{"line_number":139,"context_line":"  w v                        Server.wait                   Worker.wait"},{"line_number":140,"context_line":""},{"line_number":141,"context_line":".. [6] The path to the WQServer via AMQP broker was shortened due to"},{"line_number":142,"context_line":"   the visual simplicity reasons. But in fact it is the same as one for"},{"line_number":143,"context_line":"   the WQServer to workers."}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a7e1126_a1530f16","line":140,"updated":"2015-12-21 18:16:32.000000000","message":"A few doubts:\n1) I don\u0027t understand how we will stop/cancel task if we already published it to queue. We need queue to provide simple at-least-once delivery.\n2) It looks like async version of RPC, with stop/cancel posibility which is hard/imposible to implement usinq queue. Probably it would be better to rename rewrite this spec as Async RPC pattern?\n3) if we talk about at-least-once delivery probably we should keep in mind that this calls are most impormant and will have big timeouts. During this time RabbitMQ may crash, client may crash etc. In this case we should provide persistency and think is reply is needed for us.\n\nAs far as I know Mistral operates with work flow, it starts some async, possibly long term job and want to be sure that it will be executed somehow. When some task is done it shedule futher processing by starting new task, updated status in db, if Mistral API was crashed it is not a problem because it dont wait reply, just read current state from db.\n\nI don\u0027t know deep knowledge about Mistral, But that is my undertanding\n\nProbably I missed something, but my current vision is:\nbefore starting work we should undestand how we plan to implement stop/cancel, Do we need exactly, Async RPC, WorkQueue API, durable async task execution.\n2)","commit_id":"c1d7c45e61adf35ed74178ba37c26bea7262b5bc"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"f287538322439cb8e6a2daeac62a9cc92003fd0b","unresolved":false,"context_lines":[{"line_number":137,"context_line":"  l |  Client.check_task(    Server.stop                   Worker.stop"},{"line_number":138,"context_line":"  o |      futures)"},{"line_number":139,"context_line":"  w v                        Server.wait                   Worker.wait"},{"line_number":140,"context_line":""},{"line_number":141,"context_line":".. [6] The path to the WQServer via AMQP broker was shortened due to"},{"line_number":142,"context_line":"   the visual simplicity reasons. But in fact it is the same as one for"},{"line_number":143,"context_line":"   the WQServer to workers."}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a7e1126_92cdff32","line":140,"in_reply_to":"3a7e1126_a1530f16","updated":"2015-12-21 20:08:01.000000000","message":"AFAIK, a task published on the queue as a message remains there unless acked by consumer. Otherwise, the AMQP broker would ensure it to be retained there. Am I right? If so, please read how that works - I described ack/reject(nack) logic and provided example events flow for a call.\n\nNo, that is not an async RPC. That is a work queue pattern. If you think the work queue pattern is async RPC, may be you\u0027re right, I\u0027m not an expert in patterns :)","commit_id":"c1d7c45e61adf35ed74178ba37c26bea7262b5bc"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"0124ba5e6dc6dbfcc1af3169c4621d95d610a2b3","unresolved":false,"context_lines":[{"line_number":184,"context_line":""},{"line_number":185,"context_line":"Crash-stop failure contracts:"},{"line_number":186,"context_line":""},{"line_number":187,"context_line":"* When a worker crashes or violates task processing timeout [#]_, it shall do"},{"line_number":188,"context_line":"  the best to resubmit its task back to the queue. The worker may track its"},{"line_number":189,"context_line":"  local state of accepted and not processed tasks to try to recover after a"},{"line_number":190,"context_line":"  crash-stop failure. The acceptance report message shall be atomic with the"}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a7e1126_20a40006","line":187,"updated":"2015-12-21 16:20:34.000000000","message":"Hmmm, I don\u0027t trust workers that much, but let\u0027s see how this plays out. That\u0027s the nice thing about jobboards, especially when backed by zookeeper, if a worker crashes, due to ephemeral node usage, it automatically is available for others to work on...\n\nhttps://zookeeper.apache.org/doc/r3.2.1/zookeeperProgrammers.html#Ephemeral+Nodes","commit_id":"c1d7c45e61adf35ed74178ba37c26bea7262b5bc"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"f287538322439cb8e6a2daeac62a9cc92003fd0b","unresolved":false,"context_lines":[{"line_number":184,"context_line":""},{"line_number":185,"context_line":"Crash-stop failure contracts:"},{"line_number":186,"context_line":""},{"line_number":187,"context_line":"* When a worker crashes or violates task processing timeout [#]_, it shall do"},{"line_number":188,"context_line":"  the best to resubmit its task back to the queue. The worker may track its"},{"line_number":189,"context_line":"  local state of accepted and not processed tasks to try to recover after a"},{"line_number":190,"context_line":"  crash-stop failure. The acceptance report message shall be atomic with the"}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a7e1126_d2f6275c","line":187,"in_reply_to":"3a7e1126_20a40006","updated":"2015-12-21 20:08:01.000000000","message":"This failure mode only declares the library has nothing to do with the case, while putting some details for app devs though.\nAnyway, that is out of the library scope as each time we say \"app\" here in this spec.","commit_id":"c1d7c45e61adf35ed74178ba37c26bea7262b5bc"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"cc7659cb124caf8c4e377855b7a1312647c98ef8","unresolved":false,"context_lines":[{"line_number":302,"context_line":"* Rework existing \"at-least-once\" pattern implemented for notifications"},{"line_number":303,"context_line":"  to support a work queue"},{"line_number":304,"context_line":"* Use external task queue solutions. Leave Oslo.messaging for RPC."},{"line_number":305,"context_line":"* Put the jobboard (or similar) concept in tooz and refine it there, leave"},{"line_number":306,"context_line":"  Oslo.messaging for RPC. Because really the jobboard is the task queue. And"},{"line_number":307,"context_line":"  for example celery and taskflow is a lot more built ontop of that jobboard"},{"line_number":308,"context_line":"  or similarish."}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a7e1126_353d20dd","line":305,"updated":"2015-12-21 16:17:55.000000000","message":"Oh I still like this (but I\u0027m biased).","commit_id":"c1d7c45e61adf35ed74178ba37c26bea7262b5bc"},{"author":{"_account_id":8601,"name":"Dmitriy Ukhlov","email":"dukhlov@mirantis.com","username":"dukhlov"},"change_message_id":"5d3779ed6e1a78319066701de8d47115fb5048d3","unresolved":false,"context_lines":[{"line_number":302,"context_line":"* Rework existing \"at-least-once\" pattern implemented for notifications"},{"line_number":303,"context_line":"  to support a work queue"},{"line_number":304,"context_line":"* Use external task queue solutions. Leave Oslo.messaging for RPC."},{"line_number":305,"context_line":"* Put the jobboard (or similar) concept in tooz and refine it there, leave"},{"line_number":306,"context_line":"  Oslo.messaging for RPC. Because really the jobboard is the task queue. And"},{"line_number":307,"context_line":"  for example celery and taskflow is a lot more built ontop of that jobboard"},{"line_number":308,"context_line":"  or similarish."}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a7e1126_c680b18d","line":305,"in_reply_to":"3a7e1126_353d20dd","updated":"2015-12-21 18:16:32.000000000","message":"I have feeling this WorkQueue itself shouldn\u0027t be part of oslo.messaging too (probably it could be part of tooz, but I prefer new project). Oslo messaging is mostly about how to organize communication and message delivery.\nAsync RPC and Durable at least once delivery is ok for me.\nBut canceling of started delivery, robust work queue service I feel that to implement it really robust we need to use some key-value storage instead queue and it will far away from messaging, otherwise we risk that this functionality will not be used at all.","commit_id":"c1d7c45e61adf35ed74178ba37c26bea7262b5bc"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"de362889fa75a36bc2c1596a51ac7297afebbbd7","unresolved":false,"context_lines":[{"line_number":302,"context_line":"* Rework existing \"at-least-once\" pattern implemented for notifications"},{"line_number":303,"context_line":"  to support a work queue"},{"line_number":304,"context_line":"* Use external task queue solutions. Leave Oslo.messaging for RPC."},{"line_number":305,"context_line":"* Put the jobboard (or similar) concept in tooz and refine it there, leave"},{"line_number":306,"context_line":"  Oslo.messaging for RPC. Because really the jobboard is the task queue. And"},{"line_number":307,"context_line":"  for example celery and taskflow is a lot more built ontop of that jobboard"},{"line_number":308,"context_line":"  or similarish."}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a7e1126_d2b3874b","line":305,"in_reply_to":"3a7e1126_c680b18d","updated":"2015-12-21 19:43:05.000000000","message":"What u have described is somewhat http://docs.openstack.org/developer/taskflow/jobs.html still, and that doesn\u0027t really use a MQ (because at some point its not a queue and its something else).\n\nI prefer tooz because of tooz current mission statement of \u0027The Tooz project aims at centralizing the most common distributed primitives like group membership protocol, lock service and leader election by providing a coordination API helping developers to build distributed applications.\u0027\n\nAnd imho this is just another \u0027common distributed primitives\u0027\n\nBut it really starts to become a question of what is actually needed here and what isn\u0027t. And to answer that is the million dollar question...","commit_id":"c1d7c45e61adf35ed74178ba37c26bea7262b5bc"},{"author":{"_account_id":13290,"name":"Oleksii Zamiatin","email":"ozamiatin@mirantis.com","username":"ozamiatin"},"change_message_id":"2d387171a3cef5e6e7bc063c072a05ba825df30e","unresolved":false,"context_lines":[{"line_number":364,"context_line":""},{"line_number":365,"context_line":"Target Milestone for completion:"},{"line_number":366,"context_line":""},{"line_number":367,"context_line":"mitaka-1"},{"line_number":368,"context_line":""},{"line_number":369,"context_line":"Work Items"},{"line_number":370,"context_line":"----------"}],"source_content_type":"text/x-rst","patch_set":7,"id":"3a7e1126_003ea427","line":367,"range":{"start_line":367,"start_character":0,"end_line":367,"end_character":6},"updated":"2015-12-21 16:32:35.000000000","message":"I think we already passed this milestone. The most realistic would be M3 IMO.","commit_id":"c1d7c45e61adf35ed74178ba37c26bea7262b5bc"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"f84a06e61ed0a9c574454bcc74f44658233b4bda","unresolved":false,"context_lines":[{"line_number":7,"context_line":"OpenStack projects like Mistral have identified a work queue pattern as a"},{"line_number":8,"context_line":"use case to be supported by abstraction layers like Tooz or Oslo."},{"line_number":9,"context_line":"Historically, Oslo messaging rpc delivery model is an \"at-most-once\". While"},{"line_number":10,"context_line":"for the work queue pattern, API must as well provide the \"at-least-once\""},{"line_number":11,"context_line":"delivery, which has nothing to the existing API for rpc."},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"Problem description"}],"source_content_type":"text/x-rst","patch_set":8,"id":"3a57f1b5_a159a990","line":10,"updated":"2016-02-09 20:48:08.000000000","message":"Actually IMHO one such implementation of the API must do all this complexity to work correctly, the general API @ https://review.openstack.org/#/c/260246/ really doesn\u0027t care about the complexity of a MQ to accomplish that API.","commit_id":"625f2f2dd5406d8d6841babc9f3041f5855a9b08"},{"author":{"_account_id":10239,"name":"Dmitry Tantsur","email":"dtantsur@protonmail.com","username":"dtantsur"},"change_message_id":"fe9b0c646e6251caa65d164abc659bd35f2d67bf","unresolved":false,"context_lines":[{"line_number":8,"context_line":"use case to be supported by abstraction layers like Tooz or Oslo."},{"line_number":9,"context_line":"Historically, Oslo messaging rpc delivery model is an \"at-most-once\". While"},{"line_number":10,"context_line":"for the work queue pattern, API must as well provide the \"at-least-once\""},{"line_number":11,"context_line":"delivery, which has nothing to the existing API for rpc."},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"Problem description"},{"line_number":14,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":8,"id":"5a5ae5dd_e1fc6b74","line":11,"updated":"2016-02-09 11:04:42.000000000","message":"+1, also of potential interest for ironic-inspector","commit_id":"625f2f2dd5406d8d6841babc9f3041f5855a9b08"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"cc7e5f16c6c5e503bb590047122b4c9fc347769f","unresolved":false,"context_lines":[{"line_number":10,"context_line":"for the work queue pattern, API must as well provide the \"at-least-once\""},{"line_number":11,"context_line":"delivery, which has nothing to the existing API for rpc."},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"Problem description"},{"line_number":14,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":15,"context_line":""},{"line_number":16,"context_line":"At most once delivery ensures the \"ack\" to be send before the processing"}],"source_content_type":"text/x-rst","patch_set":8,"id":"bae84128_173f38b5","line":13,"updated":"2016-02-20 02:14:56.000000000","message":"How similar is this to:\n\nhttp://docs.openstack.org/developer/taskflow/workers.html\n\nhttp://docs.openstack.org/developer/taskflow/workers.html#high-level-architecture\n\nPerhaps that code (combined with the mistral code) can be the foundation of this impl, once we figure out what it will finally be.\n\nhttps://github.com/openstack/taskflow/tree/master/taskflow/engines/worker_based (kombu+mq-code-here, worker-code-here...); protocol @ https://github.com/openstack/taskflow/blob/master/taskflow/engines/worker_based/protocol.py","commit_id":"625f2f2dd5406d8d6841babc9f3041f5855a9b08"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"b64cd5c34fe4a752674749bc2ced2a2f9ce00e9e","unresolved":false,"context_lines":[{"line_number":20,"context_line":"That means that existing API for dispatching rpc can not fit needs of a work"},{"line_number":21,"context_line":"queue."},{"line_number":22,"context_line":""},{"line_number":23,"context_line":"From the other side, existing API for rpc can not be changed to the"},{"line_number":24,"context_line":"\"at-least-once\" delivery model as that might render rpc workers to return"},{"line_number":25,"context_line":"duplicated replies or messages being processed twice, which is highly"},{"line_number":26,"context_line":"undesirable for rpc pattern."}],"source_content_type":"text/x-rst","patch_set":8,"id":"fa0719c6_83c16b83","line":23,"range":{"start_line":23,"start_character":38,"end_line":23,"end_character":60},"updated":"2016-03-21 13:44:22.000000000","message":"Note, for Mistral there was a related w/a https://review.openstack.org/#/c/279018 which makes a WQ-like RPC.","commit_id":"625f2f2dd5406d8d6841babc9f3041f5855a9b08"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"321ac2f6abea0fd977aa4481f98b738bda2c2a2f","unresolved":false,"context_lines":[{"line_number":20,"context_line":"That means that existing API for dispatching rpc can not fit needs of a work"},{"line_number":21,"context_line":"queue."},{"line_number":22,"context_line":""},{"line_number":23,"context_line":"From the other side, existing API for rpc can not be changed to the"},{"line_number":24,"context_line":"\"at-least-once\" delivery model as that might render rpc workers to return"},{"line_number":25,"context_line":"duplicated replies or messages being processed twice, which is highly"},{"line_number":26,"context_line":"undesirable for rpc pattern."}],"source_content_type":"text/x-rst","patch_set":8,"id":"bae84128_17dbb87d","line":23,"updated":"2016-02-20 02:21:25.000000000","message":"Ok think now that this likely won\u0027t be in oslo.messaging we no longer need to call this RPC (because its really not remote procedure calls anymore) but instead sending a message to some worker to hopefully do something (which !\u003d RPC).","commit_id":"625f2f2dd5406d8d6841babc9f3041f5855a9b08"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"647b329d1b957ccc9e322780dd731bb97f399de5","unresolved":false,"context_lines":[{"line_number":20,"context_line":"That means that existing API for dispatching rpc can not fit needs of a work"},{"line_number":21,"context_line":"queue."},{"line_number":22,"context_line":""},{"line_number":23,"context_line":"From the other side, existing API for rpc can not be changed to the"},{"line_number":24,"context_line":"\"at-least-once\" delivery model as that might render rpc workers to return"},{"line_number":25,"context_line":"duplicated replies or messages being processed twice, which is highly"},{"line_number":26,"context_line":"undesirable for rpc pattern."}],"source_content_type":"text/x-rst","patch_set":8,"id":"bae84128_55e7f53f","line":23,"in_reply_to":"bae84128_17dbb87d","updated":"2016-02-25 13:48:57.000000000","message":"But that is only a problem description, which is \"existing (Oslo) API for rpc can not be changed\". And in Oslo, rpc\u0027s looks like rpc, it makes an RPC conversaitions and swims like rpc, so we have to call this a duck","commit_id":"625f2f2dd5406d8d6841babc9f3041f5855a9b08"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"a5dee3aabf23bc32de0350157c08bc0dc4486026","unresolved":false,"context_lines":[{"line_number":20,"context_line":"That means that existing API for dispatching rpc can not fit needs of a work"},{"line_number":21,"context_line":"queue."},{"line_number":22,"context_line":""},{"line_number":23,"context_line":"From the other side, existing API for rpc can not be changed to the"},{"line_number":24,"context_line":"\"at-least-once\" delivery model as that might render rpc workers to return"},{"line_number":25,"context_line":"duplicated replies or messages being processed twice, which is highly"},{"line_number":26,"context_line":"undesirable for rpc pattern."}],"source_content_type":"text/x-rst","patch_set":8,"id":"9aed3d3a_551f7107","line":23,"in_reply_to":"bae84128_55e7f53f","updated":"2016-02-29 22:58:27.000000000","message":"Ok its a duck, lol.","commit_id":"625f2f2dd5406d8d6841babc9f3041f5855a9b08"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"7df67999ac87135e497e8799e1ca0671b6edacad","unresolved":false,"context_lines":[{"line_number":36,"context_line":"Oslo.messaging also seems a bad place to provide the work queue API as it"},{"line_number":37,"context_line":"assumes a half of work, like failure modes, locking and state persistence, to"},{"line_number":38,"context_line":"be done by a server, workers and clients, therefore does not belong to the"},{"line_number":39,"context_line":"messaging library scope."},{"line_number":40,"context_line":""},{"line_number":41,"context_line":".. [#] Note that the guarantees are not strong. Duplicates are still possible."},{"line_number":42,"context_line":""}],"source_content_type":"text/x-rst","patch_set":8,"id":"3a57f1b5_7937ebe3","line":39,"updated":"2016-02-09 19:29:08.000000000","message":"+1\n\nTooz seems like the better place.\n\nI believe this is a just a more complex \u0027distributed primitive\u0027 but IMHO still a primitive...","commit_id":"625f2f2dd5406d8d6841babc9f3041f5855a9b08"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"7df67999ac87135e497e8799e1ca0671b6edacad","unresolved":false,"context_lines":[{"line_number":46,"context_line":"Given that said, the solution is to add new/rework API for a work queue as the"},{"line_number":47,"context_line":"Tooz jobboard adoption. There should be new classess:"},{"line_number":48,"context_line":""},{"line_number":49,"context_line":"* `WQClient` with at least ``create_worker``, ``submit_task``, ``wait_task``,"},{"line_number":50,"context_line":"  ``check_task`` or the like. It will controll workers and clients."},{"line_number":51,"context_line":"* `WQDispatcher` with ``get_wq_server`` and internal method ``distribute`` [#]_"},{"line_number":52,"context_line":"  providing the \"at-least-once\" messages delivery model. It may normally return"}],"source_content_type":"text/x-rst","patch_set":8,"id":"3a57f1b5_59af8fe3","line":49,"updated":"2016-02-09 19:29:08.000000000","message":"So these slightly feel out of place in tooz.\n\nIf tooz provides the primitive that is the jobboard (based on a message queue, eck... not what I really want)? why not just build these outside of tooz?","commit_id":"625f2f2dd5406d8d6841babc9f3041f5855a9b08"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"8961d41b4214acf53c6ffa8023e7f7a15f082934","unresolved":false,"context_lines":[{"line_number":46,"context_line":"Given that said, the solution is to add new/rework API for a work queue as the"},{"line_number":47,"context_line":"Tooz jobboard adoption. There should be new classess:"},{"line_number":48,"context_line":""},{"line_number":49,"context_line":"* `WQClient` with at least ``create_worker``, ``submit_task``, ``wait_task``,"},{"line_number":50,"context_line":"  ``check_task`` or the like. It will controll workers and clients."},{"line_number":51,"context_line":"* `WQDispatcher` with ``get_wq_server`` and internal method ``distribute`` [#]_"},{"line_number":52,"context_line":"  providing the \"at-least-once\" messages delivery model. It may normally return"}],"source_content_type":"text/x-rst","patch_set":8,"id":"dae33548_85a40708","line":49,"in_reply_to":"3a57f1b5_59af8fe3","updated":"2016-02-18 12:53:37.000000000","message":"Soo, the WQ on top of MQ shall live outside of Oslo, outside of Tooz, but where then? Each OpenStack project to build its own? But why they\u0027d need Tooz with WQ which is no MQ based?.. I feel confused. The idea is to make a WQ API *somewhere* to support backends like MQ-based or not (configurable in a request perhaps)","commit_id":"625f2f2dd5406d8d6841babc9f3041f5855a9b08"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"74340bf7f0e5de02724680fa6e25eaa53cfc8839","unresolved":false,"context_lines":[{"line_number":46,"context_line":"Given that said, the solution is to add new/rework API for a work queue as the"},{"line_number":47,"context_line":"Tooz jobboard adoption. There should be new classess:"},{"line_number":48,"context_line":""},{"line_number":49,"context_line":"* `WQClient` with at least ``create_worker``, ``submit_task``, ``wait_task``,"},{"line_number":50,"context_line":"  ``check_task`` or the like. It will controll workers and clients."},{"line_number":51,"context_line":"* `WQDispatcher` with ``get_wq_server`` and internal method ``distribute`` [#]_"},{"line_number":52,"context_line":"  providing the \"at-least-once\" messages delivery model. It may normally return"}],"source_content_type":"text/x-rst","patch_set":8,"id":"fa0719c6_e8bad1bf","line":49,"in_reply_to":"9aed3d3a_951579e6","updated":"2016-03-21 08:43:08.000000000","message":"Okay, Mistral folks will review this spec updated and provide their vision I hope. So we could follow-up on specific Tooz WQ PoC topics and decide either MQ based implementation or ???","commit_id":"625f2f2dd5406d8d6841babc9f3041f5855a9b08"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"a5dee3aabf23bc32de0350157c08bc0dc4486026","unresolved":false,"context_lines":[{"line_number":46,"context_line":"Given that said, the solution is to add new/rework API for a work queue as the"},{"line_number":47,"context_line":"Tooz jobboard adoption. There should be new classess:"},{"line_number":48,"context_line":""},{"line_number":49,"context_line":"* `WQClient` with at least ``create_worker``, ``submit_task``, ``wait_task``,"},{"line_number":50,"context_line":"  ``check_task`` or the like. It will controll workers and clients."},{"line_number":51,"context_line":"* `WQDispatcher` with ``get_wq_server`` and internal method ``distribute`` [#]_"},{"line_number":52,"context_line":"  providing the \"at-least-once\" messages delivery model. It may normally return"}],"source_content_type":"text/x-rst","patch_set":8,"id":"9aed3d3a_951579e6","line":49,"in_reply_to":"bae84128_35c5115b","updated":"2016-02-29 22:58:27.000000000","message":"Let\u0027s do it.","commit_id":"625f2f2dd5406d8d6841babc9f3041f5855a9b08"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"647b329d1b957ccc9e322780dd731bb97f399de5","unresolved":false,"context_lines":[{"line_number":46,"context_line":"Given that said, the solution is to add new/rework API for a work queue as the"},{"line_number":47,"context_line":"Tooz jobboard adoption. There should be new classess:"},{"line_number":48,"context_line":""},{"line_number":49,"context_line":"* `WQClient` with at least ``create_worker``, ``submit_task``, ``wait_task``,"},{"line_number":50,"context_line":"  ``check_task`` or the like. It will controll workers and clients."},{"line_number":51,"context_line":"* `WQDispatcher` with ``get_wq_server`` and internal method ``distribute`` [#]_"},{"line_number":52,"context_line":"  providing the \"at-least-once\" messages delivery model. It may normally return"}],"source_content_type":"text/x-rst","patch_set":8,"id":"bae84128_35c5115b","line":49,"in_reply_to":"bae84128_949a2613","updated":"2016-02-25 13:48:57.000000000","message":"Yes, and with interested folks from Ironic and Mistral. I spoke to Renat, he is interested. Let\u0027s try to schedule some brain storming perhaps.","commit_id":"625f2f2dd5406d8d6841babc9f3041f5855a9b08"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"faf2cdd6da4d64608144b6831029c1797888bb97","unresolved":false,"context_lines":[{"line_number":46,"context_line":"Given that said, the solution is to add new/rework API for a work queue as the"},{"line_number":47,"context_line":"Tooz jobboard adoption. There should be new classess:"},{"line_number":48,"context_line":""},{"line_number":49,"context_line":"* `WQClient` with at least ``create_worker``, ``submit_task``, ``wait_task``,"},{"line_number":50,"context_line":"  ``check_task`` or the like. It will controll workers and clients."},{"line_number":51,"context_line":"* `WQDispatcher` with ``get_wq_server`` and internal method ``distribute`` [#]_"},{"line_number":52,"context_line":"  providing the \"at-least-once\" messages delivery model. It may normally return"}],"source_content_type":"text/x-rst","patch_set":8,"id":"bae84128_949a2613","line":49,"in_reply_to":"dae33548_85a40708","updated":"2016-02-20 02:01:40.000000000","message":"We\u0027ll have to work with julien on this also (the other main tooz person).","commit_id":"625f2f2dd5406d8d6841babc9f3041f5855a9b08"},{"author":{"_account_id":10239,"name":"Dmitry Tantsur","email":"dtantsur@protonmail.com","username":"dtantsur"},"change_message_id":"fe9b0c646e6251caa65d164abc659bd35f2d67bf","unresolved":false,"context_lines":[{"line_number":47,"context_line":"Tooz jobboard adoption. There should be new classess:"},{"line_number":48,"context_line":""},{"line_number":49,"context_line":"* `WQClient` with at least ``create_worker``, ``submit_task``, ``wait_task``,"},{"line_number":50,"context_line":"  ``check_task`` or the like. It will controll workers and clients."},{"line_number":51,"context_line":"* `WQDispatcher` with ``get_wq_server`` and internal method ``distribute`` [#]_"},{"line_number":52,"context_line":"  providing the \"at-least-once\" messages delivery model. It may normally return"},{"line_number":53,"context_line":"  nothing, if being casted (in terms of rpc [#]_). When called, it"}],"source_content_type":"text/x-rst","patch_set":8,"id":"5a5ae5dd_217843f8","line":50,"range":{"start_line":50,"start_character":38,"end_line":50,"end_character":46},"updated":"2016-02-09 11:04:42.000000000","message":"nit: typo in control?","commit_id":"625f2f2dd5406d8d6841babc9f3041f5855a9b08"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"7df67999ac87135e497e8799e1ca0671b6edacad","unresolved":false,"context_lines":[{"line_number":48,"context_line":""},{"line_number":49,"context_line":"* `WQClient` with at least ``create_worker``, ``submit_task``, ``wait_task``,"},{"line_number":50,"context_line":"  ``check_task`` or the like. It will controll workers and clients."},{"line_number":51,"context_line":"* `WQDispatcher` with ``get_wq_server`` and internal method ``distribute`` [#]_"},{"line_number":52,"context_line":"  providing the \"at-least-once\" messages delivery model. It may normally return"},{"line_number":53,"context_line":"  nothing, if being casted (in terms of rpc [#]_). When called, it"},{"line_number":54,"context_line":"  may also return an error or event that says \"failed to distribute the task X"}],"source_content_type":"text/x-rst","patch_set":8,"id":"3a57f1b5_d9cfffb1","line":51,"updated":"2016-02-09 19:29:08.000000000","message":"I don\u0027t quite understand how the API (which is really simple) @ https://review.openstack.org/#/c/260246/ cares about these details.\n\nAll this seems like an implementation detail and a very complex and fragile implementation IMHO...\n\nWhy not explain how u map your MQ work queue ontop of those APIs (and tooz will also provide a zookeeper and ... ones for those who do not like fragile implementations)?","commit_id":"625f2f2dd5406d8d6841babc9f3041f5855a9b08"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"8961d41b4214acf53c6ffa8023e7f7a15f082934","unresolved":false,"context_lines":[{"line_number":48,"context_line":""},{"line_number":49,"context_line":"* `WQClient` with at least ``create_worker``, ``submit_task``, ``wait_task``,"},{"line_number":50,"context_line":"  ``check_task`` or the like. It will controll workers and clients."},{"line_number":51,"context_line":"* `WQDispatcher` with ``get_wq_server`` and internal method ``distribute`` [#]_"},{"line_number":52,"context_line":"  providing the \"at-least-once\" messages delivery model. It may normally return"},{"line_number":53,"context_line":"  nothing, if being casted (in terms of rpc [#]_). When called, it"},{"line_number":54,"context_line":"  may also return an error or event that says \"failed to distribute the task X"}],"source_content_type":"text/x-rst","patch_set":8,"id":"dae33548_25537b18","line":51,"in_reply_to":"3a57f1b5_d9cfffb1","updated":"2016-02-18 12:53:37.000000000","message":"I\u0027m not sure how I should update.","commit_id":"625f2f2dd5406d8d6841babc9f3041f5855a9b08"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"2fae26fb5efd0d8a3e4b47fe26078aa587d06a3c","unresolved":false,"context_lines":[{"line_number":48,"context_line":""},{"line_number":49,"context_line":"* `WQClient` with at least ``create_worker``, ``submit_task``, ``wait_task``,"},{"line_number":50,"context_line":"  ``check_task`` or the like. It will controll workers and clients."},{"line_number":51,"context_line":"* `WQDispatcher` with ``get_wq_server`` and internal method ``distribute`` [#]_"},{"line_number":52,"context_line":"  providing the \"at-least-once\" messages delivery model. It may normally return"},{"line_number":53,"context_line":"  nothing, if being casted (in terms of rpc [#]_). When called, it"},{"line_number":54,"context_line":"  may also return an error or event that says \"failed to distribute the task X"}],"source_content_type":"text/x-rst","patch_set":8,"id":"fa0719c6_8b96af92","line":51,"in_reply_to":"bae84128_544c1ec7","updated":"2016-03-21 08:52:38.000000000","message":"I agree, good point, but I can\u0027t do it on my own, I\u0027m not *the* API expert ;(","commit_id":"625f2f2dd5406d8d6841babc9f3041f5855a9b08"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"faf2cdd6da4d64608144b6831029c1797888bb97","unresolved":false,"context_lines":[{"line_number":48,"context_line":""},{"line_number":49,"context_line":"* `WQClient` with at least ``create_worker``, ``submit_task``, ``wait_task``,"},{"line_number":50,"context_line":"  ``check_task`` or the like. It will controll workers and clients."},{"line_number":51,"context_line":"* `WQDispatcher` with ``get_wq_server`` and internal method ``distribute`` [#]_"},{"line_number":52,"context_line":"  providing the \"at-least-once\" messages delivery model. It may normally return"},{"line_number":53,"context_line":"  nothing, if being casted (in terms of rpc [#]_). When called, it"},{"line_number":54,"context_line":"  may also return an error or event that says \"failed to distribute the task X"}],"source_content_type":"text/x-rst","patch_set":8,"id":"bae84128_544c1ec7","line":51,"in_reply_to":"dae33548_25537b18","updated":"2016-02-20 02:01:40.000000000","message":"Focus on the top-level API first, will it work with what exists? if not then what needs tweaking, then care about the low-level impl details...","commit_id":"625f2f2dd5406d8d6841babc9f3041f5855a9b08"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"4a2416ac7d852e88287869674748e7fc87e2c63a","unresolved":false,"context_lines":[{"line_number":53,"context_line":"  nothing, if being casted (in terms of rpc [#]_). When called, it"},{"line_number":54,"context_line":"  may also return an error or event that says \"failed to distribute the task X"},{"line_number":55,"context_line":"  in Y msec\" but even so it cannot guarantee that the corresponding request was"},{"line_number":56,"context_line":"  not in fact committed by the server and processed by a worker."},{"line_number":57,"context_line":""},{"line_number":58,"context_line":".. [#] High level time/flow chart is depicted below. Note, driver API details"},{"line_number":59,"context_line":"  was omitted here. There is no IPC, only distributed hosts communicating via"}],"source_content_type":"text/x-rst","patch_set":8,"id":"bae84128_d4494e8d","line":56,"updated":"2016-02-20 02:06:24.000000000","message":"Why not propose stronger guarantees instead? For example with the zookeeper jobboard I know that posting a job/work to a jobboard will either happen or not, there is no failed to distribute some percentage of work to some percentage of workers, so just wondering, why propose such weak guarantees in the first place? Confuses me as to why people would build or evangelize systems with such weak guarantees as a production grade piece of software....","commit_id":"625f2f2dd5406d8d6841babc9f3041f5855a9b08"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"6d96e69c1db544f31a2e45fac313210705c9cf7f","unresolved":false,"context_lines":[{"line_number":53,"context_line":"  nothing, if being casted (in terms of rpc [#]_). When called, it"},{"line_number":54,"context_line":"  may also return an error or event that says \"failed to distribute the task X"},{"line_number":55,"context_line":"  in Y msec\" but even so it cannot guarantee that the corresponding request was"},{"line_number":56,"context_line":"  not in fact committed by the server and processed by a worker."},{"line_number":57,"context_line":""},{"line_number":58,"context_line":".. [#] High level time/flow chart is depicted below. Note, driver API details"},{"line_number":59,"context_line":"  was omitted here. There is no IPC, only distributed hosts communicating via"}],"source_content_type":"text/x-rst","patch_set":8,"id":"fa0719c6_a8f9f94f","line":56,"in_reply_to":"bae84128_d4494e8d","updated":"2016-03-21 08:46:22.000000000","message":"By the concept of WQ, AFAIK, it is not a big deal to fail and retry, like Agile you know :)","commit_id":"625f2f2dd5406d8d6841babc9f3041f5855a9b08"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"d76cc1b2974233830ba7d03db10c119bea91a9ab","unresolved":false,"context_lines":[{"line_number":53,"context_line":"  nothing, if being casted (in terms of rpc [#]_). When called, it"},{"line_number":54,"context_line":"  may also return an error or event that says \"failed to distribute the task X"},{"line_number":55,"context_line":"  in Y msec\" but even so it cannot guarantee that the corresponding request was"},{"line_number":56,"context_line":"  not in fact committed by the server and processed by a worker."},{"line_number":57,"context_line":""},{"line_number":58,"context_line":".. [#] High level time/flow chart is depicted below. Note, driver API details"},{"line_number":59,"context_line":"  was omitted here. There is no IPC, only distributed hosts communicating via"}],"source_content_type":"text/x-rst","patch_set":8,"id":"fa0719c6_084aa583","line":56,"in_reply_to":"fa0719c6_a8f9f94f","updated":"2016-03-21 08:48:10.000000000","message":"I mean like in real world, if one assigned to a task, it has no guarantees the task to be processed and committed. And that is OK, we\u0027d be able to monitor for results/poll status etc","commit_id":"625f2f2dd5406d8d6841babc9f3041f5855a9b08"},{"author":{"_account_id":10239,"name":"Dmitry Tantsur","email":"dtantsur@protonmail.com","username":"dtantsur"},"change_message_id":"fe9b0c646e6251caa65d164abc659bd35f2d67bf","unresolved":false,"context_lines":[{"line_number":64,"context_line":"  as well implemented without the Oslo messaging rpc (see the kombu, pika or"},{"line_number":65,"context_line":"  the like), it makes sense to reuse existing APIs from Oslo. Although,"},{"line_number":66,"context_line":"  this would require to `introduce \u003chttps://review.openstack.org/#/c/229186/\u003e`_"},{"line_number":67,"context_line":"  the \"at-lest-once\" delivery mode to the Oslo messaging rpc as well."},{"line_number":68,"context_line":""},{"line_number":69,"context_line":"  TODO(bogdando) a decision to be made here!"},{"line_number":70,"context_line":""}],"source_content_type":"text/x-rst","patch_set":8,"id":"5a5ae5dd_015e6776","line":67,"range":{"start_line":67,"start_character":10,"end_line":67,"end_character":14},"updated":"2016-02-09 11:04:42.000000000","message":"typo in least","commit_id":"625f2f2dd5406d8d6841babc9f3041f5855a9b08"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"faf2cdd6da4d64608144b6831029c1797888bb97","unresolved":false,"context_lines":[{"line_number":68,"context_line":""},{"line_number":69,"context_line":"  TODO(bogdando) a decision to be made here!"},{"line_number":70,"context_line":""},{"line_number":71,"context_line":"The WorkQueue API shall implement a lightweight async cast/call flow with call"},{"line_number":72,"context_line":"results returned as futures to track the progress. The contract for the futures"},{"line_number":73,"context_line":"covers only the task distribution (accepted for processing). And doesn\u0027t covers"},{"line_number":74,"context_line":"processing itself. The driver API shall give no strong guarantees for messages"}],"source_content_type":"text/x-rst","patch_set":8,"id":"bae84128_54fa5e43","line":71,"updated":"2016-02-20 02:01:40.000000000","message":"Probably useful for u to read over, http://docs.openstack.org/developer/taskflow/jobs.html#high-level-architecture which provides a impl via zookeeper and redis. The work queue (high level API) should *not* care about such things as async cast/call, those are underlying MQ details.\n\nPerhaps split this document into high-level API changes (if any) and low-level details (such as this?) that are really only specific to a MQ impl (and not zookeeper or redis).","commit_id":"625f2f2dd5406d8d6841babc9f3041f5855a9b08"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"e1f0187eb85f738d086b51aef499441dac0e7b61","unresolved":false,"context_lines":[{"line_number":68,"context_line":""},{"line_number":69,"context_line":"  TODO(bogdando) a decision to be made here!"},{"line_number":70,"context_line":""},{"line_number":71,"context_line":"The WorkQueue API shall implement a lightweight async cast/call flow with call"},{"line_number":72,"context_line":"results returned as futures to track the progress. The contract for the futures"},{"line_number":73,"context_line":"covers only the task distribution (accepted for processing). And doesn\u0027t covers"},{"line_number":74,"context_line":"processing itself. The driver API shall give no strong guarantees for messages"}],"source_content_type":"text/x-rst","patch_set":8,"id":"fa0719c6_ab31f3ee","line":71,"in_reply_to":"bae84128_54fa5e43","updated":"2016-03-21 08:50:35.000000000","message":"For now I can say nothing about high-level API changes, so I only suggested my vision for a mid level details for one of backends - MQ. So I agree here, let\u0027s split to a high-level API details, and mid/low level MQ + ZK + Redis backend details. Although I cannot do it on my own. Let\u0027s follow up, involve interested ppl and update the spec.","commit_id":"625f2f2dd5406d8d6841babc9f3041f5855a9b08"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"6e9f6f3882524a5352bd4dc489cd1b70a9b4f827","unresolved":false,"context_lines":[{"line_number":338,"context_line":"Impact on Existing APIs"},{"line_number":339,"context_line":"-----------------------"},{"line_number":340,"context_line":""},{"line_number":341,"context_line":"The Tooz jobboard API shall be either reworked or extended to support a work"},{"line_number":342,"context_line":"queue pattern as described here. For the latter, there would be no impact on"},{"line_number":343,"context_line":"existing APIs."},{"line_number":344,"context_line":""}],"source_content_type":"text/x-rst","patch_set":8,"id":"3a57f1b5_e481ef4e","line":341,"updated":"2016-02-09 20:57:03.000000000","message":"Or the work queue API shall be reworked to be an implementation of the tooz jobboard API ;)\n\nI prefer that honestly, since the API is already known to work with zookeeper (and redis).","commit_id":"625f2f2dd5406d8d6841babc9f3041f5855a9b08"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"8961d41b4214acf53c6ffa8023e7f7a15f082934","unresolved":false,"context_lines":[{"line_number":338,"context_line":"Impact on Existing APIs"},{"line_number":339,"context_line":"-----------------------"},{"line_number":340,"context_line":""},{"line_number":341,"context_line":"The Tooz jobboard API shall be either reworked or extended to support a work"},{"line_number":342,"context_line":"queue pattern as described here. For the latter, there would be no impact on"},{"line_number":343,"context_line":"existing APIs."},{"line_number":344,"context_line":""}],"source_content_type":"text/x-rst","patch_set":8,"id":"dae33548_05c8f772","line":341,"in_reply_to":"3a57f1b5_e481ef4e","updated":"2016-02-18 12:53:37.000000000","message":"As you say, as far as it supports MQ delivery backends as well","commit_id":"625f2f2dd5406d8d6841babc9f3041f5855a9b08"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"a5dee3aabf23bc32de0350157c08bc0dc4486026","unresolved":false,"context_lines":[{"line_number":338,"context_line":"Impact on Existing APIs"},{"line_number":339,"context_line":"-----------------------"},{"line_number":340,"context_line":""},{"line_number":341,"context_line":"The Tooz jobboard API shall be either reworked or extended to support a work"},{"line_number":342,"context_line":"queue pattern as described here. For the latter, there would be no impact on"},{"line_number":343,"context_line":"existing APIs."},{"line_number":344,"context_line":""}],"source_content_type":"text/x-rst","patch_set":8,"id":"9aed3d3a_15e049f9","line":341,"in_reply_to":"bae84128_9587bd11","updated":"2016-02-29 22:58:27.000000000","message":"Ok","commit_id":"625f2f2dd5406d8d6841babc9f3041f5855a9b08"},{"author":{"_account_id":6926,"name":"Bogdan Dobrelya","email":"bdobreli@redhat.com","username":"bogdando"},"change_message_id":"647b329d1b957ccc9e322780dd731bb97f399de5","unresolved":false,"context_lines":[{"line_number":338,"context_line":"Impact on Existing APIs"},{"line_number":339,"context_line":"-----------------------"},{"line_number":340,"context_line":""},{"line_number":341,"context_line":"The Tooz jobboard API shall be either reworked or extended to support a work"},{"line_number":342,"context_line":"queue pattern as described here. For the latter, there would be no impact on"},{"line_number":343,"context_line":"existing APIs."},{"line_number":344,"context_line":""}],"source_content_type":"text/x-rst","patch_set":8,"id":"bae84128_9587bd11","line":341,"in_reply_to":"dae33548_05c8f772","updated":"2016-02-25 13:48:57.000000000","message":"NOTE, this must not *replace* but enhance API to support MQ as well, as a backend.","commit_id":"625f2f2dd5406d8d6841babc9f3041f5855a9b08"}],"specs/newton/at-least-once-guarantee.rst":[{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":6,"context_line":""},{"line_number":7,"context_line":"OpenStack projects like Mistral have identified a work queue pattern as a"},{"line_number":8,"context_line":"use case to be supported by abstraction layers like Tooz or Oslo."},{"line_number":9,"context_line":"Historically, Oslo messaging rpc delivery model is an \"at-most-once\". While"},{"line_number":10,"context_line":"for the work queue pattern, API must as well provide the \"at-least-once\""},{"line_number":11,"context_line":"delivery, which has nothing to the existing API for rpc."},{"line_number":12,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_64ad969a","line":9,"updated":"2016-07-21 15:39:13.000000000","message":"\"the Oslo RPC delivery model has \"at-most-once\" semantics\".\n\nCapitalize \"RPC\" throughout.","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":7,"context_line":"OpenStack projects like Mistral have identified a work queue pattern as a"},{"line_number":8,"context_line":"use case to be supported by abstraction layers like Tooz or Oslo."},{"line_number":9,"context_line":"Historically, Oslo messaging rpc delivery model is an \"at-most-once\". While"},{"line_number":10,"context_line":"for the work queue pattern, API must as well provide the \"at-least-once\""},{"line_number":11,"context_line":"delivery, which has nothing to the existing API for rpc."},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"Problem description"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_448b321f","line":10,"updated":"2016-07-21 15:39:13.000000000","message":"\u0027The API must also provide \"at-least-once\" delivery\u0027","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":8,"context_line":"use case to be supported by abstraction layers like Tooz or Oslo."},{"line_number":9,"context_line":"Historically, Oslo messaging rpc delivery model is an \"at-most-once\". While"},{"line_number":10,"context_line":"for the work queue pattern, API must as well provide the \"at-least-once\""},{"line_number":11,"context_line":"delivery, which has nothing to the existing API for rpc."},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"Problem description"},{"line_number":14,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_04493ad0","line":11,"updated":"2016-07-21 15:39:13.000000000","message":"\"which is not supported by the existing API\", or \"for which the existing API provides no support\"","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":13,"context_line":"Problem description"},{"line_number":14,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":15,"context_line":""},{"line_number":16,"context_line":"At most once delivery ensures the \"ack\" to be send before the processing"},{"line_number":17,"context_line":"(dispatching) of a message and guarantees there would be no duplicates [#]_."},{"line_number":18,"context_line":"At most once delivery model allows messages to disappear, which is highly"},{"line_number":19,"context_line":"annoying for the work queue pattern there each message is a task for workers."}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_2418bed2","line":16,"updated":"2016-07-21 15:39:13.000000000","message":"s/send/sent","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":16,"context_line":"At most once delivery ensures the \"ack\" to be send before the processing"},{"line_number":17,"context_line":"(dispatching) of a message and guarantees there would be no duplicates [#]_."},{"line_number":18,"context_line":"At most once delivery model allows messages to disappear, which is highly"},{"line_number":19,"context_line":"annoying for the work queue pattern there each message is a task for workers."},{"line_number":20,"context_line":"That means that existing API for dispatching rpc can not fit needs of a work"},{"line_number":21,"context_line":"queue."},{"line_number":22,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_e41a26c5","line":19,"updated":"2016-07-21 15:39:13.000000000","message":"there -\u003e where","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":17,"context_line":"(dispatching) of a message and guarantees there would be no duplicates [#]_."},{"line_number":18,"context_line":"At most once delivery model allows messages to disappear, which is highly"},{"line_number":19,"context_line":"annoying for the work queue pattern there each message is a task for workers."},{"line_number":20,"context_line":"That means that existing API for dispatching rpc can not fit needs of a work"},{"line_number":21,"context_line":"queue."},{"line_number":22,"context_line":""},{"line_number":23,"context_line":"From the other side, existing API for rpc can not be changed to the"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_04777a02","line":20,"updated":"2016-07-21 15:39:13.000000000","message":"\"fit the needs\"","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":21,"context_line":"queue."},{"line_number":22,"context_line":""},{"line_number":23,"context_line":"From the other side, existing API for rpc can not be changed to the"},{"line_number":24,"context_line":"\"at-least-once\" delivery model as that might render rpc workers to return"},{"line_number":25,"context_line":"duplicated replies or messages being processed twice, which is highly"},{"line_number":26,"context_line":"undesirable for rpc pattern."},{"line_number":27,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_78853a58","line":24,"updated":"2016-07-21 15:39:13.000000000","message":"render -\u003e cause","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":22,"context_line":""},{"line_number":23,"context_line":"From the other side, existing API for rpc can not be changed to the"},{"line_number":24,"context_line":"\"at-least-once\" delivery model as that might render rpc workers to return"},{"line_number":25,"context_line":"duplicated replies or messages being processed twice, which is highly"},{"line_number":26,"context_line":"undesirable for rpc pattern."},{"line_number":27,"context_line":""},{"line_number":28,"context_line":"At least once delivery is implemented for notification pattern already but"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_18b99e15","line":25,"updated":"2016-07-21 15:39:13.000000000","message":"\"messages being\" -\u003e \"messages to be\", so it matches \"RPC workers to return\" above.","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":26,"context_line":"undesirable for rpc pattern."},{"line_number":27,"context_line":""},{"line_number":28,"context_line":"At least once delivery is implemented for notification pattern already but"},{"line_number":29,"context_line":"Notification pattern can not be used for work queue pattern as is."},{"line_number":30,"context_line":""},{"line_number":31,"context_line":"The work queue pattern may be used in Oslo messaging by sending a message"},{"line_number":32,"context_line":"with the server value unset in the ``Target``. But it doesn\u0027t fit the"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_b8dcd246","line":29,"updated":"2016-07-21 15:39:13.000000000","message":"Capitalize \"notification\" consistently.\n\n\"[Nn]otification pattern\" -\u003e \"the [Nn]otification pattern\"","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":34,"context_line":"message back in a queue when something goes wrong."},{"line_number":35,"context_line":""},{"line_number":36,"context_line":"Oslo.messaging also seems a bad place to provide the work queue API as it"},{"line_number":37,"context_line":"assumes a half of work, like failure modes, locking and state persistence, to"},{"line_number":38,"context_line":"be done by a server, workers and clients, therefore does not belong to the"},{"line_number":39,"context_line":"messaging library scope."},{"line_number":40,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_b8e692e5","line":37,"updated":"2016-07-21 15:39:13.000000000","message":"\"it assumes half the work ... is done\"","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":43,"context_line":"Proposed change"},{"line_number":44,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":45,"context_line":""},{"line_number":46,"context_line":"Given that said, the solution is to add new/rework API for a work queue as the"},{"line_number":47,"context_line":"Tooz jobboard adoption. There should be new classess:"},{"line_number":48,"context_line":""},{"line_number":49,"context_line":"* `WQClient` with at least ``create_worker``, ``submit_task``, ``wait_task``,"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_d8f6e68d","line":46,"updated":"2016-07-21 15:39:13.000000000","message":"\"that said\" -\u003e \"the above\"","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":47,"context_line":"Tooz jobboard adoption. There should be new classess:"},{"line_number":48,"context_line":""},{"line_number":49,"context_line":"* `WQClient` with at least ``create_worker``, ``submit_task``, ``wait_task``,"},{"line_number":50,"context_line":"  ``check_task`` or the like. It will controll workers and clients."},{"line_number":51,"context_line":"* `WQDispatcher` with ``get_wq_server`` and internal method ``distribute`` [#]_"},{"line_number":52,"context_line":"  providing the \"at-least-once\" messages delivery model. It may normally return"},{"line_number":53,"context_line":"  nothing, if being casted (in terms of rpc [#]_). When called, it"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_78153a9a","line":50,"updated":"2016-07-21 15:39:13.000000000","message":"Typo: \"control\"","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":50,"context_line":"  ``check_task`` or the like. It will controll workers and clients."},{"line_number":51,"context_line":"* `WQDispatcher` with ``get_wq_server`` and internal method ``distribute`` [#]_"},{"line_number":52,"context_line":"  providing the \"at-least-once\" messages delivery model. It may normally return"},{"line_number":53,"context_line":"  nothing, if being casted (in terms of rpc [#]_). When called, it"},{"line_number":54,"context_line":"  may also return an error or event that says \"failed to distribute the task X"},{"line_number":55,"context_line":"  in Y msec\" but even so it cannot guarantee that the corresponding request was"},{"line_number":56,"context_line":"  not in fact committed by the server and processed by a worker."}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_d824260e","line":53,"updated":"2016-07-21 15:39:13.000000000","message":"\"casted\" -\u003e \"cast\"","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":56,"context_line":"  not in fact committed by the server and processed by a worker."},{"line_number":57,"context_line":""},{"line_number":58,"context_line":".. [#] High level time/flow chart is depicted below. Note, driver API details"},{"line_number":59,"context_line":"  was omitted here. There is no IPC, only distributed hosts communicating via"},{"line_number":60,"context_line":"  network in async manner."},{"line_number":61,"context_line":""},{"line_number":62,"context_line":".. [#] It is assumed all conversations across a work queue server, a broker"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_f8700a0d","line":59,"updated":"2016-07-21 15:39:13.000000000","message":"was -\u003e were","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":61,"context_line":""},{"line_number":62,"context_line":".. [#] It is assumed all conversations across a work queue server, a broker"},{"line_number":63,"context_line":"  and workers is done using the rpc cast or call semantics. While this may be"},{"line_number":64,"context_line":"  as well implemented without the Oslo messaging rpc (see the kombu, pika or"},{"line_number":65,"context_line":"  the like), it makes sense to reuse existing APIs from Oslo. Although,"},{"line_number":66,"context_line":"  this would require to `introduce \u003chttps://review.openstack.org/#/c/229186/\u003e`_"},{"line_number":67,"context_line":"  the \"at-lest-once\" delivery mode to the Oslo messaging rpc as well."}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_38fc623d","line":64,"updated":"2016-07-21 15:39:13.000000000","message":"s/the // - you don\u0027t need articles with proper nouns.","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":63,"context_line":"  and workers is done using the rpc cast or call semantics. While this may be"},{"line_number":64,"context_line":"  as well implemented without the Oslo messaging rpc (see the kombu, pika or"},{"line_number":65,"context_line":"  the like), it makes sense to reuse existing APIs from Oslo. Although,"},{"line_number":66,"context_line":"  this would require to `introduce \u003chttps://review.openstack.org/#/c/229186/\u003e`_"},{"line_number":67,"context_line":"  the \"at-lest-once\" delivery mode to the Oslo messaging rpc as well."},{"line_number":68,"context_line":""},{"line_number":69,"context_line":"  TODO(bogdando) a decision to be made here!"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_3893c2fb","line":66,"updated":"2016-07-21 15:39:13.000000000","message":"\"require us to\"","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":64,"context_line":"  as well implemented without the Oslo messaging rpc (see the kombu, pika or"},{"line_number":65,"context_line":"  the like), it makes sense to reuse existing APIs from Oslo. Although,"},{"line_number":66,"context_line":"  this would require to `introduce \u003chttps://review.openstack.org/#/c/229186/\u003e`_"},{"line_number":67,"context_line":"  the \"at-lest-once\" delivery mode to the Oslo messaging rpc as well."},{"line_number":68,"context_line":""},{"line_number":69,"context_line":"  TODO(bogdando) a decision to be made here!"},{"line_number":70,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_f88c8adb","line":67,"updated":"2016-07-21 15:39:13.000000000","message":"Typo: \"least\"","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":70,"context_line":""},{"line_number":71,"context_line":"The WorkQueue API shall implement a lightweight async cast/call flow with call"},{"line_number":72,"context_line":"results returned as futures to track the progress. The contract for the futures"},{"line_number":73,"context_line":"covers only the task distribution (accepted for processing). And doesn\u0027t covers"},{"line_number":74,"context_line":"processing itself. The driver API shall give no strong guarantees for messages"},{"line_number":75,"context_line":"delivery. Messages may be lost in distributed systems. As well as duplicated or"},{"line_number":76,"context_line":"delivered in an arbitrary order [#]_. Therefore, the ``distribute``  method"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_986f4efb","line":73,"updated":"2016-07-21 15:39:13.000000000","message":"\". And\" -\u003e \" but\"\n\n\"covers\" -\u003e \"cover\"","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":72,"context_line":"results returned as futures to track the progress. The contract for the futures"},{"line_number":73,"context_line":"covers only the task distribution (accepted for processing). And doesn\u0027t covers"},{"line_number":74,"context_line":"processing itself. The driver API shall give no strong guarantees for messages"},{"line_number":75,"context_line":"delivery. Messages may be lost in distributed systems. As well as duplicated or"},{"line_number":76,"context_line":"delivered in an arbitrary order [#]_. Therefore, the ``distribute``  method"},{"line_number":77,"context_line":"shall guarantee a `relaxed` serializable [#]_ \"at-least-once\" delivery for"},{"line_number":78,"context_line":"tasks to be distributed."}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_983d8edf","line":75,"updated":"2016-07-21 15:39:13.000000000","message":"\". As\" -\u003e \", as\"","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":82,"context_line":"   is no way to retain the original order of tasks as well."},{"line_number":83,"context_line":""},{"line_number":84,"context_line":".. [#] In distributed systems world, serializable means that the history of"},{"line_number":85,"context_line":"   all operations is equivalent to one that took place in some single atomic"},{"line_number":86,"context_line":"   order but with unknown invocation and completion times - it places no bounds"},{"line_number":87,"context_line":"   on time or order of operations. Here it is relaxed as there may be"},{"line_number":88,"context_line":"   duplicating operations."}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_1b187874","line":85,"updated":"2016-07-21 15:39:13.000000000","message":"is \"atomic\" right here? \"Serial\", maybe?","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":87,"context_line":"   on time or order of operations. Here it is relaxed as there may be"},{"line_number":88,"context_line":"   duplicating operations."},{"line_number":89,"context_line":""},{"line_number":90,"context_line":"Tooz jobs adoption must abide execution contracts but it should not handle"},{"line_number":91,"context_line":"duplicated or lost [#]_ or delivered out of order messages to a work queue."},{"line_number":92,"context_line":"That should be the app side responsibility instead. The only way to handle"},{"line_number":93,"context_line":"those properly is to handle at the app server level in an atomic way - meaning"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_7b50940b","line":90,"updated":"2016-07-21 15:39:13.000000000","message":"\"abide by\"","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":96,"context_line":"implement request idempotence and atomicity properly on the server side."},{"line_number":97,"context_line":""},{"line_number":98,"context_line":"Tasks with the result tracking contracts (futures) are `called tasks`."},{"line_number":99,"context_line":"Otherwise, when result is not important, those shall be `casted`. Note that"},{"line_number":100,"context_line":"result tracking contracts apply only up to the moment of the task acceptance"},{"line_number":101,"context_line":"and shall do nothing with the task processing."},{"line_number":102,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_5bef901d","line":99,"updated":"2016-07-21 15:39:13.000000000","message":"Is this a standard term? The usual past tense of \"to cast\" is \"cast\".","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":100,"context_line":"result tracking contracts apply only up to the moment of the task acceptance"},{"line_number":101,"context_line":"and shall do nothing with the task processing."},{"line_number":102,"context_line":""},{"line_number":103,"context_line":"The only SLA for contracts is abiding the given distribute timeout SLA."},{"line_number":104,"context_line":"It shall be evaluated by the WQServer (dispatcher) based on the given SLA data"},{"line_number":105,"context_line":"structure which consist of accept timeout SLA, process timeout SLA and number"},{"line_number":106,"context_line":"of retries. With the total `distribute timeout` ``T`` (which is also an"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_dbc26084","line":103,"updated":"2016-07-21 15:39:13.000000000","message":"\"abiding by\", though you could just say \"the only SLA for contracts is the given distribute timeout SLA\".","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":102,"context_line":""},{"line_number":103,"context_line":"The only SLA for contracts is abiding the given distribute timeout SLA."},{"line_number":104,"context_line":"It shall be evaluated by the WQServer (dispatcher) based on the given SLA data"},{"line_number":105,"context_line":"structure which consist of accept timeout SLA, process timeout SLA and number"},{"line_number":106,"context_line":"of retries. With the total `distribute timeout` ``T`` (which is also an"},{"line_number":107,"context_line":"`owner expiration timeout`) ::"},{"line_number":108,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_5b3dd08d","line":105,"updated":"2016-07-21 15:39:13.000000000","message":"consist -\u003e consists","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":108,"context_line":""},{"line_number":109,"context_line":"  T \u003d process_timeout + (accept_timeout * (retries + 1))"},{"line_number":110,"context_line":""},{"line_number":111,"context_line":"Here the process timeout is only required for the failure detection and has"},{"line_number":112,"context_line":"nothing to the task process results tracking. Accept timeout is how long the"},{"line_number":113,"context_line":"task is allowed to wait unless picked up by a worker. And the number of retries"},{"line_number":114,"context_line":"is how many times the task is allowed to be returned to the queue for another"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_1b1518e4","line":111,"updated":"2016-07-21 15:39:13.000000000","message":"\"has nothing to\" is not an idiom I know. \"is not used for\" or \"is not related to\", maybe?","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":114,"context_line":"is how many times the task is allowed to be returned to the queue for another"},{"line_number":115,"context_line":"worker to pickup and retry processing."},{"line_number":116,"context_line":""},{"line_number":117,"context_line":"A task owning shall normally allow a task to be executed by a single worker or"},{"line_number":118,"context_line":"several workers when something went wrong. This allows a task to be executed"},{"line_number":119,"context_line":"more than once. This is known as efficiency based distributed locking and does"},{"line_number":120,"context_line":"not require from a DLM backend additional complexity of correctness based"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_9bdf4839","line":117,"updated":"2016-07-21 15:39:13.000000000","message":"owning -\u003e owner?","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":179,"context_line":"Failure modes"},{"line_number":180,"context_line":"-------------"},{"line_number":181,"context_line":""},{"line_number":182,"context_line":"Distribution and processing contracts (assumes none crash-stop failures):"},{"line_number":183,"context_line":""},{"line_number":184,"context_line":"* On the driver API level, the worker shall nack/reject the message back on to"},{"line_number":185,"context_line":"  the queue if it can\u0027t process it for the given timeout or if it raised an"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_bb816cfd","line":182,"updated":"2016-07-21 15:39:13.000000000","message":"none -\u003e no","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":183,"context_line":""},{"line_number":184,"context_line":"* On the driver API level, the worker shall nack/reject the message back on to"},{"line_number":185,"context_line":"  the queue if it can\u0027t process it for the given timeout or if it raised an"},{"line_number":186,"context_line":"  exception. The worker shall atomicly report to server upon the task accepted"},{"line_number":187,"context_line":"  for processing. The worker shall ack the message to the broker once related"},{"line_number":188,"context_line":"  task processed in time."},{"line_number":189,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_1ba87876","line":186,"updated":"2016-07-21 15:39:13.000000000","message":"\"when the task is accepted\"","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":187,"context_line":"  for processing. The worker shall ack the message to the broker once related"},{"line_number":188,"context_line":"  task processed in time."},{"line_number":189,"context_line":""},{"line_number":190,"context_line":"* On the WQDispatcher server level, there are no contracts for casted tasks."},{"line_number":191,"context_line":"  There are no processing but accept only contracts for called tasks."},{"line_number":192,"context_line":"  The called task being submitted shall be dropped and the given"},{"line_number":193,"context_line":"  the error shall be passed if the futures checked, if the task violated its"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_3bc45c3e","line":190,"updated":"2016-07-21 15:39:13.000000000","message":"casted -\u003e cast, subject to outcome of the discussion above.","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":190,"context_line":"* On the WQDispatcher server level, there are no contracts for casted tasks."},{"line_number":191,"context_line":"  There are no processing but accept only contracts for called tasks."},{"line_number":192,"context_line":"  The called task being submitted shall be dropped and the given"},{"line_number":193,"context_line":"  the error shall be passed if the futures checked, if the task violated its"},{"line_number":194,"context_line":"  distribution SLA ``T``. The only fact of a successful distribution is"},{"line_number":195,"context_line":"  (a) at least one `accept` message received from a worker by the"},{"line_number":196,"context_line":"  ``accept_timeout`` AND (b) at most ``retries+1`` number of `accept` messages"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_fbfbe476","line":193,"updated":"2016-07-21 15:39:13.000000000","message":"\"the given the error\" -\u003e \"the given error\"","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":197,"context_line":"  received from workers [#]_ by the total distribution timeout SLA ``T``"},{"line_number":198,"context_line":""},{"line_number":199,"context_line":"* On the user API level, the called task shall fail after the distribution SLA"},{"line_number":200,"context_line":"  timeout ``T``, if failed processing or distributed or by other issues down"},{"line_number":201,"context_line":"  the way. Casted ones shall not be trackable by either meanings."},{"line_number":202,"context_line":""},{"line_number":203,"context_line":"* An async call may be checked by the returned futures and shall be executed"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_5b1830d3","line":200,"updated":"2016-07-21 15:39:13.000000000","message":"\"if it failed processing\"","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":198,"context_line":""},{"line_number":199,"context_line":"* On the user API level, the called task shall fail after the distribution SLA"},{"line_number":200,"context_line":"  timeout ``T``, if failed processing or distributed or by other issues down"},{"line_number":201,"context_line":"  the way. Casted ones shall not be trackable by either meanings."},{"line_number":202,"context_line":""},{"line_number":203,"context_line":"* An async call may be checked by the returned futures and shall be executed"},{"line_number":204,"context_line":"  by the timeout SLA w/o errors or with no time bounds but with the returned"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_bb0b2c7c","line":201,"updated":"2016-07-21 15:39:13.000000000","message":"meanings -\u003e means","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":205,"context_line":"  error passed from the server."},{"line_number":206,"context_line":""},{"line_number":207,"context_line":"* For consequent tasks ``A, B`` it may be executed in arbitrary order and at"},{"line_number":208,"context_line":"  lest one time, for example ``B, A, B``, or ``A, B, A``, or ``B, A`` or"},{"line_number":209,"context_line":"  ``A, B``. Which all are valid and meet the processing contracts."},{"line_number":210,"context_line":""},{"line_number":211,"context_line":"* The processing flow should be agnostic to the messaging driver and its"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_bb820cc6","line":208,"updated":"2016-07-21 15:39:13.000000000","message":"lest -\u003e least","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":210,"context_line":""},{"line_number":211,"context_line":"* The processing flow should be agnostic to the messaging driver and its"},{"line_number":212,"context_line":"  ack-then-process or process-then-ack modes. The only difference is that the"},{"line_number":213,"context_line":"  latter has stronger task execution guarantees then the former one."},{"line_number":214,"context_line":""},{"line_number":215,"context_line":".. [#] Workers may ping-pong a message back to the queue due to exceptions"},{"line_number":216,"context_line":"   raised for this particular message with each worker. These must be limited"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_9bd3c8d4","line":213,"updated":"2016-07-21 15:39:13.000000000","message":"then -\u003e than","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":225,"context_line":"  local state commit. It\u0027s up to the worker app how to recover and is out of"},{"line_number":226,"context_line":"  the scope of the Tooz jobboard."},{"line_number":227,"context_line":""},{"line_number":228,"context_line":"* Then AMQP broker crashes or fails to process publish/consume/(n)ack/reject"},{"line_number":229,"context_line":"  operations to work queues, it shall do the best to recover its state."},{"line_number":230,"context_line":"  But there is no contracts between the broker and workers and a work queue"},{"line_number":231,"context_line":"  server. It\u0027s up to the broker how to recover and is out of the scope of the"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_bbb5ec7e","line":228,"updated":"2016-07-21 15:39:13.000000000","message":"\"Then\" -\u003e \"When\"?","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":231,"context_line":"  server. It\u0027s up to the broker how to recover and is out of the scope of the"},{"line_number":232,"context_line":"  Tooz jobboard."},{"line_number":233,"context_line":""},{"line_number":234,"context_line":"* Then the work queue server crashes of fails to distribute a task by the SLA"},{"line_number":235,"context_line":"  timeout, it shall do the best to signal the caller about the failure and do"},{"line_number":236,"context_line":"  nothing for the caster. The server may track its local state of submitted by"},{"line_number":237,"context_line":"  a caller and not distributed tasks to try to recover after a crash-stop."}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_fbb3e495","line":234,"updated":"2016-07-21 15:39:13.000000000","message":"\"Then\" -\u003e \"When\"?\n\nof -\u003e or","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":233,"context_line":""},{"line_number":234,"context_line":"* Then the work queue server crashes of fails to distribute a task by the SLA"},{"line_number":235,"context_line":"  timeout, it shall do the best to signal the caller about the failure and do"},{"line_number":236,"context_line":"  nothing for the caster. The server may track its local state of submitted by"},{"line_number":237,"context_line":"  a caller and not distributed tasks to try to recover after a crash-stop."},{"line_number":238,"context_line":"  The final decision (if a task\u0027s distributed) shall be atomic with the local"},{"line_number":239,"context_line":"  state commit. It\u0027s up to the server app how to recover and is out of the scope"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_7b3414fb","line":236,"updated":"2016-07-21 15:39:13.000000000","message":"This sentence isn\u0027t very clear - maybe \"The server may locally track which states have been submitted but not distributed, so it can try to recover after a crash-stop\".","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":235,"context_line":"  timeout, it shall do the best to signal the caller about the failure and do"},{"line_number":236,"context_line":"  nothing for the caster. The server may track its local state of submitted by"},{"line_number":237,"context_line":"  a caller and not distributed tasks to try to recover after a crash-stop."},{"line_number":238,"context_line":"  The final decision (if a task\u0027s distributed) shall be atomic with the local"},{"line_number":239,"context_line":"  state commit. It\u0027s up to the server app how to recover and is out of the scope"},{"line_number":240,"context_line":"  of the Tooz jobboard."},{"line_number":241,"context_line":""}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_ae6c8808","line":238,"updated":"2016-07-21 15:39:13.000000000","message":"I think \"a task has been distributed\" would be a bit clearer\"","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":239,"context_line":"  state commit. It\u0027s up to the server app how to recover and is out of the scope"},{"line_number":240,"context_line":"  of the Tooz jobboard."},{"line_number":241,"context_line":""},{"line_number":242,"context_line":"* Then the client app crashes or raises exceptions at any point, it\u0027s up to"},{"line_number":243,"context_line":"  it how to recover and is out of the scope of the Tooz jobboard."},{"line_number":244,"context_line":""},{"line_number":245,"context_line":"* A crash-stop failure leaves no strong guarantees to a ``task_check``"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_3d8e0c2e","line":242,"updated":"2016-07-21 15:39:13.000000000","message":"s/Then/When","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":243,"context_line":"  it how to recover and is out of the scope of the Tooz jobboard."},{"line_number":244,"context_line":""},{"line_number":245,"context_line":"* A crash-stop failure leaves no strong guarantees to a ``task_check``"},{"line_number":246,"context_line":"  performed by the futures. The reported failure might not ensure there no task"},{"line_number":247,"context_line":"  was executed, nor reported success might not ensure a task was executed."},{"line_number":248,"context_line":"  It might cause a task to be lost or duplicated as well. Perhaps an"},{"line_number":249,"context_line":"  application may want to fail back to the cast, if call results cannot be"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_bd7afc21","line":246,"updated":"2016-07-21 15:39:13.000000000","message":"s/there/that","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":272,"context_line":"               |            +--------------------------\u003e"},{"line_number":273,"context_line":"               |            | a\u0027\u0027        |      |      |"},{"line_number":274,"context_line":"               |            +----\u003e       |      |      |"},{"line_number":275,"context_line":"               |            |    |       |  ACCPT    EXPIRED"},{"line_number":276,"context_line":"  +------------------------------+----+--+------+----------+"},{"line_number":277,"context_line":"  |Call a task | Distribute | Process | check1  |  check2  |"},{"line_number":278,"context_line":"  |   CLIENT   |   SERVER   | WORKER1 | CLIENT  |  CLIENT  |"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_9d3a98e5","line":275,"updated":"2016-07-21 15:39:13.000000000","message":"ACCPT -\u003e ACCEPT?","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":275,"context_line":"               |            |    |       |  ACCPT    EXPIRED"},{"line_number":276,"context_line":"  +------------------------------+----+--+------+----------+"},{"line_number":277,"context_line":"  |Call a task | Distribute | Process | check1  |  check2  |"},{"line_number":278,"context_line":"  |   CLIENT   |   SERVER   | WORKER1 | CLIENT  |  CLIENT  |"},{"line_number":279,"context_line":"  +------------+------------+---------+---------+------+---+"},{"line_number":280,"context_line":"   acc_timeout:a,               (crash)                |"},{"line_number":281,"context_line":"   proc_timeout:p,               +---------+ +---------+"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_9d16174a","line":278,"updated":"2016-07-21 15:39:13.000000000","message":"Could you arrange the different machines on the Y-axis?","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":293,"context_line":"  distribution SLA, even though the condition (a) of the distribution contract"},{"line_number":294,"context_line":"  have been met (see the failure modes section above)."},{"line_number":295,"context_line":""},{"line_number":296,"context_line":"* At some point, the `WORKER1` has been crashed. As the message remained in the"},{"line_number":297,"context_line":"  queue unacked, it was picked by the `WORKER2` by the ``a\u0027+a\u0027\u0027`` time [#]_."},{"line_number":298,"context_line":"  And the server has received the second `accept` message by that time. Had"},{"line_number":299,"context_line":"  it been given even more, it would have exited its timer loop immediately and"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_9d3d77c6","line":296,"updated":"2016-07-21 15:39:13.000000000","message":"s/been crashed/crashed","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":297,"context_line":"  queue unacked, it was picked by the `WORKER2` by the ``a\u0027+a\u0027\u0027`` time [#]_."},{"line_number":298,"context_line":"  And the server has received the second `accept` message by that time. Had"},{"line_number":299,"context_line":"  it been given even more, it would have exited its timer loop immediately and"},{"line_number":300,"context_line":"  ensured the futex to report something like ``SLAFAIL`` to the future checks"},{"line_number":301,"context_line":"  as there would be no more retries allowed by the task SLA."},{"line_number":302,"context_line":""},{"line_number":303,"context_line":"* While being processed by the `WORKER2` the client issued the task check #1 on"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_3da72b0c","line":300,"updated":"2016-07-21 15:39:13.000000000","message":"\"futex\" -\u003e \"future\"?","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":304,"context_line":"  the futures. And by that point, the result was ``ACCPT``, which is accepted"},{"line_number":305,"context_line":"  for processing. It may contain info by which worker as well."},{"line_number":306,"context_line":""},{"line_number":307,"context_line":"* Needless to say that that unlucky task rendered the `WORKER2` to crash as"},{"line_number":308,"context_line":"  well. And the timestamp of this sad event was beyond the expected SLA ``T``."},{"line_number":309,"context_line":"  That is why the next `WORKER3` picked up the task never tried to report it"},{"line_number":310,"context_line":"  accepted and process. It just acked it to remove from the queue and dropped"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_9d6bb7b6","line":307,"updated":"2016-07-21 15:39:13.000000000","message":"rendered -\u003e caused","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":315,"context_line":"  client\u0027s check #2."},{"line_number":316,"context_line":""},{"line_number":317,"context_line":"* The effective processing time, if we may call it so, was ``p\u0027``. But that"},{"line_number":318,"context_line":"  is beyond of the things SLA cares about anyway."},{"line_number":319,"context_line":""},{"line_number":320,"context_line":".. [#] Note, that a task normally shall not be picked by other workers within"},{"line_number":321,"context_line":"  a granted owner expiration timeout, which is the ``T``. For this example,"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_9dc73795","line":318,"updated":"2016-07-21 15:39:13.000000000","message":"\"beyond of\" -\u003e \"beyond the scope of\", or \"not one of\"","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":320,"context_line":".. [#] Note, that a task normally shall not be picked by other workers within"},{"line_number":321,"context_line":"  a granted owner expiration timeout, which is the ``T``. For this example,"},{"line_number":322,"context_line":"  the task may be picked by other workers when ``p\u0027\u003eT``. But as noted above,"},{"line_number":323,"context_line":"  there is no guarantees it never will be picked be several workers in the"},{"line_number":324,"context_line":"  face of network delays or an application freezes."},{"line_number":325,"context_line":""},{"line_number":326,"context_line":"Alternatives"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_fd0ab3d0","line":323,"updated":"2016-07-21 15:39:13.000000000","message":"\"guarantees\" -\u003e \"guarantee\"","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":332,"context_line":"  persistent state to Client/Worker/WQServer instances)."},{"line_number":333,"context_line":"* Add WQ API to the Oslo.messaging to support a work queue with"},{"line_number":334,"context_line":"  the \"at-least-once\" delivery mode."},{"line_number":335,"context_line":"* Rework Oslo.messaging \"at-least-once\" pattern implemented for notifications"},{"line_number":336,"context_line":"  to support a work queue"},{"line_number":337,"context_line":""},{"line_number":338,"context_line":"Impact on Existing APIs"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_9dd1f725","line":335,"updated":"2016-07-21 15:39:13.000000000","message":"At-least-once semantics won\u0027t make sense in many cases, but I guess that was your point...","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":360,"context_line":"Developer Impact"},{"line_number":361,"context_line":"----------------"},{"line_number":362,"context_line":""},{"line_number":363,"context_line":"OpenStack components developers will have to care for handling dulicated or"},{"line_number":364,"context_line":"lost or delivered in an unexpected order tasks to a work queue."},{"line_number":365,"context_line":""},{"line_number":366,"context_line":"Testing Impact"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_bdc2fb50","line":363,"updated":"2016-07-21 15:39:13.000000000","message":"Typo: \"duplicated\"","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"},{"author":{"_account_id":18893,"name":"Miles Gould","email":"mgould@redhat.com","username":"pozorvlak"},"change_message_id":"d86d898b143d527bd96367b02e3478e04040a179","unresolved":false,"context_lines":[{"line_number":361,"context_line":"----------------"},{"line_number":362,"context_line":""},{"line_number":363,"context_line":"OpenStack components developers will have to care for handling dulicated or"},{"line_number":364,"context_line":"lost or delivered in an unexpected order tasks to a work queue."},{"line_number":365,"context_line":""},{"line_number":366,"context_line":"Testing Impact"},{"line_number":367,"context_line":"--------------"}],"source_content_type":"text/x-rst","patch_set":9,"id":"dada55a8_9d7f3714","line":364,"updated":"2016-07-21 15:39:13.000000000","message":"\"developers will have to care about handling tasks that have been duplicated, lost or delivered to the work queue out-of-order\"","commit_id":"4693d2792f68fed149bf7243df1dd3e4ff258d1e"}]}