)]}'
{"specs/pike/approved/mistral-execution-event-subscription.rst":[{"author":{"_account_id":20863,"name":"Sharat Sharma","email":"sharat.sharma@in.ibm.com","username":"sharat.sharma"},"change_message_id":"3e06fd96af06ef3a3c9eb053670d49bd861e4f96","unresolved":false,"context_lines":[{"line_number":13,"context_line":"https://blueprints.launchpad.net/mistral/+spec/mistral-execution-event-subscription"},{"line_number":14,"context_line":""},{"line_number":15,"context_line":"This specification describes a new service component to allow system"},{"line_number":16,"context_line":"operators to subsribe to and consume events related to workflow execution"},{"line_number":17,"context_line":"such as workflow completion event."},{"line_number":18,"context_line":""},{"line_number":19,"context_line":"Problem description"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bff0334d_85193623","line":16,"range":{"start_line":16,"start_character":13,"end_line":16,"end_character":21},"updated":"2017-04-10 10:11:59.000000000","message":"subscribe","commit_id":"e40b40d2fd8ccd4db136cd61fe4129ff8204d1b8"},{"author":{"_account_id":20863,"name":"Sharat Sharma","email":"sharat.sharma@in.ibm.com","username":"sharat.sharma"},"change_message_id":"3e06fd96af06ef3a3c9eb053670d49bd861e4f96","unresolved":false,"context_lines":[{"line_number":25,"context_line":"query workflow execution state and that proves to be unscalable with"},{"line_number":26,"context_line":"increasing amount of queries against the Mistral API. If there is a"},{"line_number":27,"context_line":"mechanism for Mistral to make a callback to the external system, the"},{"line_number":28,"context_line":"integration will be more timely, efficient, stable, and scalable. "},{"line_number":29,"context_line":""},{"line_number":30,"context_line":"Use Cases"},{"line_number":31,"context_line":"---------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bff0334d_c5fa9ebb","line":28,"range":{"start_line":28,"start_character":65,"end_line":28,"end_character":66},"updated":"2017-04-10 10:11:59.000000000","message":"remove this","commit_id":"e40b40d2fd8ccd4db136cd61fe4129ff8204d1b8"},{"author":{"_account_id":20863,"name":"Sharat Sharma","email":"sharat.sharma@in.ibm.com","username":"sharat.sharma"},"change_message_id":"3e06fd96af06ef3a3c9eb053670d49bd861e4f96","unresolved":false,"context_lines":[{"line_number":79,"context_line":"execution. Make a POST request to the webhook URL in the decorator at each"},{"line_number":80,"context_line":"decorated methods. The event subscription is per workflow execution. But the"},{"line_number":81,"context_line":"mechanism on notification (i.e. POST to webhook) is limited and any credential"},{"line_number":82,"context_line":"required to post to the webhook is duplicated in plain text thru out the"},{"line_number":83,"context_line":"database."},{"line_number":84,"context_line":""},{"line_number":85,"context_line":"2. Publish the event to the message queue in the decorator and call it done."}],"source_content_type":"text/x-rst","patch_set":1,"id":"bff0334d_a5ce7a7f","line":82,"range":{"start_line":82,"start_character":60,"end_line":82,"end_character":64},"updated":"2017-04-10 10:11:59.000000000","message":"through","commit_id":"e40b40d2fd8ccd4db136cd61fe4129ff8204d1b8"},{"author":{"_account_id":20863,"name":"Sharat Sharma","email":"sharat.sharma@in.ibm.com","username":"sharat.sharma"},"change_message_id":"3e06fd96af06ef3a3c9eb053670d49bd861e4f96","unresolved":false,"context_lines":[{"line_number":109,"context_line":""},{"line_number":110,"context_line":"Additional database table and transactions to process records. Additional"},{"line_number":111,"context_line":"server process to process events. CPU, memory, storage, and network IO require"},{"line_number":112,"context_line":"to accomodate the additional database transactions and server process."},{"line_number":113,"context_line":""},{"line_number":114,"context_line":"Deployer impact"},{"line_number":115,"context_line":"---------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"bff0334d_0582c600","line":112,"range":{"start_line":112,"start_character":3,"end_line":112,"end_character":13},"updated":"2017-04-10 10:11:59.000000000","message":"accommodate","commit_id":"e40b40d2fd8ccd4db136cd61fe4129ff8204d1b8"},{"author":{"_account_id":8731,"name":"Renat Akhmerov","email":"renat.akhmerov@gmail.com","username":"rakhmerov"},"change_message_id":"7d31ad5c349f39ba686e4b4aa9ce92be0ad3ef02","unresolved":false,"context_lines":[{"line_number":38,"context_line":""},{"line_number":39,"context_line":"1. Workflow execution event publishing"},{"line_number":40,"context_line":""},{"line_number":41,"context_line":"    Decorate key methods that records the event in the database after the"},{"line_number":42,"context_line":"    decorated method has successfully run. The following are the proposed"},{"line_number":43,"context_line":"    list of key methods which will be decorated. The decorator will not"},{"line_number":44,"context_line":"    raise but will log the exception on error. Writing to the database"}],"source_content_type":"text/x-rst","patch_set":2,"id":"5ff73747_df74df6a","line":41,"range":{"start_line":41,"start_character":4,"end_line":41,"end_character":12},"updated":"2017-04-26 06:36:22.000000000","message":"IMO, it doesn\u0027t have to be a set of decorators. For what it\u0027s worth, decorators are limited. But that\u0027s not so important now.","commit_id":"18a1c1bde02bee4a7d95be82547e9de7dbef0c3f"},{"author":{"_account_id":16511,"name":"Bob Haddleton","email":"bob.haddleton@nokia.com","username":"bobh"},"change_message_id":"36b98ca1436b00ade7d8a85c5c9aefacf0ff4abe","unresolved":false,"context_lines":[{"line_number":38,"context_line":""},{"line_number":39,"context_line":"1. Workflow execution event publishing"},{"line_number":40,"context_line":""},{"line_number":41,"context_line":"    Decorate key methods that records the event in the database after the"},{"line_number":42,"context_line":"    decorated method has successfully run. The following are the proposed"},{"line_number":43,"context_line":"    list of key methods which will be decorated. The decorator will not"},{"line_number":44,"context_line":"    raise but will log the exception on error. Writing to the database"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7f287b81_8220eead","line":41,"in_reply_to":"5ff73747_df74df6a","updated":"2017-08-28 16:46:55.000000000","message":"I\u0027m wondering if the first step might be to create stevedore entrypoint hooks for each event type that someone might be interested in, and then implement the notifications as hooks using those entry points.  That would allow any other functionality to be introduced as a hook when needed without modifying the mainline code.  This would also eliminate the need to poll the database. It would be better to eliminate the need for polling entirely rather than just moving it from the client to the server.","commit_id":"18a1c1bde02bee4a7d95be82547e9de7dbef0c3f"},{"author":{"_account_id":16511,"name":"Bob Haddleton","email":"bob.haddleton@nokia.com","username":"bobh"},"change_message_id":"6b380c2d2d47922b0c5c7a5e64e7bef0b36264e6","unresolved":false,"context_lines":[{"line_number":38,"context_line":""},{"line_number":39,"context_line":"1. Workflow execution event publishing"},{"line_number":40,"context_line":""},{"line_number":41,"context_line":"    Decorate key methods that records the event in the database after the"},{"line_number":42,"context_line":"    decorated method has successfully run. The following are the proposed"},{"line_number":43,"context_line":"    list of key methods which will be decorated. The decorator will not"},{"line_number":44,"context_line":"    raise but will log the exception on error. Writing to the database"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7f287b81_8228bf96","line":41,"in_reply_to":"7f287b81_2b235558","updated":"2017-08-30 03:36:37.000000000","message":"It turns out that Nova has deprecated the hooks interface [0][1] in favor of notifications.  This is probably a good topic for discussion at the PTG.\n\n[0] https://openstack.nimeyo.com/75666/openstack-dev-nova-nova-hooks-document-%26-test-or-deprecate\n[1] https://review.openstack.org/#/c/286276/","commit_id":"18a1c1bde02bee4a7d95be82547e9de7dbef0c3f"},{"author":{"_account_id":16511,"name":"Bob Haddleton","email":"bob.haddleton@nokia.com","username":"bobh"},"change_message_id":"7e4d7cd25b397ae4818400a3bb3d2976de647c60","unresolved":false,"context_lines":[{"line_number":38,"context_line":""},{"line_number":39,"context_line":"1. Workflow execution event publishing"},{"line_number":40,"context_line":""},{"line_number":41,"context_line":"    Decorate key methods that records the event in the database after the"},{"line_number":42,"context_line":"    decorated method has successfully run. The following are the proposed"},{"line_number":43,"context_line":"    list of key methods which will be decorated. The decorator will not"},{"line_number":44,"context_line":"    raise but will log the exception on error. Writing to the database"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7f287b81_2b235558","line":41,"in_reply_to":"7f287b81_8220eead","updated":"2017-08-28 18:31:38.000000000","message":"Nova has an example of this [0] where they allow for hooks to be executed when a server is created[1], built[2], and deleted, and also when network information is available.\n\n[0] https://github.com/openstack/nova/blob/master/nova/hooks.py\n\n[1] https://github.com/openstack/nova/blob/master/nova/compute/api.py#L1585\n\n[2] https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1789","commit_id":"18a1c1bde02bee4a7d95be82547e9de7dbef0c3f"},{"author":{"_account_id":9712,"name":"Dougal Matthews","email":"dougal.matthews@canonical.com","username":"dougal"},"change_message_id":"6a9d7732f80fbc95dd7b5ab086ef587814ad3273","unresolved":false,"context_lines":[{"line_number":46,"context_line":"    to engine operations and workflow completion."},{"line_number":47,"context_line":""},{"line_number":48,"context_line":"    a. Workflow start"},{"line_number":49,"context_line":"    b. Workflow stop"},{"line_number":50,"context_line":"    c. Workflow resume"},{"line_number":51,"context_line":"    d. Workflow rerun"},{"line_number":52,"context_line":"    e. Workflow check and complete"}],"source_content_type":"text/x-rst","patch_set":2,"id":"5ff73747_3193ef51","line":49,"updated":"2017-04-26 15:57:00.000000000","message":"Why not use \"run\" and \"complete\"? Like in task and action below.","commit_id":"18a1c1bde02bee4a7d95be82547e9de7dbef0c3f"},{"author":{"_account_id":8731,"name":"Renat Akhmerov","email":"renat.akhmerov@gmail.com","username":"rakhmerov"},"change_message_id":"4dae8e98769c70add78de12a8b958b45dc1d3f50","unresolved":false,"context_lines":[{"line_number":47,"context_line":""},{"line_number":48,"context_line":"    a. Workflow start"},{"line_number":49,"context_line":"    b. Workflow stop"},{"line_number":50,"context_line":"    c. Workflow resume"},{"line_number":51,"context_line":"    d. Workflow rerun"},{"line_number":52,"context_line":"    e. Workflow check and complete"},{"line_number":53,"context_line":"    f. Task defer"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7ffa3b31_bbf9804e","line":50,"range":{"start_line":50,"start_character":4,"end_line":50,"end_character":22},"updated":"2017-04-14 06:14:10.000000000","message":"How about adding also \"Workflow pause\"?","commit_id":"18a1c1bde02bee4a7d95be82547e9de7dbef0c3f"},{"author":{"_account_id":8731,"name":"Renat Akhmerov","email":"renat.akhmerov@gmail.com","username":"rakhmerov"},"change_message_id":"8d5d81e1b34d9714efdfe5d51bef30002af3462f","unresolved":false,"context_lines":[{"line_number":47,"context_line":""},{"line_number":48,"context_line":"    a. Workflow start"},{"line_number":49,"context_line":"    b. Workflow stop"},{"line_number":50,"context_line":"    c. Workflow resume"},{"line_number":51,"context_line":"    d. Workflow rerun"},{"line_number":52,"context_line":"    e. Workflow check and complete"},{"line_number":53,"context_line":"    f. Task defer"}],"source_content_type":"text/x-rst","patch_set":2,"id":"5ff73747_bfd8f37e","line":50,"range":{"start_line":50,"start_character":4,"end_line":50,"end_character":22},"in_reply_to":"7ffa3b31_bbf9804e","updated":"2017-04-26 06:01:08.000000000","message":"Seems like we also need to solve the following problem: with ERROR and SUCCESS we set the state of a workflow after all tasks are in their final state (there\u0027s no RUNNING tasks). But with CANCELLED and PAUSED we first set the state of the workflow and but the real state of the workflow will still be RUNNING since there are some running tasks (we just don\u0027t schedule new tasks). I think it would be very helpful for users if we had corresponding events to track when a workflow really switched to either CANCELLED or PAUSED state, when all tasks have finished).","commit_id":"18a1c1bde02bee4a7d95be82547e9de7dbef0c3f"},{"author":{"_account_id":9712,"name":"Dougal Matthews","email":"dougal.matthews@canonical.com","username":"dougal"},"change_message_id":"6a9d7732f80fbc95dd7b5ab086ef587814ad3273","unresolved":false,"context_lines":[{"line_number":59,"context_line":""},{"line_number":60,"context_line":"2. Workflow execution event notification service"},{"line_number":61,"context_line":""},{"line_number":62,"context_line":"    Implement an event_notif_server similar to the scheduler that runs"},{"line_number":63,"context_line":"    periodically. The server will query the database for new events and"},{"line_number":64,"context_line":"    will pass each event to each consumer plugins for processing. When an"},{"line_number":65,"context_line":"    event is successfully processed, the server deletes the event from the"}],"source_content_type":"text/x-rst","patch_set":2,"id":"5ff73747_d1a3ab9d","line":62,"updated":"2017-04-26 15:57:00.000000000","message":"s/notif/notify/","commit_id":"18a1c1bde02bee4a7d95be82547e9de7dbef0c3f"},{"author":{"_account_id":8731,"name":"Renat Akhmerov","email":"renat.akhmerov@gmail.com","username":"rakhmerov"},"change_message_id":"4dae8e98769c70add78de12a8b958b45dc1d3f50","unresolved":false,"context_lines":[{"line_number":67,"context_line":""},{"line_number":68,"context_line":"3. Custom plugins for event notification"},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"    Custom plugins are implemented as stevedore extensions and registered"},{"line_number":71,"context_line":"    at setup.cfg. The custom plugin encapsulates any additional configuration,"},{"line_number":72,"context_line":"    logic, and filtering for processing. Authentication to external system"},{"line_number":73,"context_line":"    using API key or other mean can be configured here."},{"line_number":74,"context_line":""},{"line_number":75,"context_line":"Alternatives"},{"line_number":76,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7ffa3b31_be316eef","line":73,"range":{"start_line":70,"start_character":4,"end_line":73,"end_character":55},"updated":"2017-04-14 06:14:10.000000000","message":"How will external systems be able to subscribe for certain event types? Via API? I\u0027d like to understand some details on subscription mechanism.\n\nAlso could you maybe provide an example of such a plugin and how it will be processing events and deliver them to external systems.","commit_id":"18a1c1bde02bee4a7d95be82547e9de7dbef0c3f"},{"author":{"_account_id":8731,"name":"Renat Akhmerov","email":"renat.akhmerov@gmail.com","username":"rakhmerov"},"change_message_id":"7d31ad5c349f39ba686e4b4aa9ce92be0ad3ef02","unresolved":false,"context_lines":[{"line_number":67,"context_line":""},{"line_number":68,"context_line":"3. Custom plugins for event notification"},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"    Custom plugins are implemented as stevedore extensions and registered"},{"line_number":71,"context_line":"    at setup.cfg. The custom plugin encapsulates any additional configuration,"},{"line_number":72,"context_line":"    logic, and filtering for processing. Authentication to external system"},{"line_number":73,"context_line":"    using API key or other mean can be configured here."},{"line_number":74,"context_line":""},{"line_number":75,"context_line":"Alternatives"},{"line_number":76,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":2,"id":"5ff73747_dfa3bf9d","line":73,"range":{"start_line":70,"start_character":4,"end_line":73,"end_character":55},"in_reply_to":"7ffa3b31_13fa8d41","updated":"2017-04-26 06:36:22.000000000","message":"Plugins are OK, no question here. Although some rough example of a plugin right here in the spec would be really useful. I believe not only me will be asking this question.\n\nI\u0027m just trying to convince you that the idea to create another Scheduler-like thing most likely is not the best. We got a huge number of troubles from Scheduler for many reasons. It becomes a bottleneck when the load increases. I already removed using Scheduler from many places because it didn\u0027t work on practice well enough. In perspective, I\u0027d like to get rid of it at all in favor of using some real asynchronous mechanism.\nSo, having that said, I\u0027m just appealing you to think about changing a transport between Mistral engine and event notification plugins. Polling DB has lots of negative sides. I\u0027m not talking about contention, on that I think you\u0027re mostly right, if it\u0027s a separate table there won\u0027t be a contention (only if such a service runs in many processes) and wont\u0027 hinder anything else. But DB polling has delays between queries, the size of a delay is always a tricky thing to deal with, if it\u0027s small there will be a load on DB, but if it\u0027s big, it will increase memory footprint. Etc. etc. That\u0027s why I thought that engine could just send a message into a queue and once it\u0027s delivered to a plugin it\u0027s considered handled. And then it\u0027s up to a plugin how to deliver it to subscribers.\n\nI\u0027m not sure I\u0027m 100%, maybe I\u0027m not seeing something. But I\u0027d like to discuss it first.","commit_id":"18a1c1bde02bee4a7d95be82547e9de7dbef0c3f"},{"author":{"_account_id":5558,"name":"Winson Chan","email":"wcchan@stackstorm.com","username":"winson-c-chan"},"change_message_id":"708d79f2a215146f4fad04db3cad6ce0396d9986","unresolved":false,"context_lines":[{"line_number":67,"context_line":""},{"line_number":68,"context_line":"3. Custom plugins for event notification"},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"    Custom plugins are implemented as stevedore extensions and registered"},{"line_number":71,"context_line":"    at setup.cfg. The custom plugin encapsulates any additional configuration,"},{"line_number":72,"context_line":"    logic, and filtering for processing. Authentication to external system"},{"line_number":73,"context_line":"    using API key or other mean can be configured here."},{"line_number":74,"context_line":""},{"line_number":75,"context_line":"Alternatives"},{"line_number":76,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7ffa3b31_13fa8d41","line":73,"range":{"start_line":70,"start_character":4,"end_line":73,"end_character":55},"in_reply_to":"7ffa3b31_be316eef","updated":"2017-04-14 19:39:38.000000000","message":"First, the reason I prefer a plugin approach is that we will not be restricted to one form of callback mechanism (i.e. just HTTP post with a restricted approach to authentication or none). \n\nI\u0027m thinking that the specific events that an external system subscribe to are defined in configuration file and exposed over a plugin interface that the event notification server will call to determine whether the event be processed for that subscriber or not.\n\nThe alternative would be to define that on a case by case basis per workflow execution over an API. But from a system-system integration, the external system will know which events it need to subscribe to and so they can be pre-configure in advance.\n\nI can provide an example of such a plugin. But this mean, I will have to submit a patch when this spec hasn\u0027t been approved. But basically, it\u0027ll have an abstract method which the event notification server calls to pass the event into for processing. How the event is processed and which endpoints and what auth to use  is entirely up to the plugin implementation. But if this method succeeds, the event notification server will assume the event is processed for this subscriber.","commit_id":"18a1c1bde02bee4a7d95be82547e9de7dbef0c3f"},{"author":{"_account_id":8731,"name":"Renat Akhmerov","email":"renat.akhmerov@gmail.com","username":"rakhmerov"},"change_message_id":"4dae8e98769c70add78de12a8b958b45dc1d3f50","unresolved":false,"context_lines":[{"line_number":84,"context_line":""},{"line_number":85,"context_line":"2. Publish the event to the message queue in the decorator and call it done."},{"line_number":86,"context_line":"This is inconsistent with regards to Mistral\u0027s principle (i.e. guarantee)"},{"line_number":87,"context_line":"and API. If Mistral just publish to the message queue, there\u0027s no guarantee"},{"line_number":88,"context_line":"that the message will reach the external system. Also, API is over REST and"},{"line_number":89,"context_line":"publishing to a message queue for an external system to consume is"},{"line_number":90,"context_line":"inconsistent."},{"line_number":91,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"7ffa3b31_1e12a259","line":88,"range":{"start_line":87,"start_character":55,"end_line":88,"end_character":47},"updated":"2017-04-14 06:14:10.000000000","message":"Guaranteed delivery is a fundamental problem. For those custom plugins it will also be not easy to provide such a guarantee. I think using MQ is not a bad idea actually but with having some additional component that will be polling the event and then process it (maybe delegating to a custom plugin). Such approach seems better in terms of responsiveness (just a message sent via MQ vs periodic service calls). Scalability would also be better because we could, for example, run one workflow engine but many event notification engines, or the way around (which makes sense because this part related to processing events should be much lighter on CPU/memory than processing workflow itself). With the DB approach each engine will have to run with this additional event notification engine unless we make it possible to run it independently. BTW, Scheduler now is a problem for scalability since it increases load on DB if we run more engines.\n\nSo I\u0027d suggest we think about it. In my opinion, this approach may be better than creating a separate DB table and increase load on DB.","commit_id":"18a1c1bde02bee4a7d95be82547e9de7dbef0c3f"},{"author":{"_account_id":9712,"name":"Dougal Matthews","email":"dougal.matthews@canonical.com","username":"dougal"},"change_message_id":"6a9d7732f80fbc95dd7b5ab086ef587814ad3273","unresolved":false,"context_lines":[{"line_number":85,"context_line":"2. Publish the event to the message queue in the decorator and call it done."},{"line_number":86,"context_line":"This is inconsistent with regards to Mistral\u0027s principle (i.e. guarantee)"},{"line_number":87,"context_line":"and API. If Mistral just publish to the message queue, there\u0027s no guarantee"},{"line_number":88,"context_line":"that the message will reach the external system. Also, API is over REST and"},{"line_number":89,"context_line":"publishing to a message queue for an external system to consume is"},{"line_number":90,"context_line":"inconsistent."},{"line_number":91,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"5ff73747_e1d12ccf","line":88,"in_reply_to":"5ff73747_5fc8cf20","updated":"2017-04-26 15:57:00.000000000","message":"My first thought when I started reading this spec was that a MQ would be a good fit.\n\nIn TripleO we essentially do something like this, by sending messages from our workflows with Zaqar (then a GUI consumes the messages via web sockets). It generally works well, except we have no way of hooking in to all of the events described here.","commit_id":"18a1c1bde02bee4a7d95be82547e9de7dbef0c3f"},{"author":{"_account_id":16511,"name":"Bob Haddleton","email":"bob.haddleton@nokia.com","username":"bobh"},"change_message_id":"36b98ca1436b00ade7d8a85c5c9aefacf0ff4abe","unresolved":false,"context_lines":[{"line_number":85,"context_line":"2. Publish the event to the message queue in the decorator and call it done."},{"line_number":86,"context_line":"This is inconsistent with regards to Mistral\u0027s principle (i.e. guarantee)"},{"line_number":87,"context_line":"and API. If Mistral just publish to the message queue, there\u0027s no guarantee"},{"line_number":88,"context_line":"that the message will reach the external system. Also, API is over REST and"},{"line_number":89,"context_line":"publishing to a message queue for an external system to consume is"},{"line_number":90,"context_line":"inconsistent."},{"line_number":91,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"7f287b81_c516004a","line":88,"in_reply_to":"5ff73747_e1d12ccf","updated":"2017-08-28 16:46:55.000000000","message":"I agree with the MQ implementation - that\u0027s what I was looking for to support our use case.","commit_id":"18a1c1bde02bee4a7d95be82547e9de7dbef0c3f"},{"author":{"_account_id":5558,"name":"Winson Chan","email":"wcchan@stackstorm.com","username":"winson-c-chan"},"change_message_id":"708d79f2a215146f4fad04db3cad6ce0396d9986","unresolved":false,"context_lines":[{"line_number":84,"context_line":""},{"line_number":85,"context_line":"2. Publish the event to the message queue in the decorator and call it done."},{"line_number":86,"context_line":"This is inconsistent with regards to Mistral\u0027s principle (i.e. guarantee)"},{"line_number":87,"context_line":"and API. If Mistral just publish to the message queue, there\u0027s no guarantee"},{"line_number":88,"context_line":"that the message will reach the external system. Also, API is over REST and"},{"line_number":89,"context_line":"publishing to a message queue for an external system to consume is"},{"line_number":90,"context_line":"inconsistent."},{"line_number":91,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"7ffa3b31_9371fde3","line":88,"range":{"start_line":87,"start_character":55,"end_line":88,"end_character":47},"in_reply_to":"7ffa3b31_1e12a259","updated":"2017-04-14 19:39:38.000000000","message":"I am thinking that the event notification runs in another process, separate from the engine. It can be launched separately at mistral/cmd/launch.py and the operator decides where and how many to run.\n\nThe event table will be a separate table with no relationships to other tables. So the DB transaction and locking, conceptually will be limited to that table. It will consume more DB connections though that I agree.\n\nIf this is a major concern, I guess I can live with the events not going into a DB table and just push them into a MQ where the event notification server is listening to. We need to implement an API or command to allow for manual recovery in case of lost messages. For example, an API call to re-publish the events for a specific workflow execution.","commit_id":"18a1c1bde02bee4a7d95be82547e9de7dbef0c3f"},{"author":{"_account_id":8731,"name":"Renat Akhmerov","email":"renat.akhmerov@gmail.com","username":"rakhmerov"},"change_message_id":"7d31ad5c349f39ba686e4b4aa9ce92be0ad3ef02","unresolved":false,"context_lines":[{"line_number":84,"context_line":""},{"line_number":85,"context_line":"2. Publish the event to the message queue in the decorator and call it done."},{"line_number":86,"context_line":"This is inconsistent with regards to Mistral\u0027s principle (i.e. guarantee)"},{"line_number":87,"context_line":"and API. If Mistral just publish to the message queue, there\u0027s no guarantee"},{"line_number":88,"context_line":"that the message will reach the external system. Also, API is over REST and"},{"line_number":89,"context_line":"publishing to a message queue for an external system to consume is"},{"line_number":90,"context_line":"inconsistent."},{"line_number":91,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"5ff73747_5fc8cf20","line":88,"range":{"start_line":87,"start_character":55,"end_line":88,"end_character":47},"in_reply_to":"7ffa3b31_9371fde3","updated":"2017-04-26 06:36:22.000000000","message":"Yep, I think I already answered this (see line 73, last comment). In addition, I can only say that if we need to provide a guaranteed delivery we could think about it separately w/o considering engine at all. MQ protocol has message acknowledges, they could be sent by plugins once they processed (i.e. delivered to subscribers). But that makes a risk of delivering notifications more than once. Probably it has to be a part of the \"subscriber - notification plugin\" protocol to say that \"a notifications in rare cases can be delivered more than once, please use notification unique ID if you need a guarantee of not handling same notifications twice\".","commit_id":"18a1c1bde02bee4a7d95be82547e9de7dbef0c3f"}]}
