)]}'
{"README.md":[{"author":{"_account_id":11094,"name":"Tim Kuhlman","email":"timothy.kuhlman@canonical.com","username":"kuhlmant"},"change_message_id":"442bf951f8e3866fff48f51aa1ae6de6bb0b89da","unresolved":false,"context_lines":[{"line_number":3,"context_line":"This engine reads alarms from Kafka and then notifies the customer using their configured notification method."},{"line_number":4,"context_line":""},{"line_number":5,"context_line":"# Architecture"},{"line_number":6,"context_line":"There are four processing steps separated by queues implemented with python multiprocessing. The steps are:"},{"line_number":7,"context_line":""},{"line_number":8,"context_line":"1. Reads Alarms from Kafka, with no auto commit. - KafkaConsumer class"},{"line_number":9,"context_line":"2. Determine notification type for an alarm. Done by reading from mysql. - AlarmProcessor class"}],"source_content_type":"text/x-gfm","patch_set":2,"id":"da86d52c_77891c6e","side":"PARENT","line":6,"updated":"2015-02-12 18:39:38.000000000","message":"I think it would be helpful to mention that multiple instances can be run up to the number of Kafka partitions. I know it says it below but it is pretty key to understanding how it is built so is helpful to be part of the initial discussion. It may be nice to mention that it uses zookeeper for automatic rebalancing consumers also.","commit_id":"e5ede479181d7bdd603fc3f3ce5630b7c104bbc2"},{"author":{"_account_id":11094,"name":"Tim Kuhlman","email":"timothy.kuhlman@canonical.com","username":"kuhlmant"},"change_message_id":"442bf951f8e3866fff48f51aa1ae6de6bb0b89da","unresolved":false,"context_lines":[{"line_number":54,"context_line":""},{"line_number":55,"context_line":"## Monitoring"},{"line_number":56,"context_line":"statsd is incorporated into the daemon and will send all stats to localhost on udp port 8125. In many cases the stats"},{"line_number":57,"context_line":"are gathered per thread, the thread number is indicated by a -# at the end of the name."},{"line_number":58,"context_line":""},{"line_number":59,"context_line":"- Counters"},{"line_number":60,"context_line":"    - ConsumedFromKafka"}],"source_content_type":"text/x-gfm","patch_set":2,"id":"da86d52c_d7dba84c","side":"PARENT","line":57,"updated":"2015-02-12 18:39:38.000000000","message":"There is no per-thread gathering now right? However there is a big question of how statsd works with the different processes? Do the metrics that come through to monitoring make sense when multiple engines are running on one box?","commit_id":"e5ede479181d7bdd603fc3f3ce5630b7c104bbc2"},{"author":{"_account_id":11094,"name":"Tim Kuhlman","email":"timothy.kuhlman@canonical.com","username":"kuhlmant"},"change_message_id":"442bf951f8e3866fff48f51aa1ae6de6bb0b89da","unresolved":false,"context_lines":[{"line_number":72,"context_line":"    - SMTPTime"},{"line_number":73,"context_line":"    - OffsetCommitTime"},{"line_number":74,"context_line":""},{"line_number":75,"context_line":"# Future Considerations"},{"line_number":76,"context_line":"- Currently I lock the topic rather than the partitions. This effectively means there is only one active notification"},{"line_number":77,"context_line":"  engine at any given time. In the future to share the load among multiple daemons we could lock by partition."},{"line_number":78,"context_line":"- More extensive load testing is needed"}],"source_content_type":"text/x-gfm","patch_set":2,"id":"da86d52c_77c11c4d","side":"PARENT","line":75,"updated":"2015-02-12 18:39:38.000000000","message":"We talked about having some basic numbers on the speed of execution, how the testing was done and the suggestion for not committing every message as a way to speed things up if needed.","commit_id":"e5ede479181d7bdd603fc3f3ce5630b7c104bbc2"},{"author":{"_account_id":11094,"name":"Tim Kuhlman","email":"timothy.kuhlman@canonical.com","username":"kuhlmant"},"change_message_id":"442bf951f8e3866fff48f51aa1ae6de6bb0b89da","unresolved":false,"context_lines":[{"line_number":78,"context_line":"- More extensive load testing is needed"},{"line_number":79,"context_line":"  - How fast is the mysql db? How much load do we put on it. Initially I think it makes most sense to read notification"},{"line_number":80,"context_line":"    details for each alarm but eventually I may want to cache that info."},{"line_number":81,"context_line":"  - I am starting with a single KafkaConsumer and a single SentNotificationProcessor depending on load this may need"},{"line_number":82,"context_line":"    to scale."},{"line_number":83,"context_line":"  - How fast is the state tracker? Do I need to scale or speed that up at all?"},{"line_number":84,"context_line":""}],"source_content_type":"text/x-gfm","patch_set":2,"id":"da86d52c_b76394af","side":"PARENT","line":81,"updated":"2015-02-12 18:39:38.000000000","message":"A couple of these points no longer make sense.","commit_id":"e5ede479181d7bdd603fc3f3ce5630b7c104bbc2"},{"author":{"_account_id":11809,"name":"Craig Bryant","email":"craig.bryant@hpe.com","username":"craig-bryant"},"change_message_id":"ba8292b3ff52fc74a8a2afeb73a43d28210a4e3b","unresolved":false,"context_lines":[{"line_number":7,"context_line":"1. Reads Alarms from Kafka, with no auto commit. - KafkaConsumer class"},{"line_number":8,"context_line":"2. Determine notification type for an alarm. Done by reading from mysql. - AlarmProcessor class"},{"line_number":9,"context_line":"3. Send Notification. - NotificationProcessor class"},{"line_number":10,"context_line":"4. Successful notifictions are added to a sent notification topic. - NotificationEngine class"},{"line_number":11,"context_line":"5. Failed notifications are added to a retry topic. - NotificationEngine class"},{"line_number":12,"context_line":"6. Commit offset to Kafka - KafkaConsumer class"},{"line_number":13,"context_line":""}],"source_content_type":"text/x-gfm","patch_set":2,"id":"da86d52c_5747b8bb","line":10,"updated":"2015-02-12 18:38:44.000000000","message":"second word is misspelled","commit_id":"d882a287a7192f66ab1b861357be944f95da8cb4"},{"author":{"_account_id":11809,"name":"Craig Bryant","email":"craig.bryant@hpe.com","username":"craig-bryant"},"change_message_id":"ba8292b3ff52fc74a8a2afeb73a43d28210a4e3b","unresolved":false,"context_lines":[{"line_number":17,"context_line":"3. notification_retry_topic: Unsuccessful notifications."},{"line_number":18,"context_line":""},{"line_number":19,"context_line":"A retry engine runs in parallel with the notification engine and gives any"},{"line_number":20,"context_line":"failed notification a few extra chances at success."},{"line_number":21,"context_line":""},{"line_number":22,"context_line":"The retry engine generates notifications using the following steps:"},{"line_number":23,"context_line":"1. Reads Notification json data from Kafka, with no auto commit. - KafkaConsumer class"}],"source_content_type":"text/x-gfm","patch_set":2,"id":"da86d52c_1728e0e0","line":20,"updated":"2015-02-12 18:38:44.000000000","message":"s/few extra/configurable number of/","commit_id":"d882a287a7192f66ab1b861357be944f95da8cb4"}]}
