)]}'
{"doc/source/developer/specs/scale-out-scheduler.rst":[{"author":{"_account_id":7069,"name":"Joshua Hesketh","email":"josh@nitrotech.org","username":"jhesketh"},"change_message_id":"768aea9dc2c07c43c4739f3591c871e6fab8acd9","unresolved":false,"context_lines":[{"line_number":122,"context_line":"we need to find a way to handle duplicated events. This type of event gathering"},{"line_number":123,"context_line":"can run within the scheduler process. However while we don\u0027t have multiple"},{"line_number":124,"context_line":"scheduler support yet at this stage we may want to be able to start this as a"},{"line_number":125,"context_line":"separate process or include it in zuul-web."},{"line_number":126,"context_line":""},{"line_number":127,"context_line":"**Variant A**"},{"line_number":128,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"3f79a3b5_e8bec27b","line":125,"range":{"start_line":125,"start_character":33,"end_line":125,"end_character":42},"updated":"2018-12-05 08:18:10.000000000","message":"Why would the active event gathering go into zuul-web? (Seems more appropriate for the passive one).","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":7069,"name":"Joshua Hesketh","email":"josh@nitrotech.org","username":"jhesketh"},"change_message_id":"108b37281ad3416b25199722ce057b5f0b0728f8","unresolved":false,"context_lines":[{"line_number":122,"context_line":"we need to find a way to handle duplicated events. This type of event gathering"},{"line_number":123,"context_line":"can run within the scheduler process. However while we don\u0027t have multiple"},{"line_number":124,"context_line":"scheduler support yet at this stage we may want to be able to start this as a"},{"line_number":125,"context_line":"separate process or include it in zuul-web."},{"line_number":126,"context_line":""},{"line_number":127,"context_line":"**Variant A**"},{"line_number":128,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"3f79a3b5_a5177aef","line":125,"range":{"start_line":125,"start_character":33,"end_line":125,"end_character":42},"in_reply_to":"3f79a3b5_3ecdd05a","updated":"2018-12-06 03:25:58.000000000","message":"I think making it a deployers choice is unnecessary complexity. My preference is to keep it in the scheduler which can, after this spec, be scaled itself.","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"47c81b2f2d48a98f5e75dc2cbe805e50ed65578f","unresolved":false,"context_lines":[{"line_number":122,"context_line":"we need to find a way to handle duplicated events. This type of event gathering"},{"line_number":123,"context_line":"can run within the scheduler process. However while we don\u0027t have multiple"},{"line_number":124,"context_line":"scheduler support yet at this stage we may want to be able to start this as a"},{"line_number":125,"context_line":"separate process or include it in zuul-web."},{"line_number":126,"context_line":""},{"line_number":127,"context_line":"**Variant A**"},{"line_number":128,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"3f79a3b5_3ecdd05a","line":125,"range":{"start_line":125,"start_character":33,"end_line":125,"end_character":42},"in_reply_to":"3f79a3b5_e8bec27b","updated":"2018-12-05 09:15:33.000000000","message":"You\u0027re right, while this would be a possible lazy choice, the schedulers are probably the better place if one doesn\u0027t want to start a separate service for this. So I think we should leave this as a deployment choice if the user wants this inside the scheduler(s) or as separate services.","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":6488,"name":"Clint Byrum","email":"clint@fewbar.com","username":"clint-fewbar"},"change_message_id":"bf564f5bc53af78ada69794b84bccd06de7933e1","unresolved":false,"context_lines":[{"line_number":130,"context_line":"receiving the events. This makes sure that we don\u0027t need to handle duplicated"},{"line_number":131,"context_line":"events at all. A drawback is that there is a short time when the current leader"},{"line_number":132,"context_line":"stops until the next leader has started event gathering. This could lead to a"},{"line_number":133,"context_line":"few missed events."},{"line_number":134,"context_line":""},{"line_number":135,"context_line":"**Variant B**"},{"line_number":136,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"3f79a3b5_966add4d","line":133,"updated":"2018-12-04 18:26:44.000000000","message":"I was thinking the same thing, and I like it. But I\u0027m also concerned we\u0027re over-valueing 100% ingestion.","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"db19c9cbd23ac5dd7b305f54336e1bb731a8dd17","unresolved":false,"context_lines":[{"line_number":130,"context_line":"receiving the events. This makes sure that we don\u0027t need to handle duplicated"},{"line_number":131,"context_line":"events at all. A drawback is that there is a short time when the current leader"},{"line_number":132,"context_line":"stops until the next leader has started event gathering. This could lead to a"},{"line_number":133,"context_line":"few missed events."},{"line_number":134,"context_line":""},{"line_number":135,"context_line":"**Variant B**"},{"line_number":136,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"3f79a3b5_b1bce0b6","line":133,"updated":"2018-12-03 22:42:14.000000000","message":"Non-leaders could watch both the incoming and outgoing stream, keep a running history of the last X minutes of events which haven\u0027t made it into zookeeper, and, if they become leader, report those from the backlog.\n\nIt\u0027s a lot like B, but storing the data in memory twice (in the ingestor processes) instead of in ZK twice.","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"82e9dc4124df3bd1884f4edcf121ed9686706660","unresolved":false,"context_lines":[{"line_number":130,"context_line":"receiving the events. This makes sure that we don\u0027t need to handle duplicated"},{"line_number":131,"context_line":"events at all. A drawback is that there is a short time when the current leader"},{"line_number":132,"context_line":"stops until the next leader has started event gathering. This could lead to a"},{"line_number":133,"context_line":"few missed events."},{"line_number":134,"context_line":""},{"line_number":135,"context_line":"**Variant B**"},{"line_number":136,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"3f79a3b5_8c841059","line":133,"in_reply_to":"3f79a3b5_966add4d","updated":"2018-12-04 20:01:36.000000000","message":"How about we start with pure leader election and support a standalong ingestor? That way we have most of the time ingestion and could later extend that to the 100% variant if we want to.\n\nThinking more about that, in case of connection drops and reconnects whithout a crash of the ingestor we can loose events. So if we want 100% ingestion we would also need to cancel leadership instead of a reconnect.","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":6488,"name":"Clint Byrum","email":"clint@fewbar.com","username":"clint-fewbar"},"change_message_id":"bf564f5bc53af78ada69794b84bccd06de7933e1","unresolved":false,"context_lines":[{"line_number":137,"context_line":"All instances receive events and store them in Zookeeper. The event processing"},{"line_number":138,"context_line":"part now need to take care of deduplicating the events. This could be done by"},{"line_number":139,"context_line":"storing the hashes of the event payload of the last x minutes and ignoring any"},{"line_number":140,"context_line":"event that is duplicated."},{"line_number":141,"context_line":""},{"line_number":142,"context_line":"Although slightly more complicated we probably should go with variant B."},{"line_number":143,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"3f79a3b5_3687490d","line":140,"updated":"2018-12-04 18:26:44.000000000","message":"Dedupe by hash is actually the perfect strategy. Don\u0027t even store an event if it already exists. Ordering them can be a bit complex though, as the receivers will need to keep track of which event they got previously and assign an order counter when they write, but if they don\u0027t write because the record already exists, they need to ensure that they adapt their ordering properly. I believe we\u0027ve *almost* reimplemented Kafka at that point.","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":6488,"name":"Clint Byrum","email":"clint@fewbar.com","username":"clint-fewbar"},"change_message_id":"bf564f5bc53af78ada69794b84bccd06de7933e1","unresolved":false,"context_lines":[{"line_number":140,"context_line":"event that is duplicated."},{"line_number":141,"context_line":""},{"line_number":142,"context_line":"Although slightly more complicated we probably should go with variant B."},{"line_number":143,"context_line":""},{"line_number":144,"context_line":"Passive event gathering"},{"line_number":145,"context_line":"~~~~~~~~~~~~~~~~~~~~~~~"},{"line_number":146,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"3f79a3b5_568a0512","line":143,"updated":"2018-12-04 18:26:44.000000000","message":"Is the stream implemented as a safe messaging system? IIRC, it\u0027s just a stream, and as such, messages will be sent if there are subscribers, and if not, they just go into the bit bucket. As such I\u0027d say it\u0027s got to be expected that we might miss messages (such as when the ingestors are partitioned from Gerrit), and so just keeping the ingestor light, reliable, and consistent with a very low leader election/lock timeout would be the simplest course.","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"db19c9cbd23ac5dd7b305f54336e1bb731a8dd17","unresolved":false,"context_lines":[{"line_number":140,"context_line":"event that is duplicated."},{"line_number":141,"context_line":""},{"line_number":142,"context_line":"Although slightly more complicated we probably should go with variant B."},{"line_number":143,"context_line":""},{"line_number":144,"context_line":"Passive event gathering"},{"line_number":145,"context_line":"~~~~~~~~~~~~~~~~~~~~~~~"},{"line_number":146,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"3f79a3b5_d1b71cd1","line":143,"updated":"2018-12-03 22:42:14.000000000","message":"We could consider not supporting multiple ingestors, and instead make them standalone processes which are very small and quick to restart.  I think this would be acceptable, but if we can do A or B, we should.  They degrade to this case anyway if you only run one.","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"e1d91c0813064d5ce6f97ac8c2de391904f5e6df","unresolved":false,"context_lines":[{"line_number":140,"context_line":"event that is duplicated."},{"line_number":141,"context_line":""},{"line_number":142,"context_line":"Although slightly more complicated we probably should go with variant B."},{"line_number":143,"context_line":""},{"line_number":144,"context_line":"Passive event gathering"},{"line_number":145,"context_line":"~~~~~~~~~~~~~~~~~~~~~~~"},{"line_number":146,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"3f79a3b5_3702ae61","line":143,"in_reply_to":"3f79a3b5_568a0512","updated":"2018-12-06 19:28:59.000000000","message":"I think we should start with a simple leader election approach.","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":6488,"name":"Clint Byrum","email":"clint@fewbar.com","username":"clint-fewbar"},"change_message_id":"bf564f5bc53af78ada69794b84bccd06de7933e1","unresolved":false,"context_lines":[{"line_number":146,"context_line":""},{"line_number":147,"context_line":"In case of passive event gathering the events are sent to Zuul typically via"},{"line_number":148,"context_line":"webhooks. These types of events will be received in zuul-web that stores them in"},{"line_number":149,"context_line":"Zookeeper. This type of event gathering is used by the Guthub driver. In this"},{"line_number":150,"context_line":"case we can have multiple instances but still receive only one event. So we"},{"line_number":151,"context_line":"don\u0027t need to take special care of event deduplication. However it is beneficial"},{"line_number":152,"context_line":"to share the code that pops the events from Zookeeper it won\u0027t harm."}],"source_content_type":"text/x-rst","patch_set":2,"id":"3f79a3b5_f690f143","line":149,"updated":"2018-12-04 18:26:44.000000000","message":"s/Guthub/Github/","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"82e9dc4124df3bd1884f4edcf121ed9686706660","unresolved":false,"context_lines":[{"line_number":146,"context_line":""},{"line_number":147,"context_line":"In case of passive event gathering the events are sent to Zuul typically via"},{"line_number":148,"context_line":"webhooks. These types of events will be received in zuul-web that stores them in"},{"line_number":149,"context_line":"Zookeeper. This type of event gathering is used by the Guthub driver. In this"},{"line_number":150,"context_line":"case we can have multiple instances but still receive only one event. So we"},{"line_number":151,"context_line":"don\u0027t need to take special care of event deduplication. However it is beneficial"},{"line_number":152,"context_line":"to share the code that pops the events from Zookeeper it won\u0027t harm."}],"source_content_type":"text/x-rst","patch_set":2,"id":"3f79a3b5_0ceec01f","line":149,"in_reply_to":"3f79a3b5_f690f143","updated":"2018-12-04 20:01:36.000000000","message":"oops","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":6488,"name":"Clint Byrum","email":"clint@fewbar.com","username":"clint-fewbar"},"change_message_id":"bf564f5bc53af78ada69794b84bccd06de7933e1","unresolved":false,"context_lines":[{"line_number":149,"context_line":"Zookeeper. This type of event gathering is used by the Guthub driver. In this"},{"line_number":150,"context_line":"case we can have multiple instances but still receive only one event. So we"},{"line_number":151,"context_line":"don\u0027t need to take special care of event deduplication. However it is beneficial"},{"line_number":152,"context_line":"to share the code that pops the events from Zookeeper it won\u0027t harm."},{"line_number":153,"context_line":""},{"line_number":154,"context_line":""},{"line_number":155,"context_line":"Executor via Zookeeper"}],"source_content_type":"text/x-rst","patch_set":2,"id":"3f79a3b5_16948d34","line":152,"updated":"2018-12-04 18:26:44.000000000","message":"Did you mean to remove the \"it won\u0027t harm\" phrase here?","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"82e9dc4124df3bd1884f4edcf121ed9686706660","unresolved":false,"context_lines":[{"line_number":149,"context_line":"Zookeeper. This type of event gathering is used by the Guthub driver. In this"},{"line_number":150,"context_line":"case we can have multiple instances but still receive only one event. So we"},{"line_number":151,"context_line":"don\u0027t need to take special care of event deduplication. However it is beneficial"},{"line_number":152,"context_line":"to share the code that pops the events from Zookeeper it won\u0027t harm."},{"line_number":153,"context_line":""},{"line_number":154,"context_line":""},{"line_number":155,"context_line":"Executor via Zookeeper"}],"source_content_type":"text/x-rst","patch_set":2,"id":"3f79a3b5_ec38249f","line":152,"in_reply_to":"3f79a3b5_16948d34","updated":"2018-12-04 20:01:36.000000000","message":"I think I can remove the whole sentence as this was written under the assumption that the receiver performs deduplication where it\u0027s actually better to do this on the writer side.","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":7069,"name":"Joshua Hesketh","email":"josh@nitrotech.org","username":"jhesketh"},"change_message_id":"768aea9dc2c07c43c4739f3591c871e6fab8acd9","unresolved":false,"context_lines":[{"line_number":176,"context_line":"Currently the parsed branch config is stored globally but"},{"line_number":177,"context_line":"actually updated during tenant specific reconfigurations. When running multiple"},{"line_number":178,"context_line":"tenants in parallel this might lead to races. So the parsed branch config will"},{"line_number":179,"context_line":"be stored per tenant and thus protected an anyway required lock of a tenant. It"},{"line_number":180,"context_line":"will be stored in the path ``/zuul/tenant/\u003ctenant\u003e/config/\u003cproject\u003e/\u003cbranch\u003e``."},{"line_number":181,"context_line":"In case this gets large we could consider storing them compressed."},{"line_number":182,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"3f79a3b5_e3f6e32c","line":179,"range":{"start_line":179,"start_character":40,"end_line":179,"end_character":50},"updated":"2018-12-05 08:18:10.000000000","message":"by a?","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"47c81b2f2d48a98f5e75dc2cbe805e50ed65578f","unresolved":false,"context_lines":[{"line_number":176,"context_line":"Currently the parsed branch config is stored globally but"},{"line_number":177,"context_line":"actually updated during tenant specific reconfigurations. When running multiple"},{"line_number":178,"context_line":"tenants in parallel this might lead to races. So the parsed branch config will"},{"line_number":179,"context_line":"be stored per tenant and thus protected an anyway required lock of a tenant. It"},{"line_number":180,"context_line":"will be stored in the path ``/zuul/tenant/\u003ctenant\u003e/config/\u003cproject\u003e/\u003cbranch\u003e``."},{"line_number":181,"context_line":"In case this gets large we could consider storing them compressed."},{"line_number":182,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"3f79a3b5_9ef7448f","line":179,"range":{"start_line":179,"start_character":40,"end_line":179,"end_character":50},"in_reply_to":"3f79a3b5_e3f6e32c","updated":"2018-12-05 09:15:33.000000000","message":"will fix","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":6488,"name":"Clint Byrum","email":"clint@fewbar.com","username":"clint-fewbar"},"change_message_id":"bf564f5bc53af78ada69794b84bccd06de7933e1","unresolved":false,"context_lines":[{"line_number":178,"context_line":"tenants in parallel this might lead to races. So the parsed branch config will"},{"line_number":179,"context_line":"be stored per tenant and thus protected an anyway required lock of a tenant. It"},{"line_number":180,"context_line":"will be stored in the path ``/zuul/tenant/\u003ctenant\u003e/config/\u003cproject\u003e/\u003cbranch\u003e``."},{"line_number":181,"context_line":"In case this gets large we could consider storing them compressed."},{"line_number":182,"context_line":""},{"line_number":183,"context_line":""},{"line_number":184,"context_line":"Store pipeline and tenant state in Zookeeper"}],"source_content_type":"text/x-rst","patch_set":2,"id":"3f79a3b5_b69a795e","line":181,"updated":"2018-12-04 18:26:44.000000000","message":"I think as little as possible should be in ZK. If it can be re-calculated in a small amount of time, it should just be in memory, but if it\u0027s something large like the global config, it makes sense to maintain a consistent cache for schedulers to share and not have to re-fetch all the git trees.","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"0008c58802c4cc05e0078a49f3b7d7c6583699d3","unresolved":false,"context_lines":[{"line_number":178,"context_line":"tenants in parallel this might lead to races. So the parsed branch config will"},{"line_number":179,"context_line":"be stored per tenant and thus protected an anyway required lock of a tenant. It"},{"line_number":180,"context_line":"will be stored in the path ``/zuul/tenant/\u003ctenant\u003e/config/\u003cproject\u003e/\u003cbranch\u003e``."},{"line_number":181,"context_line":"In case this gets large we could consider storing them compressed."},{"line_number":182,"context_line":""},{"line_number":183,"context_line":""},{"line_number":184,"context_line":"Store pipeline and tenant state in Zookeeper"}],"source_content_type":"text/x-rst","patch_set":2,"id":"dfd5e7cf_52ccb788","line":181,"updated":"2019-01-07 22:30:09.000000000","message":"It sounds like we\u0027re pretty close to agreement on this.  Yes, when I said \"global parsed config\" I did mean the \"per branch parsed config\".  That is -- the thing where we read in the YAML from every repo and store it as a python dictionary (obviously, that turns into a ZK json blob).  But there is no tenant information and it hasn\u0027t been turned into zuul.model objects yet.  It\u0027s more or less just a cache of what\u0027s in the repos.\n\nI think it would be okay to store one copy of this -- I don\u0027t think we need to store a copy for each tenant.  We can \"lock\" the global config when someone is updating it so that if anyone needs to generate a tenant config, they will wait until the update is complete.  Then, obviously, once it is updated and unlocked, all components should regenerate their tenant config(s).","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"db19c9cbd23ac5dd7b305f54336e1bb731a8dd17","unresolved":false,"context_lines":[{"line_number":178,"context_line":"tenants in parallel this might lead to races. So the parsed branch config will"},{"line_number":179,"context_line":"be stored per tenant and thus protected an anyway required lock of a tenant. It"},{"line_number":180,"context_line":"will be stored in the path ``/zuul/tenant/\u003ctenant\u003e/config/\u003cproject\u003e/\u003cbranch\u003e``."},{"line_number":181,"context_line":"In case this gets large we could consider storing them compressed."},{"line_number":182,"context_line":""},{"line_number":183,"context_line":""},{"line_number":184,"context_line":"Store pipeline and tenant state in Zookeeper"}],"source_content_type":"text/x-rst","patch_set":2,"id":"3f79a3b5_71d408f1","line":181,"updated":"2018-12-03 22:42:14.000000000","message":"This is where the biggest unanswered questions are.\n\nAre you suggesting we store the parsed config as a single object, or is that just the root, and under there we would have \".../jobs/...\" etc?\n\nWe can still store the parsed branch config globally.  When a change happens, we can just lock the global config and whoever gets the lock gets to update it.\n\nIf we only store the global config, should we also store per-tenant configs?  Should they reference global config objects \"by reference\" (eg, znode ids?).\n\nSame question for dynamic layouts (ie, per-change configs).\n\nIn general, these suggest two ends of a spectrum: storing a lot of config in zk, or storing very little.  If we store the global parsed config, per-tenant layout, and (where necessary) per-change layout, we\u0027ll be storing a lot in ZK.\n\nIf we wanted to store as little as possible, I\u0027d suggest only storing the global parsed config in zk, and then have each scheduler process generate its own copy of the per-tenant config (whenever the underlying objects or configuration changed).  The zuul-web process could do the same so that it sees the same view of the data.  And whenever a dynamic layout is needed, the scheduler which is processing that pipeline at the time can generate the dynamic layout, keep it in memory until it has finished the pipeline, then discard it (after having recorded the frozen jobs in ZK).\n\nThe middle ground is probably closer to what you describe -- keep both the global and per-tenant configs in zk (perhaps with objects by reference), but don\u0027t store dynamic layouts in ZK.  That has the best scaling performance too (in that it doesn\u0027t cause ZK to scale eponentially with change volume).","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"82e9dc4124df3bd1884f4edcf121ed9686706660","unresolved":false,"context_lines":[{"line_number":178,"context_line":"tenants in parallel this might lead to races. So the parsed branch config will"},{"line_number":179,"context_line":"be stored per tenant and thus protected an anyway required lock of a tenant. It"},{"line_number":180,"context_line":"will be stored in the path ``/zuul/tenant/\u003ctenant\u003e/config/\u003cproject\u003e/\u003cbranch\u003e``."},{"line_number":181,"context_line":"In case this gets large we could consider storing them compressed."},{"line_number":182,"context_line":""},{"line_number":183,"context_line":""},{"line_number":184,"context_line":"Store pipeline and tenant state in Zookeeper"}],"source_content_type":"text/x-rst","patch_set":2,"id":"3f79a3b5_0cb1e000","line":181,"in_reply_to":"3f79a3b5_b69a795e","updated":"2018-12-04 20:01:36.000000000","message":"My thinking was that we store the per branch parsed config (which you are referring to as global config?) as json blobs under this path. This way we can quickly recalculate the layout of a tenant without having to save too much data in zk which we should avoid. I suggested to store this under a tenant specific path in order to not having to care about race conditions when reconfiguring several tenants sharing a few shared at the same time. I\u0027m not sure if that\u0027s a problem so that\u0027s just pure precaution. We also can keep that really global and protect the updates with locks per project. That way we don\u0027t serialize reconfigurations of several tenants.\n\nThe per tenant config (current non dynamic layout) could be easily cached in memory and updated by each scheduler based on the version and cversion (child version) of each project that is part of the tenant. In case a version or cversion changed, we would need to inject a tenant reconfiguration (possibly without asking the mergers) to rebuild the layout of the tenant.\n\nRegarding the dynamic layout. I also thought that we should just store the frozen jobs in zk as part of the pipeline state. After that we probably don\u0027t need the dynamic layout so there should be no need to store it in zk.","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":6488,"name":"Clint Byrum","email":"clint@fewbar.com","username":"clint-fewbar"},"change_message_id":"bf564f5bc53af78ada69794b84bccd06de7933e1","unresolved":false,"context_lines":[{"line_number":195,"context_line":"stored in ``/zuul/tenant/\u003ctenant\u003e/semaphores/\u003cname\u003e``."},{"line_number":196,"context_line":""},{"line_number":197,"context_line":"Further also the times database must be stored in Zookeeper. This will be stored"},{"line_number":198,"context_line":"in ``/zuul/tenant/\u003ctenant\u003e/times/\u003cproject\u003e/\u003cbranch\u003e/\u003cjob\u003e``."},{"line_number":199,"context_line":""},{"line_number":200,"context_line":""},{"line_number":201,"context_line":"Parallelize tenant processing"}],"source_content_type":"text/x-rst","patch_set":2,"id":"3f79a3b5_d695b530","line":198,"updated":"2018-12-04 18:26:44.000000000","message":"+1 for that, I think it\u0027s clear that the DB isn\u0027t quite in the right place as a reporter anyway.","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"db19c9cbd23ac5dd7b305f54336e1bb731a8dd17","unresolved":false,"context_lines":[{"line_number":195,"context_line":"stored in ``/zuul/tenant/\u003ctenant\u003e/semaphores/\u003cname\u003e``."},{"line_number":196,"context_line":""},{"line_number":197,"context_line":"Further also the times database must be stored in Zookeeper. This will be stored"},{"line_number":198,"context_line":"in ``/zuul/tenant/\u003ctenant\u003e/times/\u003cproject\u003e/\u003cbranch\u003e/\u003cjob\u003e``."},{"line_number":199,"context_line":""},{"line_number":200,"context_line":""},{"line_number":201,"context_line":"Parallelize tenant processing"}],"source_content_type":"text/x-rst","patch_set":2,"id":"3f79a3b5_91d764f4","line":198,"updated":"2018-12-03 22:42:14.000000000","message":"Or, we could make the SQL database integrated (ie, not a reporter) and drop the times database.","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"0008c58802c4cc05e0078a49f3b7d7c6583699d3","unresolved":false,"context_lines":[{"line_number":195,"context_line":"stored in ``/zuul/tenant/\u003ctenant\u003e/semaphores/\u003cname\u003e``."},{"line_number":196,"context_line":""},{"line_number":197,"context_line":"Further also the times database must be stored in Zookeeper. This will be stored"},{"line_number":198,"context_line":"in ``/zuul/tenant/\u003ctenant\u003e/times/\u003cproject\u003e/\u003cbranch\u003e/\u003cjob\u003e``."},{"line_number":199,"context_line":""},{"line_number":200,"context_line":""},{"line_number":201,"context_line":"Parallelize tenant processing"}],"source_content_type":"text/x-rst","patch_set":2,"id":"dfd5e7cf_f2c84b75","line":198,"updated":"2019-01-07 22:30:09.000000000","message":"The SQL database can accomplish what the time database is used for.  It would just be a very simple query to get the last 10 runtimes for successful runs of a tenant-project-branch-job.","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":7069,"name":"Joshua Hesketh","email":"josh@nitrotech.org","username":"jhesketh"},"change_message_id":"768aea9dc2c07c43c4739f3591c871e6fab8acd9","unresolved":false,"context_lines":[{"line_number":195,"context_line":"stored in ``/zuul/tenant/\u003ctenant\u003e/semaphores/\u003cname\u003e``."},{"line_number":196,"context_line":""},{"line_number":197,"context_line":"Further also the times database must be stored in Zookeeper. This will be stored"},{"line_number":198,"context_line":"in ``/zuul/tenant/\u003ctenant\u003e/times/\u003cproject\u003e/\u003cbranch\u003e/\u003cjob\u003e``."},{"line_number":199,"context_line":""},{"line_number":200,"context_line":""},{"line_number":201,"context_line":"Parallelize tenant processing"}],"source_content_type":"text/x-rst","patch_set":2,"id":"3f79a3b5_636173d5","line":198,"in_reply_to":"3f79a3b5_4cb91879","updated":"2018-12-05 08:18:10.000000000","message":"I agree. I think for a while we\u0027ve known that we\u0027ll likely require some kind of database to run zuul. We may even choose something that isn\u0027t SQL, but that is a topic for another spec. I think it would be safe to assume for this spec that some sort of database is both required and available. I also think that this change should be part of v4.","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"82e9dc4124df3bd1884f4edcf121ed9686706660","unresolved":false,"context_lines":[{"line_number":195,"context_line":"stored in ``/zuul/tenant/\u003ctenant\u003e/semaphores/\u003cname\u003e``."},{"line_number":196,"context_line":""},{"line_number":197,"context_line":"Further also the times database must be stored in Zookeeper. This will be stored"},{"line_number":198,"context_line":"in ``/zuul/tenant/\u003ctenant\u003e/times/\u003cproject\u003e/\u003cbranch\u003e/\u003cjob\u003e``."},{"line_number":199,"context_line":""},{"line_number":200,"context_line":""},{"line_number":201,"context_line":"Parallelize tenant processing"}],"source_content_type":"text/x-rst","patch_set":2,"id":"3f79a3b5_4cb91879","line":198,"in_reply_to":"3f79a3b5_d695b530","updated":"2018-12-04 20:01:36.000000000","message":"+1 for moving sql reporting into zuul itself. But I\u0027m not sure if we should move the times database into sql. I hadn\u0027t yet a really close look into how it exactly works yet. I just saw that the times database consists of small binary files per project/branch/job.","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"db19c9cbd23ac5dd7b305f54336e1bb731a8dd17","unresolved":false,"context_lines":[{"line_number":216,"context_line":"In order to reduce stalls when doing reconfigurations or tenant reconfigurations"},{"line_number":217,"context_line":"we can easily run more than one tenant processor in a thread pool per scheduler."},{"line_number":218,"context_line":"This way a tenant that is running a longer reconfiguration won\u0027t block other"},{"line_number":219,"context_line":"tenants."},{"line_number":220,"context_line":""},{"line_number":221,"context_line":""},{"line_number":222,"context_line":"Zuul-web changes"}],"source_content_type":"text/x-rst","patch_set":2,"id":"3f79a3b5_31de900f","line":219,"updated":"2018-12-03 22:42:14.000000000","message":"It\u0027s possible that everything you wrote here could apply to \"pipelines\" rather than tenants.  I think we should seek to have parallel pipeline processors within a tenant, not just parallel tenant processing.  Fortunately, I think that\u0027s not a big change from this.\n\nWe might accomplish this by having the trigger event processor dispatch to tenant+pipeline rather than just tenant.","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"82e9dc4124df3bd1884f4edcf121ed9686706660","unresolved":false,"context_lines":[{"line_number":216,"context_line":"In order to reduce stalls when doing reconfigurations or tenant reconfigurations"},{"line_number":217,"context_line":"we can easily run more than one tenant processor in a thread pool per scheduler."},{"line_number":218,"context_line":"This way a tenant that is running a longer reconfiguration won\u0027t block other"},{"line_number":219,"context_line":"tenants."},{"line_number":220,"context_line":""},{"line_number":221,"context_line":""},{"line_number":222,"context_line":"Zuul-web changes"}],"source_content_type":"text/x-rst","patch_set":2,"id":"3f79a3b5_ac38eccb","line":219,"in_reply_to":"3f79a3b5_31de900f","updated":"2018-12-04 20:01:36.000000000","message":"Right, I\u0027ll change that. This makes the management events slightly more complex but I think that\u0027s ok. These events like reconfigure or tenant-reconfigure typically operate on tenant level. So we might need an rw lock on the tenant and a normal lock on the pipeline. A pipeline would lock both, a management event would only lock the tenant lock.","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":7069,"name":"Joshua Hesketh","email":"josh@nitrotech.org","username":"jhesketh"},"change_message_id":"768aea9dc2c07c43c4739f3591c871e6fab8acd9","unresolved":false,"context_lines":[{"line_number":216,"context_line":"In order to reduce stalls when doing reconfigurations or tenant reconfigurations"},{"line_number":217,"context_line":"we can easily run more than one tenant processor in a thread pool per scheduler."},{"line_number":218,"context_line":"This way a tenant that is running a longer reconfiguration won\u0027t block other"},{"line_number":219,"context_line":"tenants."},{"line_number":220,"context_line":""},{"line_number":221,"context_line":""},{"line_number":222,"context_line":"Zuul-web changes"}],"source_content_type":"text/x-rst","patch_set":2,"id":"3f79a3b5_c33ba797","line":219,"in_reply_to":"3f79a3b5_ac38eccb","updated":"2018-12-05 08:18:10.000000000","message":"+1 for locking on the pipeline.","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":6488,"name":"Clint Byrum","email":"clint@fewbar.com","username":"clint-fewbar"},"change_message_id":"c3052fc3a24216756e100ef067c016324d63dbb4","unresolved":false,"context_lines":[{"line_number":232,"context_line":"When switching the executor job queue to Zookeeper we need to take precautions"},{"line_number":233,"context_line":"because this will also contain decrypted secrets. In order to secure this"},{"line_number":234,"context_line":"communication channel we need to make sure that we use authenticated and"},{"line_number":235,"context_line":"encrypted connections to zookeeper."},{"line_number":236,"context_line":""},{"line_number":237,"context_line":"* There is already a change that adds Zookeeper auth:"},{"line_number":238,"context_line":"  https://review.openstack.org/619156"}],"source_content_type":"text/x-rst","patch_set":2,"id":"3f79a3b5_0687780c","line":235,"updated":"2018-12-05 00:13:45.000000000","message":"I\u0027d like for the scheduler to also encrypt the keys on disk so I don\u0027t have to use encrypted volumes. But that\u0027s a story for another day. Suffice to say that you\u0027re right, the trust levels are the same, but I\u0027m holding new development to a higher standard.","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":6488,"name":"Clint Byrum","email":"clint@fewbar.com","username":"clint-fewbar"},"change_message_id":"bf564f5bc53af78ada69794b84bccd06de7933e1","unresolved":false,"context_lines":[{"line_number":232,"context_line":"When switching the executor job queue to Zookeeper we need to take precautions"},{"line_number":233,"context_line":"because this will also contain decrypted secrets. In order to secure this"},{"line_number":234,"context_line":"communication channel we need to make sure that we use authenticated and"},{"line_number":235,"context_line":"encrypted connections to zookeeper."},{"line_number":236,"context_line":""},{"line_number":237,"context_line":"* There is already a change that adds Zookeeper auth:"},{"line_number":238,"context_line":"  https://review.openstack.org/619156"}],"source_content_type":"text/x-rst","patch_set":2,"id":"3f79a3b5_76a4e1a4","line":235,"updated":"2018-12-04 18:26:44.000000000","message":"IMO it\u0027s not enough to have auth and TLS. We will need to keep those secrets encrypted in ZK, since writing anything to ZK means it will be in the ZK transaction log, unlike gearman which keeps the queues in memory. We can have the executors place a public key into ZK, and schedulers then re-encrypt job secrets to that key upon an executor claiming the job.","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"47c81b2f2d48a98f5e75dc2cbe805e50ed65578f","unresolved":false,"context_lines":[{"line_number":232,"context_line":"When switching the executor job queue to Zookeeper we need to take precautions"},{"line_number":233,"context_line":"because this will also contain decrypted secrets. In order to secure this"},{"line_number":234,"context_line":"communication channel we need to make sure that we use authenticated and"},{"line_number":235,"context_line":"encrypted connections to zookeeper."},{"line_number":236,"context_line":""},{"line_number":237,"context_line":"* There is already a change that adds Zookeeper auth:"},{"line_number":238,"context_line":"  https://review.openstack.org/619156"}],"source_content_type":"text/x-rst","patch_set":2,"id":"3f79a3b5_1e19f4d0","line":235,"in_reply_to":"3f79a3b5_0687780c","updated":"2018-12-05 09:15:33.000000000","message":"Ok, so we should at least encrypt the secrets inside zk with a symmetric cipher (e.g. aes?). I think using public keys will be complicated as this involves a lot of additional round trips when we have multiple schedulers and exeuctors that need secrets or private keys.","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"82e9dc4124df3bd1884f4edcf121ed9686706660","unresolved":false,"context_lines":[{"line_number":232,"context_line":"When switching the executor job queue to Zookeeper we need to take precautions"},{"line_number":233,"context_line":"because this will also contain decrypted secrets. In order to secure this"},{"line_number":234,"context_line":"communication channel we need to make sure that we use authenticated and"},{"line_number":235,"context_line":"encrypted connections to zookeeper."},{"line_number":236,"context_line":""},{"line_number":237,"context_line":"* There is already a change that adds Zookeeper auth:"},{"line_number":238,"context_line":"  https://review.openstack.org/619156"}],"source_content_type":"text/x-rst","patch_set":2,"id":"3f79a3b5_512c4fb0","line":235,"in_reply_to":"3f79a3b5_76a4e1a4","updated":"2018-12-04 20:01:36.000000000","message":"Interesting point, but this assumes that you can trust your zookeeper less than your scheduler (as it stores the private keys unencrypted)? Is that really a problem (as we do trust the disks of the scheduler)?","commit_id":"fcd20da6c39729abc70ba2fb1b1b3b386aef76fb"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"7b930be34becd43c0a897d1b23f4eedd4b7d19df","unresolved":false,"context_lines":[{"line_number":98,"context_line":""},{"line_number":99,"context_line":"We will split the drivers into an event receiving and an event processing"},{"line_number":100,"context_line":"component. The event receiving component will store the events in a squenced"},{"line_number":101,"context_line":"znode in the path ``/events/connection/\u003cconnection name\u003e/\u003csequence\u003e``."},{"line_number":102,"context_line":"The event receiving part may or may not run within the scheduler context."},{"line_number":103,"context_line":"The event processing part will be part of the scheduler context."},{"line_number":104,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"1f769fc5_0df5c5cb","line":101,"range":{"start_line":101,"start_character":20,"end_line":101,"end_character":67},"updated":"2018-12-23 11:12:55.000000000","message":"It\u0027s /zuul/events/...","commit_id":"6ba86da65fa6a25cb72e1deecf2f17030e121358"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"28c6289022e4567efa909af1824dd84ad13680d7","unresolved":false,"context_lines":[{"line_number":135,"context_line":"If there is a need to guarantee that there is no missed event during a"},{"line_number":136,"context_line":"leadership change the above algorithm can be enhanced later with parallel"},{"line_number":137,"context_line":"gathering and deduplication strategies. As this is much more complicated this"},{"line_number":138,"context_line":"will not be in scope of this spec."},{"line_number":139,"context_line":""},{"line_number":140,"context_line":""},{"line_number":141,"context_line":"Passive event gathering"}],"source_content_type":"text/x-rst","patch_set":4,"id":"dfd5e7cf_525cb74a","line":138,"updated":"2019-01-07 23:35:14.000000000","message":"Thinking out loud would another option be to avoid leader election and allow every active event gatherer process to listen for and write events. We would then dedup via a hash of each event whcih could be locked for before writing the event data.\n\nOne drawback to this appraoch is it is computationally more expensive, but should avoid losing any events if you restart one process at a time.","commit_id":"6ba86da65fa6a25cb72e1deecf2f17030e121358"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"feeb13db5761efcd71624ac447c7a8d072b82c5e","unresolved":false,"context_lines":[{"line_number":135,"context_line":"If there is a need to guarantee that there is no missed event during a"},{"line_number":136,"context_line":"leadership change the above algorithm can be enhanced later with parallel"},{"line_number":137,"context_line":"gathering and deduplication strategies. As this is much more complicated this"},{"line_number":138,"context_line":"will not be in scope of this spec."},{"line_number":139,"context_line":""},{"line_number":140,"context_line":""},{"line_number":141,"context_line":"Passive event gathering"}],"source_content_type":"text/x-rst","patch_set":4,"id":"dfd5e7cf_60dc7374","line":138,"in_reply_to":"dfd5e7cf_2dd0b2f7","updated":"2019-01-08 16:08:25.000000000","message":"Yes, I had this in the first version as an option. But I think it\u0027s much more complex than simple leader election and could be an enhancement after the scale out implementation. So I think this doesn\u0027t need to be in the scope of this spec.","commit_id":"6ba86da65fa6a25cb72e1deecf2f17030e121358"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"952d66a26c34ba0364645da2ef9bbe5bcc35f979","unresolved":false,"context_lines":[{"line_number":135,"context_line":"If there is a need to guarantee that there is no missed event during a"},{"line_number":136,"context_line":"leadership change the above algorithm can be enhanced later with parallel"},{"line_number":137,"context_line":"gathering and deduplication strategies. As this is much more complicated this"},{"line_number":138,"context_line":"will not be in scope of this spec."},{"line_number":139,"context_line":""},{"line_number":140,"context_line":""},{"line_number":141,"context_line":"Passive event gathering"}],"source_content_type":"text/x-rst","patch_set":4,"id":"dfd5e7cf_2dd0b2f7","line":138,"in_reply_to":"dfd5e7cf_525cb74a","updated":"2019-01-08 00:11:06.000000000","message":"Thinking about this more I\u0027m not sure how feasible this is as we\u0027d need to keep a record of every event processed for some potentially long period of time in order to check in the future if some event we\u0027ve received is a duplicate.\n\nWe might be able to avoid that issue with hashes if we can guarantee that all of the events we are receiving already come with some monotonically increasing identifier value (then its a simple \u003c or \u003e comparison). I don\u0027t know that we can rely on that for all potential event inputs so the leader election design may be simplest.","commit_id":"6ba86da65fa6a25cb72e1deecf2f17030e121358"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"56fcedd86d8b5ef3945ca4b33ab481687855c251","unresolved":false,"context_lines":[{"line_number":153,"context_line":""},{"line_number":154,"context_line":"We need to store the global configuration in zookeeper. However zookeeper is not"},{"line_number":155,"context_line":"designed as a database with a large amount of data we should store as little as"},{"line_number":156,"context_line":"possible in zookeeper. Thus we only store the per project-branch parsed config"},{"line_number":157,"context_line":"in zookeeper. From this every part of zuul like the scheduler or also zuul-web"},{"line_number":158,"context_line":"can quickly recalculate the layout of each tenant and keep it up to date by"},{"line_number":159,"context_line":"watching for changes in the parsed project-branch-config. The configs will be"}],"source_content_type":"text/x-rst","patch_set":4,"id":"1f769fc5_ed964995","line":156,"range":{"start_line":156,"start_character":23,"end_line":156,"end_character":78},"updated":"2018-12-23 10:11:22.000000000","message":"I think we probably end up storing the unparsed config too(or only). This is needed for efficient tenant reconfigurations which will be done distributed.","commit_id":"6ba86da65fa6a25cb72e1deecf2f17030e121358"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"a8efb8d2c2bcc5aecee220301e55090390888468","unresolved":false,"context_lines":[{"line_number":160,"context_line":"stored in the path ``/zuul/tenant/\u003ctenant\u003e/config/\u003cproject\u003e/\u003cbranch\u003e`` as json"},{"line_number":161,"context_line":"blobs per branch. If this data gets large we can consider compressing these"},{"line_number":162,"context_line":"blobs. If we do that we should support both formats in parallel to be able"},{"line_number":163,"context_line":"to disable the compression e.g. for debugging purposes."},{"line_number":164,"context_line":""},{"line_number":165,"context_line":""},{"line_number":166,"context_line":"Store pipeline and tenant state in Zookeeper"}],"source_content_type":"text/x-rst","patch_set":4,"id":"dfd5e7cf_9288df1f","line":163,"updated":"2019-01-07 22:35:17.000000000","message":"As mentioned on PS2 (but after you updated to PS4), I think we should only store the global project-branch config in ZK.  I don\u0027t think we need to do so per-tenant.  I think if we just use it as a cache of JSON representing the YAML in files, then each component can generate per-tenant configs from the ZK data.  We can use locking to avoid race conditions.  This lets us store the minimum necessary in ZK.","commit_id":"6ba86da65fa6a25cb72e1deecf2f17030e121358"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"c836757b5f9d0bdf9f133e2c94d6d6bf8b50f90f","unresolved":false,"context_lines":[{"line_number":160,"context_line":"stored in the path ``/zuul/tenant/\u003ctenant\u003e/config/\u003cproject\u003e/\u003cbranch\u003e`` as json"},{"line_number":161,"context_line":"blobs per branch. If this data gets large we can consider compressing these"},{"line_number":162,"context_line":"blobs. If we do that we should support both formats in parallel to be able"},{"line_number":163,"context_line":"to disable the compression e.g. for debugging purposes."},{"line_number":164,"context_line":""},{"line_number":165,"context_line":""},{"line_number":166,"context_line":"Store pipeline and tenant state in Zookeeper"}],"source_content_type":"text/x-rst","patch_set":4,"id":"dfd5e7cf_0dbcce03","line":163,"updated":"2019-01-07 23:42:53.000000000","message":"Clark, asking for the content from the mergers is time-consuming and not something that every component should need to do, which is why I think that at least (but also, at most) we should store the read zuul.yaml data (the global project-branch config) in ZK.  It\u0027s not clear to me whether you support that or not, but I think it\u0027s important.\n\nBeyond that, yes, I imagine essentially a checkpoint system (was this tenant config generated from the current global config sequence number, or a previous one? if previous, it\u0027s invalid, need to update) would work well.","commit_id":"6ba86da65fa6a25cb72e1deecf2f17030e121358"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"195e0418d4193a33c36af360ebd00aac4c7d75b7","unresolved":false,"context_lines":[{"line_number":160,"context_line":"stored in the path ``/zuul/tenant/\u003ctenant\u003e/config/\u003cproject\u003e/\u003cbranch\u003e`` as json"},{"line_number":161,"context_line":"blobs per branch. If this data gets large we can consider compressing these"},{"line_number":162,"context_line":"blobs. If we do that we should support both formats in parallel to be able"},{"line_number":163,"context_line":"to disable the compression e.g. for debugging purposes."},{"line_number":164,"context_line":""},{"line_number":165,"context_line":""},{"line_number":166,"context_line":"Store pipeline and tenant state in Zookeeper"}],"source_content_type":"text/x-rst","patch_set":4,"id":"dfd5e7cf_2df8b2c2","line":163,"in_reply_to":"dfd5e7cf_0dbcce03","updated":"2019-01-07 23:47:17.000000000","message":"I was mostly brainstorming ways to avoid storing much data in zk and was semi familiar with how kafka does it (I mean at a high level I\u0027ve never read the code).\n\nIf we think the cost of many git merges is worse than zk storing some data (likely is given git mergers are computations) then this setup with checkpoints off of the yaml data would work well.","commit_id":"6ba86da65fa6a25cb72e1deecf2f17030e121358"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"feeb13db5761efcd71624ac447c7a8d072b82c5e","unresolved":false,"context_lines":[{"line_number":160,"context_line":"stored in the path ``/zuul/tenant/\u003ctenant\u003e/config/\u003cproject\u003e/\u003cbranch\u003e`` as json"},{"line_number":161,"context_line":"blobs per branch. If this data gets large we can consider compressing these"},{"line_number":162,"context_line":"blobs. If we do that we should support both formats in parallel to be able"},{"line_number":163,"context_line":"to disable the compression e.g. for debugging purposes."},{"line_number":164,"context_line":""},{"line_number":165,"context_line":""},{"line_number":166,"context_line":"Store pipeline and tenant state in Zookeeper"}],"source_content_type":"text/x-rst","patch_set":4,"id":"dfd5e7cf_003327ad","line":163,"in_reply_to":"dfd5e7cf_2df8b2c2","updated":"2019-01-08 16:08:25.000000000","message":"Ok, that makes sense. So we could store the global config per project and branch in distinct objects (because the max blob size is by default 1MB) but lock and checkpoint the complete global config.","commit_id":"6ba86da65fa6a25cb72e1deecf2f17030e121358"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"28c6289022e4567efa909af1824dd84ad13680d7","unresolved":false,"context_lines":[{"line_number":160,"context_line":"stored in the path ``/zuul/tenant/\u003ctenant\u003e/config/\u003cproject\u003e/\u003cbranch\u003e`` as json"},{"line_number":161,"context_line":"blobs per branch. If this data gets large we can consider compressing these"},{"line_number":162,"context_line":"blobs. If we do that we should support both formats in parallel to be able"},{"line_number":163,"context_line":"to disable the compression e.g. for debugging purposes."},{"line_number":164,"context_line":""},{"line_number":165,"context_line":""},{"line_number":166,"context_line":"Store pipeline and tenant state in Zookeeper"}],"source_content_type":"text/x-rst","patch_set":4,"id":"dfd5e7cf_9218df81","line":163,"in_reply_to":"dfd5e7cf_9288df1f","updated":"2019-01-07 23:35:14.000000000","message":"I believe that the way kafka does this is to only record journal type data in zookeeper and refer to the external source of truth as necessary to roll things through the system.\n\nApplying a similar appraoch to Zuul I think we could record canonical repo locations and branch/sha1 data in zk to represent the most up to date config and if a local scheduler\u0027s config is behind that sha1 we update to it.\n\nFor updates we would have events (driven by the active and passive event handling above) that would update the sha1 in zk.\n\nI don\u0027t know that this is necessarily better than what is proposed above but would minimize the data stored in zk.","commit_id":"6ba86da65fa6a25cb72e1deecf2f17030e121358"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"c836757b5f9d0bdf9f133e2c94d6d6bf8b50f90f","unresolved":false,"context_lines":[{"line_number":191,"context_line":"there. But this can be done separately and is not within the scope of this spec."},{"line_number":192,"context_line":"If sql is mandatory during the implementation phase it can be used otherwise"},{"line_number":193,"context_line":"the times database will be stored in zk just like it is now on disk of the"},{"line_number":194,"context_line":"scheduler."},{"line_number":195,"context_line":""},{"line_number":196,"context_line":""},{"line_number":197,"context_line":"Executor via Zookeeper"}],"source_content_type":"text/x-rst","patch_set":4,"id":"dfd5e7cf_ad36e2b3","line":194,"updated":"2019-01-07 23:42:53.000000000","message":"Everything is worse without SQL.  We should just use it.","commit_id":"6ba86da65fa6a25cb72e1deecf2f17030e121358"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"a8efb8d2c2bcc5aecee220301e55090390888468","unresolved":false,"context_lines":[{"line_number":191,"context_line":"there. But this can be done separately and is not within the scope of this spec."},{"line_number":192,"context_line":"If sql is mandatory during the implementation phase it can be used otherwise"},{"line_number":193,"context_line":"the times database will be stored in zk just like it is now on disk of the"},{"line_number":194,"context_line":"scheduler."},{"line_number":195,"context_line":""},{"line_number":196,"context_line":""},{"line_number":197,"context_line":"Executor via Zookeeper"}],"source_content_type":"text/x-rst","patch_set":4,"id":"dfd5e7cf_327d731c","line":194,"updated":"2019-01-07 22:35:17.000000000","message":"We need to move the time database out of the filesystem, so let\u0027s make \"drop the time database and require SQL\" part of this effort.  I don\u0027t think we should use zk for it.","commit_id":"6ba86da65fa6a25cb72e1deecf2f17030e121358"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"28c6289022e4567efa909af1824dd84ad13680d7","unresolved":false,"context_lines":[{"line_number":191,"context_line":"there. But this can be done separately and is not within the scope of this spec."},{"line_number":192,"context_line":"If sql is mandatory during the implementation phase it can be used otherwise"},{"line_number":193,"context_line":"the times database will be stored in zk just like it is now on disk of the"},{"line_number":194,"context_line":"scheduler."},{"line_number":195,"context_line":""},{"line_number":196,"context_line":""},{"line_number":197,"context_line":"Executor via Zookeeper"}],"source_content_type":"text/x-rst","patch_set":4,"id":"dfd5e7cf_6d08fa27","line":194,"in_reply_to":"dfd5e7cf_327d731c","updated":"2019-01-07 23:35:14.000000000","message":"Is it feasible to require SQL db for time db, but continue to make the SQL db optional?","commit_id":"6ba86da65fa6a25cb72e1deecf2f17030e121358"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"feeb13db5761efcd71624ac447c7a8d072b82c5e","unresolved":false,"context_lines":[{"line_number":191,"context_line":"there. But this can be done separately and is not within the scope of this spec."},{"line_number":192,"context_line":"If sql is mandatory during the implementation phase it can be used otherwise"},{"line_number":193,"context_line":"the times database will be stored in zk just like it is now on disk of the"},{"line_number":194,"context_line":"scheduler."},{"line_number":195,"context_line":""},{"line_number":196,"context_line":""},{"line_number":197,"context_line":"Executor via Zookeeper"}],"source_content_type":"text/x-rst","patch_set":4,"id":"dfd5e7cf_40184f1c","line":194,"in_reply_to":"dfd5e7cf_ad36e2b3","updated":"2019-01-08 16:08:25.000000000","message":"ok, that makes sense. I\u0027ll change that.","commit_id":"6ba86da65fa6a25cb72e1deecf2f17030e121358"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"04e196ef5cc58db8122597dc5fac761461960943","unresolved":false,"context_lines":[{"line_number":206,"context_line":"job execution. This way the scheduler doesn\u0027t need to lock the nodes so it can"},{"line_number":207,"context_line":"be restarted without loosing the nodes of all jobs."},{"line_number":208,"context_line":""},{"line_number":209,"context_line":"During execution it can directly update the state of the job in the pipeline."},{"line_number":210,"context_line":""},{"line_number":211,"context_line":""},{"line_number":212,"context_line":"Parallelize pipeline processing"}],"source_content_type":"text/x-rst","patch_set":4,"id":"dfd5e7cf_2c93b3a5","line":209,"updated":"2019-01-09 09:50:00.000000000","message":"Thinking more about that I think we\u0027ll still need an execution request similar to the node requests in nodepool as a queue at a specific location. Otherwise all executors will have to do permanent searches over all pipelines. Further in order to keep the event based approach the executor probably needs to add finished/updated events to the respective pipeline event queues.","commit_id":"6ba86da65fa6a25cb72e1deecf2f17030e121358"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"01fcd71fcaca866bdced8e0c802a24abf9cf55b4","unresolved":false,"context_lines":[{"line_number":206,"context_line":"job execution. This way the scheduler doesn\u0027t need to lock the nodes so it can"},{"line_number":207,"context_line":"be restarted without loosing the nodes of all jobs."},{"line_number":208,"context_line":""},{"line_number":209,"context_line":"During execution it can directly update the state of the job in the pipeline."},{"line_number":210,"context_line":""},{"line_number":211,"context_line":""},{"line_number":212,"context_line":"Parallelize pipeline processing"}],"source_content_type":"text/x-rst","patch_set":4,"id":"dfd5e7cf_e974be9b","line":209,"updated":"2019-01-09 08:46:21.000000000","message":"Thinking more about that I think we\u0027ll still need an execution request similar to the node requests in nodepool as a queue at a specific location. Otherwise all executors will have to do permanent searches over all pipelines. Further in order to keep the event based approach the executor probably needs to add finished/updated events to the respective pipeline event queues.","commit_id":"6ba86da65fa6a25cb72e1deecf2f17030e121358"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"28c6289022e4567efa909af1824dd84ad13680d7","unresolved":false,"context_lines":[{"line_number":206,"context_line":"job execution. This way the scheduler doesn\u0027t need to lock the nodes so it can"},{"line_number":207,"context_line":"be restarted without loosing the nodes of all jobs."},{"line_number":208,"context_line":""},{"line_number":209,"context_line":"During execution it can directly update the state of the job in the pipeline."},{"line_number":210,"context_line":""},{"line_number":211,"context_line":""},{"line_number":212,"context_line":"Parallelize pipeline processing"}],"source_content_type":"text/x-rst","patch_set":4,"id":"dfd5e7cf_ad4ca276","line":209,"updated":"2019-01-07 23:35:14.000000000","message":"We may want to capture how we expect to recover in the event of losing an executor. If that happens the running job state is lost and we need to restart the jobs with new nodesets.\n\nI think this means we need to have watches for the executor locks on the schedulers? And handle that lock going away without job completed data in the data structure? And then we want to be able to handle this in multiple schedulers? Anyways it seems sufficiently complicated that we may want something written down here about how we can handle it.","commit_id":"6ba86da65fa6a25cb72e1deecf2f17030e121358"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"feeb13db5761efcd71624ac447c7a8d072b82c5e","unresolved":false,"context_lines":[{"line_number":206,"context_line":"job execution. This way the scheduler doesn\u0027t need to lock the nodes so it can"},{"line_number":207,"context_line":"be restarted without loosing the nodes of all jobs."},{"line_number":208,"context_line":""},{"line_number":209,"context_line":"During execution it can directly update the state of the job in the pipeline."},{"line_number":210,"context_line":""},{"line_number":211,"context_line":""},{"line_number":212,"context_line":"Parallelize pipeline processing"}],"source_content_type":"text/x-rst","patch_set":4,"id":"dfd5e7cf_40edafed","line":209,"in_reply_to":"dfd5e7cf_ad4ca276","updated":"2019-01-08 16:08:25.000000000","message":"I\u0027ll think more about that and extend this.","commit_id":"6ba86da65fa6a25cb72e1deecf2f17030e121358"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"65f4faac813b1101f797e192a7d2f50db0dd3d17","unresolved":false,"context_lines":[{"line_number":232,"context_line":"fails it will continue with pipelines in the the next tenant having outstanding"},{"line_number":233,"context_line":"events. If it got the tenant lock it will try to lock the pipeline. If it fails"},{"line_number":234,"context_line":"it will continue with the next pipeline. If it succeeds it will process all"},{"line_number":235,"context_line":"outstanding events of that pipeline."},{"line_number":236,"context_line":""},{"line_number":237,"context_line":"In order to reduce stalls when doing reconfigurations or tenant reconfigurations"},{"line_number":238,"context_line":"we can easily run more than one pipeline processor in a thread pool per"}],"source_content_type":"text/x-rst","patch_set":4,"id":"bfdaf3ff_d5ab5f86","line":235,"updated":"2019-01-16 18:36:58.000000000","message":"Let\u0027s wait and see if that\u0027s a problem; the max processing time sounds complex and I don\u0027t think it should be necessary.","commit_id":"6ba86da65fa6a25cb72e1deecf2f17030e121358"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"28c6289022e4567efa909af1824dd84ad13680d7","unresolved":false,"context_lines":[{"line_number":232,"context_line":"fails it will continue with pipelines in the the next tenant having outstanding"},{"line_number":233,"context_line":"events. If it got the tenant lock it will try to lock the pipeline. If it fails"},{"line_number":234,"context_line":"it will continue with the next pipeline. If it succeeds it will process all"},{"line_number":235,"context_line":"outstanding events of that pipeline."},{"line_number":236,"context_line":""},{"line_number":237,"context_line":"In order to reduce stalls when doing reconfigurations or tenant reconfigurations"},{"line_number":238,"context_line":"we can easily run more than one pipeline processor in a thread pool per"}],"source_content_type":"text/x-rst","patch_set":4,"id":"dfd5e7cf_8d82fe4e","line":235,"updated":"2019-01-07 23:35:14.000000000","message":"This depth first approach may also starve other pipelines and tenants. Do we need to rotate in a breadth first fashion instead to ensure events coming in sufficiently quickly prevent us from processing another tenant or pipeline?","commit_id":"6ba86da65fa6a25cb72e1deecf2f17030e121358"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"feeb13db5761efcd71624ac447c7a8d072b82c5e","unresolved":false,"context_lines":[{"line_number":232,"context_line":"fails it will continue with pipelines in the the next tenant having outstanding"},{"line_number":233,"context_line":"events. If it got the tenant lock it will try to lock the pipeline. If it fails"},{"line_number":234,"context_line":"it will continue with the next pipeline. If it succeeds it will process all"},{"line_number":235,"context_line":"outstanding events of that pipeline."},{"line_number":236,"context_line":""},{"line_number":237,"context_line":"In order to reduce stalls when doing reconfigurations or tenant reconfigurations"},{"line_number":238,"context_line":"we can easily run more than one pipeline processor in a thread pool per"}],"source_content_type":"text/x-rst","patch_set":4,"id":"dfd5e7cf_209e6b63","line":235,"in_reply_to":"dfd5e7cf_8d82fe4e","updated":"2019-01-08 16:08:25.000000000","message":"The event loop is expected to loop fast as it is already the case. Further I\u0027d expect multiple pipeline processor threads per scheduler so that might be less of a problem. But we could add a max processing time as a safety net that forces switching to the next tenant after some threshold.","commit_id":"6ba86da65fa6a25cb72e1deecf2f17030e121358"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"28c6289022e4567efa909af1824dd84ad13680d7","unresolved":false,"context_lines":[{"line_number":262,"context_line":"Further we will encrypt every secret that is stored in zookeeper using a"},{"line_number":263,"context_line":"symmetric cipher with a shared key that is known to all zuul services but not"},{"line_number":264,"context_line":"zookeeper. This way we can avoid dumping decrypted secrets into the transaction"},{"line_number":265,"context_line":"log of zookeeper."}],"source_content_type":"text/x-rst","patch_set":4,"id":"dfd5e7cf_8da99ee0","line":265,"updated":"2019-01-07 23:35:14.000000000","message":"Setting things up in the kafka model as described above may simplify this? We wouldn\u0027t need to store any secrets in zk, but would need to share keys.","commit_id":"6ba86da65fa6a25cb72e1deecf2f17030e121358"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"feeb13db5761efcd71624ac447c7a8d072b82c5e","unresolved":false,"context_lines":[{"line_number":262,"context_line":"Further we will encrypt every secret that is stored in zookeeper using a"},{"line_number":263,"context_line":"symmetric cipher with a shared key that is known to all zuul services but not"},{"line_number":264,"context_line":"zookeeper. This way we can avoid dumping decrypted secrets into the transaction"},{"line_number":265,"context_line":"log of zookeeper."}],"source_content_type":"text/x-rst","patch_set":4,"id":"dfd5e7cf_404ccfc6","line":265,"in_reply_to":"dfd5e7cf_8da99ee0","updated":"2019-01-08 16:08:25.000000000","message":"That\u0027s exactly the case here. We share keys to decrypt the secret stuff (like secrets or kubernetes credentials from nodepool). Not everything will be necessarily somewhere in git. Further doing cat operations for every dereference is much more complicated and expensive.","commit_id":"6ba86da65fa6a25cb72e1deecf2f17030e121358"},{"author":{"_account_id":3099,"name":"David Shrewsbury","email":"dshrewsb@redhat.com","username":"dshrews"},"change_message_id":"ef29c8d54e3bae8ff4166143721ba035fc9b1221","unresolved":false,"context_lines":[{"line_number":61,"context_line":"As it is today different event types will have different priorities. This will"},{"line_number":62,"context_line":"be expressed like in node-requests with a prefix."},{"line_number":63,"context_line":""},{"line_number":64,"context_line":"The event queues will be stored in Zookeeper in the following paths:"},{"line_number":65,"context_line":""},{"line_number":66,"context_line":"* ``/zuul/events/connection/\u003cconnection name\u003e/\u003csequence\u003e``: Event queue of a"},{"line_number":67,"context_line":"  connection"}],"source_content_type":"text/x-rst","patch_set":5,"id":"bfdaf3ff_8f7cd11f","line":64,"range":{"start_line":64,"start_character":0,"end_line":64,"end_character":68},"updated":"2019-01-17 16:53:59.000000000","message":"This raises some concerns for me. Notably:\n\n* Lots of events (which we are sure to have) will cause event sequence numbers to rollover. Could that put a wrinkle in processing the events in order?\n\n* Assuming you plan to use the kazoo queue recipe, that is documented to be buggy, causing memory leaks over long periods of usage. https://kazoo.readthedocs.io/en/latest/api/recipe/queue.html#module-kazoo.recipe.queue","commit_id":"5e9dea3be7c11fb0826727676907b6e37bd09f48"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"216c7e573e248b901e64c8020b8c3abc1c3d47af","unresolved":false,"context_lines":[{"line_number":61,"context_line":"As it is today different event types will have different priorities. This will"},{"line_number":62,"context_line":"be expressed like in node-requests with a prefix."},{"line_number":63,"context_line":""},{"line_number":64,"context_line":"The event queues will be stored in Zookeeper in the following paths:"},{"line_number":65,"context_line":""},{"line_number":66,"context_line":"* ``/zuul/events/connection/\u003cconnection name\u003e/\u003csequence\u003e``: Event queue of a"},{"line_number":67,"context_line":"  connection"}],"source_content_type":"text/x-rst","patch_set":5,"id":"bfdaf3ff_cf02d96f","line":64,"updated":"2019-01-17 16:59:40.000000000","message":"We could look a little harder at dropping these event queues.  Zuul 1.0\u0027s scheduler was very much event-driven, with the idea that if, say, a build finished, we only needed to look at that queue item.  But that was error-prone, and Zuul has gotten much more complex since then.  Now we do some actions on events, but not very much, and mostly we just use the events to set a flag saying \"run the pipeline queue processor\".  We might be able to do the things we need to do quickly with watches on objects and otherwise just run the queue processor.  If folks think this approach is worthwhile, I can take a closer look at the current queues and figure out how we could rearrange them.","commit_id":"5e9dea3be7c11fb0826727676907b6e37bd09f48"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"cc4033dbcc8fa43edfe78d5abd4928909625e707","unresolved":false,"context_lines":[{"line_number":166,"context_line":"updated incrementally."},{"line_number":167,"context_line":""},{"line_number":168,"context_line":"The configs will be stored in the path ``/zuul/config/\u003cproject\u003e/\u003cbranch\u003e`` as"},{"line_number":169,"context_line":"json blobs per branch. If this data gets large we can consider compressing these"},{"line_number":170,"context_line":"blobs. If we do that we should support both formats in parallel to be able"},{"line_number":171,"context_line":"to disable the compression e.g. for debugging purposes."},{"line_number":172,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"bfdaf3ff_d55cbf7f","line":169,"updated":"2019-01-16 18:50:16.000000000","message":"We can also segment it if necessary.  /zuul/config/project/branch/jobs/ could be a directory of json blobs.  We wouldn\u0027t need to make one zobject per job -- we would end up with too many zobjects that way.  But we could have /zuul/config/project/branch/jobs/1 be the first 100 jobs, and jobs/2 be the next 100.  That way we remove the size limitation, and when small changes happen in big repos, we reduce the amount of data we need to change in zk (for example, we might only replace the jobs/2 object if job #157 is updated).","commit_id":"5e9dea3be7c11fb0826727676907b6e37bd09f48"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"e40c09ac4d4bd166e81f45ce133fe3e7851151df","unresolved":false,"context_lines":[{"line_number":166,"context_line":"updated incrementally."},{"line_number":167,"context_line":""},{"line_number":168,"context_line":"The configs will be stored in the path ``/zuul/config/\u003cproject\u003e/\u003cbranch\u003e`` as"},{"line_number":169,"context_line":"json blobs per branch. If this data gets large we can consider compressing these"},{"line_number":170,"context_line":"blobs. If we do that we should support both formats in parallel to be able"},{"line_number":171,"context_line":"to disable the compression e.g. for debugging purposes."},{"line_number":172,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"bfdaf3ff_f53ea3d8","line":169,"in_reply_to":"bfdaf3ff_d55cbf7f","updated":"2019-01-16 18:58:58.000000000","message":"I like this idea","commit_id":"5e9dea3be7c11fb0826727676907b6e37bd09f48"},{"author":{"_account_id":30637,"name":"Jan Kubovy","email":"jan.kubovy@bmw.de","username":"kubovy"},"change_message_id":"4c7ae2252eacb6d914c036cd302910175ce8c06f","unresolved":false,"context_lines":[{"line_number":167,"context_line":""},{"line_number":168,"context_line":"The configs will be stored in the path ``/zuul/config/\u003cproject\u003e/\u003cbranch\u003e`` as"},{"line_number":169,"context_line":"json blobs per branch. If this data gets large we can consider compressing these"},{"line_number":170,"context_line":"blobs. If we do that we should support both formats in parallel to be able"},{"line_number":171,"context_line":"to disable the compression e.g. for debugging purposes."},{"line_number":172,"context_line":""},{"line_number":173,"context_line":""},{"line_number":174,"context_line":"Store pipeline and tenant state in Zookeeper"}],"source_content_type":"text/x-rst","patch_set":5,"id":"5faad753_4c91fc46","line":171,"range":{"start_line":170,"start_character":7,"end_line":171,"end_character":55},"updated":"2019-09-13 14:36:12.000000000","message":"I would worry that if the compression\u0027s purpose it to fit certain limit size turning it off may cause some problems (in case of e.g. debugging). Therefore maybe a more broken up format, i.e., the one mentioned above may be a better approach.","commit_id":"5e9dea3be7c11fb0826727676907b6e37bd09f48"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"c8ca3d708c85f9cec6aa11f2d2f9f0bd977c9ccc","unresolved":false,"context_lines":[{"line_number":208,"context_line":"the primary. If there are more configured connections one will need to be"},{"line_number":209,"context_line":"configured as primary database. Every pipeline will report to the primary"},{"line_number":210,"context_line":"database in any case. However it will still be possible to attach further sql"},{"line_number":211,"context_line":"reporters to any pipelines."},{"line_number":212,"context_line":""},{"line_number":213,"context_line":"The primary database can be used to query the last 10 successful build times"},{"line_number":214,"context_line":"and use this as the times database."}],"source_content_type":"text/x-rst","patch_set":5,"id":"bfdaf3ff_70e08197","line":211,"updated":"2019-01-16 19:33:22.000000000","message":"I feel like with the API and dashboard allowing flexible queries (including pipeline), we\u0027ve largely superceded any use like that.","commit_id":"5e9dea3be7c11fb0826727676907b6e37bd09f48"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"cc4033dbcc8fa43edfe78d5abd4928909625e707","unresolved":false,"context_lines":[{"line_number":208,"context_line":"the primary. If there are more configured connections one will need to be"},{"line_number":209,"context_line":"configured as primary database. Every pipeline will report to the primary"},{"line_number":210,"context_line":"database in any case. However it will still be possible to attach further sql"},{"line_number":211,"context_line":"reporters to any pipelines."},{"line_number":212,"context_line":""},{"line_number":213,"context_line":"The primary database can be used to query the last 10 successful build times"},{"line_number":214,"context_line":"and use this as the times database."}],"source_content_type":"text/x-rst","patch_set":5,"id":"bfdaf3ff_f55f836f","line":211,"updated":"2019-01-16 18:50:16.000000000","message":"Is there a reason to still have sql reporters in pipelines?  Why not just attach them in tenants (so different tenants can use different sql connections, but each tenant always uses exactly one)?","commit_id":"5e9dea3be7c11fb0826727676907b6e37bd09f48"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"a98f32a9d2442ecd88e867c7600c13c223b07a60","unresolved":false,"context_lines":[{"line_number":208,"context_line":"the primary. If there are more configured connections one will need to be"},{"line_number":209,"context_line":"configured as primary database. Every pipeline will report to the primary"},{"line_number":210,"context_line":"database in any case. However it will still be possible to attach further sql"},{"line_number":211,"context_line":"reporters to any pipelines."},{"line_number":212,"context_line":""},{"line_number":213,"context_line":"The primary database can be used to query the last 10 successful build times"},{"line_number":214,"context_line":"and use this as the times database."}],"source_content_type":"text/x-rst","patch_set":5,"id":"bfdaf3ff_d80cd342","line":211,"updated":"2019-01-18 08:47:58.000000000","message":"Ok, I\u0027m happy to remove the sql reporters from the pipeline config entirely. So we could define an sql connection as default in zuul.conf and make it possible to override it per tenant in the tenant config.","commit_id":"5e9dea3be7c11fb0826727676907b6e37bd09f48"},{"author":{"_account_id":30637,"name":"Jan Kubovy","email":"jan.kubovy@bmw.de","username":"kubovy"},"change_message_id":"4c7ae2252eacb6d914c036cd302910175ce8c06f","unresolved":false,"context_lines":[{"line_number":208,"context_line":"the primary. If there are more configured connections one will need to be"},{"line_number":209,"context_line":"configured as primary database. Every pipeline will report to the primary"},{"line_number":210,"context_line":"database in any case. However it will still be possible to attach further sql"},{"line_number":211,"context_line":"reporters to any pipelines."},{"line_number":212,"context_line":""},{"line_number":213,"context_line":"The primary database can be used to query the last 10 successful build times"},{"line_number":214,"context_line":"and use this as the times database."}],"source_content_type":"text/x-rst","patch_set":5,"id":"5faad753_2c8600dd","line":211,"in_reply_to":"bfdaf3ff_d80cd342","updated":"2019-09-13 14:36:12.000000000","message":"Since the purpose of this is High-Availability shouldn\u0027t we think about some HA-Store. An SQL database feels a bit like a possible future bottleneck here when multiple, possibly many, schedulers will try to write there simultaneously.","commit_id":"5e9dea3be7c11fb0826727676907b6e37bd09f48"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"e40c09ac4d4bd166e81f45ce133fe3e7851151df","unresolved":false,"context_lines":[{"line_number":208,"context_line":"the primary. If there are more configured connections one will need to be"},{"line_number":209,"context_line":"configured as primary database. Every pipeline will report to the primary"},{"line_number":210,"context_line":"database in any case. However it will still be possible to attach further sql"},{"line_number":211,"context_line":"reporters to any pipelines."},{"line_number":212,"context_line":""},{"line_number":213,"context_line":"The primary database can be used to query the last 10 successful build times"},{"line_number":214,"context_line":"and use this as the times database."}],"source_content_type":"text/x-rst","patch_set":5,"id":"bfdaf3ff_f570e3de","line":211,"in_reply_to":"bfdaf3ff_f55f836f","updated":"2019-01-16 18:58:58.000000000","message":"The reason was mostly backwards compatibility as today one can choose to report per pipeline. But I\u0027m open to just remove this as this is probably just a hypothetical use case.","commit_id":"5e9dea3be7c11fb0826727676907b6e37bd09f48"},{"author":{"_account_id":30637,"name":"Jan Kubovy","email":"jan.kubovy@bmw.de","username":"kubovy"},"change_message_id":"4c7ae2252eacb6d914c036cd302910175ce8c06f","unresolved":false,"context_lines":[{"line_number":210,"context_line":"database in any case. However it will still be possible to attach further sql"},{"line_number":211,"context_line":"reporters to any pipelines."},{"line_number":212,"context_line":""},{"line_number":213,"context_line":"The primary database can be used to query the last 10 successful build times"},{"line_number":214,"context_line":"and use this as the times database."},{"line_number":215,"context_line":""},{"line_number":216,"context_line":""},{"line_number":217,"context_line":"Executor via Zookeeper"}],"source_content_type":"text/x-rst","patch_set":5,"id":"5faad753_ec546855","line":214,"range":{"start_line":213,"start_character":0,"end_line":214,"end_character":35},"updated":"2019-09-13 14:36:12.000000000","message":"If the purpose of this is to know the times of last 10 successful builds wouldn\u0027t it be worth to think about storing  it in Zookeeper too. And by that limiting the complexity of both, code and deployment - less mandatory deployment nodes?","commit_id":"5e9dea3be7c11fb0826727676907b6e37bd09f48"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"ced4fbb38c4bf3405dbf26c56e7d76276dccf59c","unresolved":false,"context_lines":[{"line_number":210,"context_line":"database in any case. However it will still be possible to attach further sql"},{"line_number":211,"context_line":"reporters to any pipelines."},{"line_number":212,"context_line":""},{"line_number":213,"context_line":"The primary database can be used to query the last 10 successful build times"},{"line_number":214,"context_line":"and use this as the times database."},{"line_number":215,"context_line":""},{"line_number":216,"context_line":""},{"line_number":217,"context_line":"Executor via Zookeeper"}],"source_content_type":"text/x-rst","patch_set":5,"id":"3fa7e38b_d573e2b2","line":214,"range":{"start_line":213,"start_character":0,"end_line":214,"end_character":35},"in_reply_to":"5faad753_ec546855","updated":"2020-01-09 15:31:58.000000000","message":"We anyway store all builds in the db. However we should think about caching those times to not overload the db because of that.","commit_id":"5e9dea3be7c11fb0826727676907b6e37bd09f48"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"cc4033dbcc8fa43edfe78d5abd4928909625e707","unresolved":false,"context_lines":[{"line_number":284,"context_line":"it will continue with the next pipeline. If it succeeds it will process all"},{"line_number":285,"context_line":"outstanding events of that pipeline. To prevent starvation of tenants we can"},{"line_number":286,"context_line":"define a max processing time after which the pipeline processor will switch to"},{"line_number":287,"context_line":"the next tenant or pipeline even if there are outstanding events."},{"line_number":288,"context_line":""},{"line_number":289,"context_line":"In order to reduce stalls when doing reconfigurations or tenant reconfigurations"},{"line_number":290,"context_line":"we can easily run more than one pipeline processor in a thread pool per"}],"source_content_type":"text/x-rst","patch_set":5,"id":"bfdaf3ff_9556375c","line":287,"updated":"2019-01-16 18:50:16.000000000","message":"I don\u0027t think we should include the max processing time to start with; I think that will introduce a lot of edge cases we would need to handle and complexity.  If we find we need it we can add it later.","commit_id":"5e9dea3be7c11fb0826727676907b6e37bd09f48"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"e40c09ac4d4bd166e81f45ce133fe3e7851151df","unresolved":false,"context_lines":[{"line_number":284,"context_line":"it will continue with the next pipeline. If it succeeds it will process all"},{"line_number":285,"context_line":"outstanding events of that pipeline. To prevent starvation of tenants we can"},{"line_number":286,"context_line":"define a max processing time after which the pipeline processor will switch to"},{"line_number":287,"context_line":"the next tenant or pipeline even if there are outstanding events."},{"line_number":288,"context_line":""},{"line_number":289,"context_line":"In order to reduce stalls when doing reconfigurations or tenant reconfigurations"},{"line_number":290,"context_line":"we can easily run more than one pipeline processor in a thread pool per"}],"source_content_type":"text/x-rst","patch_set":5,"id":"bfdaf3ff_b5849bb2","line":287,"in_reply_to":"bfdaf3ff_9556375c","updated":"2019-01-16 18:58:58.000000000","message":"++","commit_id":"5e9dea3be7c11fb0826727676907b6e37bd09f48"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"a98f32a9d2442ecd88e867c7600c13c223b07a60","unresolved":false,"context_lines":[{"line_number":314,"context_line":"Further we will encrypt every secret that is stored in zookeeper using a"},{"line_number":315,"context_line":"symmetric cipher with a shared key that is known to all zuul services but not"},{"line_number":316,"context_line":"zookeeper. This way we can avoid dumping decrypted secrets into the transaction"},{"line_number":317,"context_line":"log of zookeeper."}],"source_content_type":"text/x-rst","patch_set":5,"id":"bfdaf3ff_7805a713","line":317,"updated":"2019-01-18 08:47:58.000000000","message":"Good point. I\u0027ll add this. If we can manage it (and I hope this works out) I\u0027d be in favor of a series of moderate changes instead of a big bang release.","commit_id":"5e9dea3be7c11fb0826727676907b6e37bd09f48"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"030a7d38be877af6d5ef39ac22bedcdcafccb350","unresolved":false,"context_lines":[{"line_number":314,"context_line":"Further we will encrypt every secret that is stored in zookeeper using a"},{"line_number":315,"context_line":"symmetric cipher with a shared key that is known to all zuul services but not"},{"line_number":316,"context_line":"zookeeper. This way we can avoid dumping decrypted secrets into the transaction"},{"line_number":317,"context_line":"log of zookeeper."}],"source_content_type":"text/x-rst","patch_set":5,"id":"bfdaf3ff_4f4fc933","line":317,"updated":"2019-01-17 16:46:16.000000000","message":"We should probably add a section on work items.  It would be good to list some of the major steps we need to do, so we can get an idea of how much work will need to be done, whether we can split it up among different authors, and how we should handle merging and releasing.\n\nParticularly, it would be good to figure out whether we want to break it up into a series of moderate changes (such as \"require sql\", \"zk ingestor\", \"zk executor\", \"zk scheduler\"), each of which gets a 3.x release with an upgrade note, or whether we want to maintain a long stack of changes in gerrit with rebasing and land it all then release 4.0, or whether we want to make a feature branch.  I think making the list of work items as a start will help us find the right answer.","commit_id":"5e9dea3be7c11fb0826727676907b6e37bd09f48"}],"doc/source/reference/developer/specs/scale-out-scheduler.rst":[{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"448b11f8c0116d5e757686eb3a0813c5da9af0d0","unresolved":false,"context_lines":[{"line_number":130,"context_line":"events at all. A drawback is that there is a short time when the current leader"},{"line_number":131,"context_line":"stops until the next leader has started event gathering. This could lead to a"},{"line_number":132,"context_line":"few missed events. But as this is the most easiest way we can accept this in"},{"line_number":133,"context_line":"the initial version."},{"line_number":134,"context_line":""},{"line_number":135,"context_line":"If there is a need to guarantee that there is no missed event during a"},{"line_number":136,"context_line":"leadership change the above algorithm can be enhanced later with parallel"}],"source_content_type":"text/x-rst","patch_set":14,"id":"3fa7e38b_688b2216","line":133,"updated":"2020-01-29 19:17:24.000000000","message":"0: I don\u0027t remember that, but would love it if it were the case.  If it doesn\u0027t exist, we can implement gertty-style queries.\n\nOverall, I think this section is fine as written.","commit_id":"06b6eeb56c4c8debf30f50a36e7f8e072e1c48d8"},{"author":{"_account_id":2,"name":"Monty Taylor","email":"mordred@inaugust.com","username":"mordred"},"change_message_id":"e1351e33149cadd0deeccacf73551aca08ab5a1f","unresolved":false,"context_lines":[{"line_number":130,"context_line":"events at all. A drawback is that there is a short time when the current leader"},{"line_number":131,"context_line":"stops until the next leader has started event gathering. This could lead to a"},{"line_number":132,"context_line":"few missed events. But as this is the most easiest way we can accept this in"},{"line_number":133,"context_line":"the initial version."},{"line_number":134,"context_line":""},{"line_number":135,"context_line":"If there is a need to guarantee that there is no missed event during a"},{"line_number":136,"context_line":"leadership change the above algorithm can be enhanced later with parallel"}],"source_content_type":"text/x-rst","patch_set":14,"id":"3fa7e38b_7a9f8450","line":133,"updated":"2020-01-28 10:05:25.000000000","message":"We should also be able to improve the stream-events command - I believe there is a \"get events since X\" api call we could make - so the first thing after a leader election could be \"go grab events since the last serial number I have in zk\". But that\u0027s a big fat \"I think\"","commit_id":"06b6eeb56c4c8debf30f50a36e7f8e072e1c48d8"},{"author":{"_account_id":2,"name":"Monty Taylor","email":"mordred@inaugust.com","username":"mordred"},"change_message_id":"e1351e33149cadd0deeccacf73551aca08ab5a1f","unresolved":false,"context_lines":[{"line_number":143,"context_line":""},{"line_number":144,"context_line":"In case of passive event gathering the events are sent to Zuul typically via"},{"line_number":145,"context_line":"webhooks. These types of events will be received in zuul-web that stores them in"},{"line_number":146,"context_line":"Zookeeper. This type of event gathering is used by the Github driver. In this"},{"line_number":147,"context_line":"case we can have multiple instances but still receive only one event. So we"},{"line_number":148,"context_line":"don\u0027t need to take special care of event deduplication or leader election."},{"line_number":149,"context_line":""}],"source_content_type":"text/x-rst","patch_set":14,"id":"3fa7e38b_fad93427","line":146,"range":{"start_line":146,"start_character":55,"end_line":146,"end_character":68},"updated":"2020-01-28 10:05:25.000000000","message":"We might also want to investigate the gerrit webhook plugin?","commit_id":"06b6eeb56c4c8debf30f50a36e7f8e072e1c48d8"},{"author":{"_account_id":30637,"name":"Jan Kubovy","email":"jan.kubovy@bmw.de","username":"kubovy"},"change_message_id":"f138b152f9f9095601ef8c896802bb7b7a063c83","unresolved":false,"context_lines":[{"line_number":143,"context_line":""},{"line_number":144,"context_line":"In case of passive event gathering the events are sent to Zuul typically via"},{"line_number":145,"context_line":"webhooks. These types of events will be received in zuul-web that stores them in"},{"line_number":146,"context_line":"Zookeeper. This type of event gathering is used by the Github driver. In this"},{"line_number":147,"context_line":"case we can have multiple instances but still receive only one event. So we"},{"line_number":148,"context_line":"don\u0027t need to take special care of event deduplication or leader election."},{"line_number":149,"context_line":""}],"source_content_type":"text/x-rst","patch_set":14,"id":"3fa7e38b_3911f35d","line":146,"range":{"start_line":146,"start_character":55,"end_line":146,"end_character":68},"in_reply_to":"3fa7e38b_fad93427","updated":"2020-01-30 13:29:01.000000000","message":"Sure, do you want to list it here right away? Or will we leave it for the task eventually.","commit_id":"06b6eeb56c4c8debf30f50a36e7f8e072e1c48d8"},{"author":{"_account_id":2,"name":"Monty Taylor","email":"mordred@inaugust.com","username":"mordred"},"change_message_id":"e1351e33149cadd0deeccacf73551aca08ab5a1f","unresolved":false,"context_lines":[{"line_number":144,"context_line":"In case of passive event gathering the events are sent to Zuul typically via"},{"line_number":145,"context_line":"webhooks. These types of events will be received in zuul-web that stores them in"},{"line_number":146,"context_line":"Zookeeper. This type of event gathering is used by the Github driver. In this"},{"line_number":147,"context_line":"case we can have multiple instances but still receive only one event. So we"},{"line_number":148,"context_line":"don\u0027t need to take special care of event deduplication or leader election."},{"line_number":149,"context_line":""},{"line_number":150,"context_line":""}],"source_content_type":"text/x-rst","patch_set":14,"id":"3fa7e38b_9ac36039","line":147,"updated":"2020-01-28 10:05:25.000000000","message":"might be worth mentioning, if you do another pass - that multiple instances behind a load balancer are safe in this case, and in fact are probably recommended for things like the github driver.","commit_id":"06b6eeb56c4c8debf30f50a36e7f8e072e1c48d8"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"448b11f8c0116d5e757686eb3a0813c5da9af0d0","unresolved":false,"context_lines":[{"line_number":172,"context_line":"and ``/zuul/config/\u003cproject\u003e/\u003cbranch\u003e/jobs/2`` be the next 100."},{"line_number":173,"context_line":"This should allow to stay below the 1MB limit per node and, at the same time,"},{"line_number":174,"context_line":"reducing the amount of data needed to be updated in case of small changes"},{"line_number":175,"context_line":"in big repositories."},{"line_number":176,"context_line":""},{"line_number":177,"context_line":""},{"line_number":178,"context_line":"Store pipeline and tenant state in Zookeeper"}],"source_content_type":"text/x-rst","patch_set":14,"id":"3fa7e38b_888e9e23","line":175,"updated":"2020-01-29 19:17:24.000000000","message":"0: It\u0027s probably worth noting that we would also have /nodesets/ etc for the other config object types.\n\nThis does mean there will be a small amount of additional processing needed to split the raw yaml into chunks corresponding to the config objects.  That shouldn\u0027t be too hard.","commit_id":"06b6eeb56c4c8debf30f50a36e7f8e072e1c48d8"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"448b11f8c0116d5e757686eb3a0813c5da9af0d0","unresolved":false,"context_lines":[{"line_number":214,"context_line":"the primary. If there are more configured connections one will need to be"},{"line_number":215,"context_line":"configured as primary database. Reporters will use the primary"},{"line_number":216,"context_line":"database in any case. However it will still be possible to attach further sql"},{"line_number":217,"context_line":"reporters to any tenant."},{"line_number":218,"context_line":""},{"line_number":219,"context_line":"The primary database can be used to query the last 10 successful build times"},{"line_number":220,"context_line":"and use this as the times database."}],"source_content_type":"text/x-rst","patch_set":14,"id":"3fa7e38b_2891aa45","line":217,"updated":"2020-01-29 19:17:24.000000000","message":"-1: I missed the use case for multiple databases per tenant.  I think we should only have one sql connection per tenant.  I don\u0027t see the purpose of having more than one, and we\u0027ve already seen that having more than one is fairly complicated to deal with.\n\nIf someone has a use-case for more than one database/connection per tenant, let\u0027s describe it here so we know why we\u0027re supporting the extra complexity.  Otherwise, let\u0027s simplify this so that there is a single one per tenant (and it doesn\u0027t need to be specified if there is a global default).","commit_id":"06b6eeb56c4c8debf30f50a36e7f8e072e1c48d8"},{"author":{"_account_id":9311,"name":"Tristan Cacqueray","email":"tdecacqu@redhat.com","username":"tristanC"},"change_message_id":"a912786eafa1a4d156ebbaeebfb2fafb3c92f739","unresolved":false,"context_lines":[{"line_number":214,"context_line":"the primary. If there are more configured connections one will need to be"},{"line_number":215,"context_line":"configured as primary database. Reporters will use the primary"},{"line_number":216,"context_line":"database in any case. However it will still be possible to attach further sql"},{"line_number":217,"context_line":"reporters to any tenant."},{"line_number":218,"context_line":""},{"line_number":219,"context_line":"The primary database can be used to query the last 10 successful build times"},{"line_number":220,"context_line":"and use this as the times database."}],"source_content_type":"text/x-rst","patch_set":14,"id":"3fa7e38b_28c9aa22","line":217,"in_reply_to":"3fa7e38b_2891aa45","updated":"2020-01-29 19:31:47.000000000","message":"One sql connection per zuul or per tenant should covers all of our use-cases.","commit_id":"06b6eeb56c4c8debf30f50a36e7f8e072e1c48d8"},{"author":{"_account_id":7069,"name":"Joshua Hesketh","email":"josh@nitrotech.org","username":"jhesketh"},"change_message_id":"57032d4ded44ac55ddbc732de699b826cffcb86e","unresolved":false,"context_lines":[{"line_number":214,"context_line":"the primary. If there are more configured connections one will need to be"},{"line_number":215,"context_line":"configured as primary database. Reporters will use the primary"},{"line_number":216,"context_line":"database in any case. However it will still be possible to attach further sql"},{"line_number":217,"context_line":"reporters to any tenant."},{"line_number":218,"context_line":""},{"line_number":219,"context_line":"The primary database can be used to query the last 10 successful build times"},{"line_number":220,"context_line":"and use this as the times database."}],"source_content_type":"text/x-rst","patch_set":14,"id":"3fa7e38b_a7a14495","line":217,"in_reply_to":"3fa7e38b_2891aa45","updated":"2020-01-30 00:28:55.000000000","message":"Originally the SQL reporter was just that - a reporter. Thus being able to choose where the results went (much like you can choose with the smtp reporter) was useful. For example, it was a way to get results out of zuul so that you could then do your own analysis. This might have been done by putting a job you were interested in in a pipeline just for this purpose (eg an `sql-pipeline`).\n\nObviously now zuul does a lot more with the information out of the reporter. So the question I think we should look at is even if it is considered a reporter. The \"primary database\" connection described here alludes to something slightly different to a reporter.\n\nSo a single SQL connection for a tenant I think is enough, and a requirement, to achieve what zuul needs to with the data. Specifically this is internal to zuul\u0027s needs and it is debatable if it should even be exposed as a reporter.\n\nThen there is an obvious regression and the question becomes if we should keep SQL reporters, and specifically ones that can use an alternative SQL connection/database.\n\nI think there is an argument to be made that this is still useful. However, I also think it can be separate from this spec and implemented as a standalone reporter at some point.\n\nBut to go further than that, it is likely that any argument or use case made in favour of a separate database can be implemented by other means. For example, by using cross-database views, or by writing a small script to pull out the parts you need from the master database into your analysis database etc.\n\nTherefore, I would suggest that only 1 connection is appropriate and necessary for this spec. I would also suggest investigating not exposing it as a reporter, but just an assumed function, as zuul has expectations about what is actually in the database.","commit_id":"06b6eeb56c4c8debf30f50a36e7f8e072e1c48d8"},{"author":{"_account_id":2,"name":"Monty Taylor","email":"mordred@inaugust.com","username":"mordred"},"change_message_id":"991ecc3201d9869d99098ce8b13f8cc3bd1a9557","unresolved":false,"context_lines":[{"line_number":214,"context_line":"the primary. If there are more configured connections one will need to be"},{"line_number":215,"context_line":"configured as primary database. Reporters will use the primary"},{"line_number":216,"context_line":"database in any case. However it will still be possible to attach further sql"},{"line_number":217,"context_line":"reporters to any tenant."},{"line_number":218,"context_line":""},{"line_number":219,"context_line":"The primary database can be used to query the last 10 successful build times"},{"line_number":220,"context_line":"and use this as the times database."}],"source_content_type":"text/x-rst","patch_set":14,"id":"3fa7e38b_c30f9729","line":217,"in_reply_to":"3fa7e38b_28c9aa22","updated":"2020-01-29 19:48:06.000000000","message":"++","commit_id":"06b6eeb56c4c8debf30f50a36e7f8e072e1c48d8"},{"author":{"_account_id":2,"name":"Monty Taylor","email":"mordred@inaugust.com","username":"mordred"},"change_message_id":"e1351e33149cadd0deeccacf73551aca08ab5a1f","unresolved":false,"context_lines":[{"line_number":221,"context_line":""},{"line_number":222,"context_line":""},{"line_number":223,"context_line":"Executor via Zookeeper"},{"line_number":224,"context_line":"----------------------"},{"line_number":225,"context_line":""},{"line_number":226,"context_line":"In order to prepare for distributed pipeline execution we need to use Zookeeper"},{"line_number":227,"context_line":"for scheduling jobs on the executors. This is needed so that any scheduler can"}],"source_content_type":"text/x-rst","patch_set":14,"id":"3fa7e38b_5a3c082c","line":224,"updated":"2020-01-28 10:05:25.000000000","message":"I think it\u0027s out of scope for this spec - but fdegir has requested resumable executors before... he\u0027s got jobs that sometimes run for several days so an executor interruption is costly for him like a scheduler is for everyone. While I don\u0027t think we need to account for that in this spec - maybe it\u0027s worth noting that it could be a followup ... and executors over zookeeper are likely an important step towards being able to think about that.","commit_id":"06b6eeb56c4c8debf30f50a36e7f8e072e1c48d8"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"448b11f8c0116d5e757686eb3a0813c5da9af0d0","unresolved":false,"context_lines":[{"line_number":273,"context_line":"will receive the trigger, management and result events that are not tenant"},{"line_number":274,"context_line":"specific. The purpose of this queue is to take these events and dispatch them to"},{"line_number":275,"context_line":"the pipeline queues of the tenants as appropriate. This event queue can easily"},{"line_number":276,"context_line":"processed using leader election."},{"line_number":277,"context_line":""},{"line_number":278,"context_line":"We also have tenant global events like tenant reconfigurations. These need"},{"line_number":279,"context_line":"exclusive access to all pipelines in the tenant. So we need a two layer locking"}],"source_content_type":"text/x-rst","patch_set":14,"id":"3fa7e38b_48942635","line":276,"updated":"2020-01-29 19:17:24.000000000","message":"0/-1? We don\u0027t even really need an election, right?  We can just have all the schedulers fight for the lock.  (I guess that\u0027s a form of leader election?  Just a very simple one.)","commit_id":"06b6eeb56c4c8debf30f50a36e7f8e072e1c48d8"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"448b11f8c0116d5e757686eb3a0813c5da9af0d0","unresolved":false,"context_lines":[{"line_number":295,"context_line":"In order to reduce stalls when doing reconfigurations or tenant reconfigurations"},{"line_number":296,"context_line":"we can easily run more than one pipeline processor in a thread pool per"},{"line_number":297,"context_line":"scheduler. This way a tenant that is running a longer reconfiguration won\u0027t"},{"line_number":298,"context_line":"block other tenants."},{"line_number":299,"context_line":""},{"line_number":300,"context_line":""},{"line_number":301,"context_line":"Zuul-web changes"}],"source_content_type":"text/x-rst","patch_set":14,"id":"3fa7e38b_e89e3253","line":298,"updated":"2020-01-29 19:17:24.000000000","message":"-1: I think the only time this would help is when we\u0027re waiting on merger jobs in a reconfiguration.  The rest of the time we will be CPU-bound with the GIL and therefore more threads will only hurt.  So we might want to plan on only having a single pipeline processer in a scheduler, but perhaps handle reconfigurations in a second thread.","commit_id":"06b6eeb56c4c8debf30f50a36e7f8e072e1c48d8"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"fd015ab47dc75662f13a0af998d2587d5ecfb3a6","unresolved":false,"context_lines":[{"line_number":177,"context_line":"first 100 jobs, and ``/zuul/config/\u003cproject\u003e/\u003cbranch\u003e/jobs/2`` be the next 100."},{"line_number":178,"context_line":"This should allow to stay below the 1MB limit per node and, at the same time,"},{"line_number":179,"context_line":"reducing the amount of data needed to be updated in case of small changes"},{"line_number":180,"context_line":"in big repositories."},{"line_number":181,"context_line":""},{"line_number":182,"context_line":""},{"line_number":183,"context_line":"Store pipeline and tenant state in Zookeeper"}],"source_content_type":"text/x-rst","patch_set":15,"id":"3fa7e38b_c4f2336a","line":180,"updated":"2020-01-30 15:26:04.000000000","message":"I just realized a problem with this.  We now support different tenants reading different files, so we still need to store the unparsed config by files.  I see the following options:\n\n1) Add in a filename to the path (eg /zuul/config/\u003cproject\u003e/\u003cbranch\u003e/\u003cfilename\u003e/jobs/1)\n2) Add the filename and drop the object type (eg /zuul/config/\u003cproject\u003e/\u003cbranch\u003e/\u003cfilename\u003e/1), and just shard when the raw string data gets near 1MB.\n\nI lean slightly toward #2.  The downside is that in the case of a large file, a change in the 999th KB would require rewriting objects /1 and /2, because we\u0027re not breaking on a semantic boundary but rather mere byte boundaries.\n\nOn the other hand, most files are probably *much* smaller than that (openstack\u0027s projects.yaml is 137k), so we\u0027re unlikely to need to shard anyway since we\u0027re splitting by files.  This is probably easier to implement too, since it\u0027s close to the current cache system.","commit_id":"22b5dec24c80ccd1d1fc91f5f04ccff1135c0d11"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"a230b47896180302441537899b8c82770b6e37db","unresolved":false,"context_lines":[{"line_number":177,"context_line":"first 100 jobs, and ``/zuul/config/\u003cproject\u003e/\u003cbranch\u003e/jobs/2`` be the next 100."},{"line_number":178,"context_line":"This should allow to stay below the 1MB limit per node and, at the same time,"},{"line_number":179,"context_line":"reducing the amount of data needed to be updated in case of small changes"},{"line_number":180,"context_line":"in big repositories."},{"line_number":181,"context_line":""},{"line_number":182,"context_line":""},{"line_number":183,"context_line":"Store pipeline and tenant state in Zookeeper"}],"source_content_type":"text/x-rst","patch_set":15,"id":"3fa7e38b_bfa4e0a5","line":180,"in_reply_to":"","updated":"2020-01-30 15:57:27.000000000","message":"++ for #2.","commit_id":"22b5dec24c80ccd1d1fc91f5f04ccff1135c0d11"},{"author":{"_account_id":30637,"name":"Jan Kubovy","email":"jan.kubovy@bmw.de","username":"kubovy"},"change_message_id":"5f7d92ebb81f3cba557df777127adbd3fb0830f6","unresolved":false,"context_lines":[{"line_number":177,"context_line":"first 100 jobs, and ``/zuul/config/\u003cproject\u003e/\u003cbranch\u003e/jobs/2`` be the next 100."},{"line_number":178,"context_line":"This should allow to stay below the 1MB limit per node and, at the same time,"},{"line_number":179,"context_line":"reducing the amount of data needed to be updated in case of small changes"},{"line_number":180,"context_line":"in big repositories."},{"line_number":181,"context_line":""},{"line_number":182,"context_line":""},{"line_number":183,"context_line":"Store pipeline and tenant state in Zookeeper"}],"source_content_type":"text/x-rst","patch_set":15,"id":"3fa7e38b_90e1ca34","line":180,"in_reply_to":"3fa7e38b_bfa4e0a5","updated":"2020-01-31 05:50:39.000000000","message":"Done","commit_id":"22b5dec24c80ccd1d1fc91f5f04ccff1135c0d11"},{"author":{"_account_id":30637,"name":"Jan Kubovy","email":"jan.kubovy@bmw.de","username":"kubovy"},"change_message_id":"5f7d92ebb81f3cba557df777127adbd3fb0830f6","unresolved":false,"context_lines":[{"line_number":177,"context_line":"first 100 jobs, and ``/zuul/config/\u003cproject\u003e/\u003cbranch\u003e/jobs/2`` be the next 100."},{"line_number":178,"context_line":"This should allow to stay below the 1MB limit per node and, at the same time,"},{"line_number":179,"context_line":"reducing the amount of data needed to be updated in case of small changes"},{"line_number":180,"context_line":"in big repositories."},{"line_number":181,"context_line":""},{"line_number":182,"context_line":""},{"line_number":183,"context_line":"Store pipeline and tenant state in Zookeeper"}],"source_content_type":"text/x-rst","patch_set":15,"id":"3fa7e38b_b0e64649","line":180,"in_reply_to":"3fa7e38b_c4f2336a","updated":"2020-01-31 05:50:39.000000000","message":"Done","commit_id":"22b5dec24c80ccd1d1fc91f5f04ccff1135c0d11"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"fd015ab47dc75662f13a0af998d2587d5ecfb3a6","unresolved":false,"context_lines":[{"line_number":299,"context_line":"   receiving and event processing components)"},{"line_number":300,"context_line":"#. Parallel pipeline processing"},{"line_number":301,"context_line":"#. Switch to using zookeeper instead of gearman for jobs (keep gearman for"},{"line_number":302,"context_line":"   mergers)"},{"line_number":303,"context_line":""},{"line_number":304,"context_line":"Future work"},{"line_number":305,"context_line":"-----------"}],"source_content_type":"text/x-rst","patch_set":15,"id":"3fa7e38b_648d7fd9","line":302,"updated":"2020-01-30 15:26:04.000000000","message":"Come to think of it, we should really move the mergers to ZK too -- we need to get rid of gearman in order to remove all spofs.  We should add one more section about that (it should be easy though, it\u0027s just a simple job queue).","commit_id":"22b5dec24c80ccd1d1fc91f5f04ccff1135c0d11"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"a230b47896180302441537899b8c82770b6e37db","unresolved":false,"context_lines":[{"line_number":299,"context_line":"   receiving and event processing components)"},{"line_number":300,"context_line":"#. Parallel pipeline processing"},{"line_number":301,"context_line":"#. Switch to using zookeeper instead of gearman for jobs (keep gearman for"},{"line_number":302,"context_line":"   mergers)"},{"line_number":303,"context_line":""},{"line_number":304,"context_line":"Future work"},{"line_number":305,"context_line":"-----------"}],"source_content_type":"text/x-rst","patch_set":15,"id":"3fa7e38b_1f8ad41d","line":302,"in_reply_to":"","updated":"2020-01-30 15:57:27.000000000","message":"Fine for me, I\u0027d like to get rid of it too. We initially kept this in the spec because I think there was some opposition about completely getting rid of gearman.","commit_id":"22b5dec24c80ccd1d1fc91f5f04ccff1135c0d11"},{"author":{"_account_id":30637,"name":"Jan Kubovy","email":"jan.kubovy@bmw.de","username":"kubovy"},"change_message_id":"5f7d92ebb81f3cba557df777127adbd3fb0830f6","unresolved":false,"context_lines":[{"line_number":299,"context_line":"   receiving and event processing components)"},{"line_number":300,"context_line":"#. Parallel pipeline processing"},{"line_number":301,"context_line":"#. Switch to using zookeeper instead of gearman for jobs (keep gearman for"},{"line_number":302,"context_line":"   mergers)"},{"line_number":303,"context_line":""},{"line_number":304,"context_line":"Future work"},{"line_number":305,"context_line":"-----------"}],"source_content_type":"text/x-rst","patch_set":15,"id":"3fa7e38b_50eb5210","line":302,"in_reply_to":"3fa7e38b_648d7fd9","updated":"2020-01-31 05:50:39.000000000","message":"Done","commit_id":"22b5dec24c80ccd1d1fc91f5f04ccff1135c0d11"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"fd015ab47dc75662f13a0af998d2587d5ecfb3a6","unresolved":false,"context_lines":[{"line_number":309,"context_line":"document."},{"line_number":310,"context_line":""},{"line_number":311,"context_line":"Executor via Zookeeper"},{"line_number":312,"context_line":"......................"},{"line_number":313,"context_line":""},{"line_number":314,"context_line":"In order to prepare for distributed pipeline execution we need to use Zookeeper"},{"line_number":315,"context_line":"for scheduling jobs on the executors. This is needed so that any scheduler can"}],"source_content_type":"text/x-rst","patch_set":15,"id":"3fa7e38b_8492fbfa","line":312,"updated":"2020-01-30 15:26:04.000000000","message":"I\u0027m confused why this was moved to future work; isn\u0027t this very much in scope?  The gearman job is only held by one scheduler, so a scheduler restart would lose the connection to the build.","commit_id":"22b5dec24c80ccd1d1fc91f5f04ccff1135c0d11"},{"author":{"_account_id":30637,"name":"Jan Kubovy","email":"jan.kubovy@bmw.de","username":"kubovy"},"change_message_id":"5f7d92ebb81f3cba557df777127adbd3fb0830f6","unresolved":false,"context_lines":[{"line_number":309,"context_line":"document."},{"line_number":310,"context_line":""},{"line_number":311,"context_line":"Executor via Zookeeper"},{"line_number":312,"context_line":"......................"},{"line_number":313,"context_line":""},{"line_number":314,"context_line":"In order to prepare for distributed pipeline execution we need to use Zookeeper"},{"line_number":315,"context_line":"for scheduling jobs on the executors. This is needed so that any scheduler can"}],"source_content_type":"text/x-rst","patch_set":15,"id":"3fa7e38b_70f0cefd","line":312,"in_reply_to":"3fa7e38b_1fc5f45d","updated":"2020-01-31 05:50:39.000000000","message":"I got the idea from a comment on last patch that we may do something later on. Probably just got confused.","commit_id":"22b5dec24c80ccd1d1fc91f5f04ccff1135c0d11"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"a230b47896180302441537899b8c82770b6e37db","unresolved":false,"context_lines":[{"line_number":309,"context_line":"document."},{"line_number":310,"context_line":""},{"line_number":311,"context_line":"Executor via Zookeeper"},{"line_number":312,"context_line":"......................"},{"line_number":313,"context_line":""},{"line_number":314,"context_line":"In order to prepare for distributed pipeline execution we need to use Zookeeper"},{"line_number":315,"context_line":"for scheduling jobs on the executors. This is needed so that any scheduler can"}],"source_content_type":"text/x-rst","patch_set":15,"id":"3fa7e38b_1fc5f45d","line":312,"in_reply_to":"3fa7e38b_8492fbfa","updated":"2020-01-30 15:57:27.000000000","message":"Actually I have no idea why I put this here as I can clearly remember a discussion about this on irc back then. we\u0027ll move this to the main part of the spec","commit_id":"22b5dec24c80ccd1d1fc91f5f04ccff1135c0d11"},{"author":{"_account_id":30637,"name":"Jan Kubovy","email":"jan.kubovy@bmw.de","username":"kubovy"},"change_message_id":"e56b2dbbde4de7ff55de700ab8e8df884d995802","unresolved":false,"context_lines":[{"line_number":170,"context_line":"updated incrementally."},{"line_number":171,"context_line":""},{"line_number":172,"context_line":"The configs will be stored in the path ``/zuul/config/\u003cproject\u003e/\u003cbranch\u003e`` per"},{"line_number":173,"context_line":"branch segmented in ``/zuul/config/\u003cproject\u003e/\u003cbranch\u003e/\u003cfilename\u003e/\u003cshard\u003e``."},{"line_number":174,"context_line":"The ``shard`` is a sequence number and will be used to store larger than 1MB"},{"line_number":175,"context_line":"files due to the limitation mentioned above."},{"line_number":176,"context_line":""}],"source_content_type":"text/x-rst","patch_set":16,"id":"3fa7e38b_43e939a9","line":173,"range":{"start_line":173,"start_character":54,"end_line":173,"end_character":64},"updated":"2020-02-03 10:03:01.000000000","message":"I will change that to to \u003cpath-to-config\u003e since \u003cfilename\u003e is not enough here and may be misleading here.","commit_id":"ca9fa4845d6c2a69eab06931c6115c8b9c293a1d"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"fd40026ed07d14e38c02aa20bcf9f0023df6ed41","unresolved":false,"context_lines":[{"line_number":170,"context_line":"updated incrementally."},{"line_number":171,"context_line":""},{"line_number":172,"context_line":"The configs will be stored in the path ``/zuul/config/\u003cproject\u003e/\u003cbranch\u003e`` per"},{"line_number":173,"context_line":"branch segmented in ``/zuul/config/\u003cproject\u003e/\u003cbranch\u003e/\u003cfilename\u003e/\u003cshard\u003e``."},{"line_number":174,"context_line":"The ``shard`` is a sequence number and will be used to store larger than 1MB"},{"line_number":175,"context_line":"files due to the limitation mentioned above."},{"line_number":176,"context_line":""}],"source_content_type":"text/x-rst","patch_set":16,"id":"3fa7e38b_6400f104","line":173,"updated":"2020-02-03 21:51:12.000000000","message":"Indeed, \"\u003cfilename\u003e\" might be a value like \"zuul.d/jobs.yaml\".  So whatever we have to do to support that.  :)","commit_id":"ca9fa4845d6c2a69eab06931c6115c8b9c293a1d"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"f01c92f311b80f667817e5984dcd23ba861f2393","unresolved":false,"context_lines":[{"line_number":170,"context_line":"updated incrementally."},{"line_number":171,"context_line":""},{"line_number":172,"context_line":"The configs will be stored in the path ``/zuul/config/\u003cproject\u003e/\u003cbranch\u003e`` per"},{"line_number":173,"context_line":"branch segmented in ``/zuul/config/\u003cproject\u003e/\u003cbranch\u003e/\u003cfilename\u003e/\u003cshard\u003e``."},{"line_number":174,"context_line":"The ``shard`` is a sequence number and will be used to store larger than 1MB"},{"line_number":175,"context_line":"files due to the limitation mentioned above."},{"line_number":176,"context_line":""}],"source_content_type":"text/x-rst","patch_set":16,"id":"3fa7e38b_9402d770","line":173,"in_reply_to":"3fa7e38b_6400f104","updated":"2020-02-05 17:50:00.000000000","message":"Done","commit_id":"ca9fa4845d6c2a69eab06931c6115c8b9c293a1d"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"b3941ae68f747d31541a5a4c23c055208e73cc4a","unresolved":false,"context_lines":[{"line_number":67,"context_line":"  connection"},{"line_number":68,"context_line":""},{"line_number":69,"context_line":"* ``/zuul/events/scheduler-global/\u003cprio\u003e-\u003csequence\u003e``: Global event queue of"},{"line_number":70,"context_line":"  scheduler"},{"line_number":71,"context_line":""},{"line_number":72,"context_line":"* ``/zuul/events/tenant/\u003ctenant name\u003e/\u003cpipeline\u003e/\u003cprio\u003e-\u003csequence\u003e``: Pipeline"},{"line_number":73,"context_line":"  event queue"}],"source_content_type":"text/x-rst","patch_set":18,"id":"1fa4df85_e61e376f","line":70,"updated":"2020-02-25 21:30:00.000000000","message":"One thing that has created some confusion with nodepool node request priorities is that they are sorted alphanumerically and not by integer value.\n\nI think we should be consistent in the priority system we use here to avoid further confusion.","commit_id":"5a52bac6459821dd8656b79d6d7680e317a18fa2"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"b3941ae68f747d31541a5a4c23c055208e73cc4a","unresolved":false,"context_lines":[{"line_number":289,"context_line":"it will continue with the next pipeline. If it succeeds it will process all"},{"line_number":290,"context_line":"outstanding events of that pipeline. To prevent starvation of tenants we can"},{"line_number":291,"context_line":"define a max processing time after which the pipeline processor will switch to"},{"line_number":292,"context_line":"the next tenant or pipeline even if there are outstanding events."},{"line_number":293,"context_line":""},{"line_number":294,"context_line":"In order to reduce stalls when doing reconfigurations or tenant reconfigurations"},{"line_number":295,"context_line":"we can run one pipeline processor in one thread and reconfigurations in a"}],"source_content_type":"text/x-rst","patch_set":18,"id":"1fa4df85_691e980d","line":292,"updated":"2020-02-25 21:30:00.000000000","message":"When we get to the point where this has been implemented we should try to remember to document the behavior. I think this has the possibility of changing current expected behavior for existing users and helping them understand why jobs are scheduled in some order will help them. At the very least it will help me field questions when they come up :)","commit_id":"5a52bac6459821dd8656b79d6d7680e317a18fa2"}]}