)]}'
{"/PATCHSET_LEVEL":[{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"c16da6ce529b05422bddc27b52ab8cc53ce1a745","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"6a6b583d_f8fc72a5","updated":"2021-10-25 23:18:19.000000000","message":"https://zuul.opendev.org/t/zuul/build/96d222c7f0554b01a783ab9005459473/log/job-output.txt#11047 shows us governing during a failed test. Logging doesn\u0027t report every failure after that but I suspect at least part of the issue is we are evaluating things too slowly? That said system load may also be a slow to update value?","commit_id":"58ecc9c944f4280d74a8f5c13448a7366666bb99"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"fd44758534486881503ca95d55690570e0fc5826","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":2,"id":"e7c5d4ad_9f8c5493","updated":"2021-10-26 14:50:07.000000000","message":"recheck gather more data","commit_id":"292036213348cca80e64cd905ac3942054e31a61"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"6ff5898fe373395316ec4860d918dfef17396b1e","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"46c1e819_87547de2","updated":"2021-10-28 16:16:38.000000000","message":"That last failure was due to high system load: https://zuul.opendev.org/t/zuul/build/1549250827ff4d9ebc149b10e8ac36eb/log/job-output.txt#10910 and we never seemed to drop off low enough after that to reregister. I suspect that polling more frequently isn\u0027t fixing this.\n\nWe might need to disable the sensors in testing instead? But that causes sensor specific tests to fail which means we need to do a bit more work in that space.","commit_id":"c37cbec140c99f733259a1e448c90da38108e8ea"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"2bd3198632307f22ea69f1c280971ae4632b3ccb","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"53a6a3de_d75d4779","updated":"2021-10-29 21:31:52.000000000","message":"There as another failure due to this problem [1].\n\nJudging from the discussion here I think we have the following possibilities:\n1. disable sensors in test\n2. tweak sensor parameters in test (probably increase executor.load_multiplier in the zuul.conf test fixtures)\n3. increase all the timeouts where we wait for jobs\n4. reduce concurrency\n5. larger nodes\n\nSo I guess we decide against #1 it seems to me that probably the easiest thing would be to try out #2.\n\n[1] https://11a58606918618718cd5-a21c5a2f7ec31a719791313ddc031133.ssl.cf5.rackcdn.com/815764/4/gate/zuul-tox-py38/a1e5f86/testr_results.html","commit_id":"c37cbec140c99f733259a1e448c90da38108e8ea"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"35ca4a9b226b2e30d36f0ae39f3ef6da530e8221","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"51b1b1ec_46425ae0","updated":"2021-10-26 20:57:50.000000000","message":"recheck","commit_id":"c37cbec140c99f733259a1e448c90da38108e8ea"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"d58d73a347fc19386a231b195569e8279561b64f","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"da6b8f0e_93e3b173","updated":"2021-10-28 14:11:26.000000000","message":"recheck gather more data","commit_id":"c37cbec140c99f733259a1e448c90da38108e8ea"}],"tests/base.py":[{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"bf72a4af045a43a9c46e7a47642080b5d6183a50","unresolved":false,"context_lines":[{"line_number":3569,"context_line":"        self.job_builds \u003d {}"},{"line_number":3570,"context_line":""},{"line_number":3571,"context_line":"    def run_governor(self):"},{"line_number":3572,"context_line":"        while not self.governor_stop_event.wait(1):"},{"line_number":3573,"context_line":"            try:"},{"line_number":3574,"context_line":"                self.manageLoad()"},{"line_number":3575,"context_line":"            except Exception:"}],"source_content_type":"text/x-python","patch_set":1,"id":"a8acd5ef_e5e1fdf8","line":3572,"updated":"2021-10-25 23:17:15.000000000","message":"Could we just make the wait a class constant that we override in tests, that way we don\u0027t have to override the whole method (which makes it untested)?","commit_id":"58ecc9c944f4280d74a8f5c13448a7366666bb99"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"dd7cf81501460b3d0f725c75d699252e3439bfb0","unresolved":false,"context_lines":[{"line_number":3569,"context_line":"        self.job_builds \u003d {}"},{"line_number":3570,"context_line":""},{"line_number":3571,"context_line":"    def run_governor(self):"},{"line_number":3572,"context_line":"        while not self.governor_stop_event.wait(1):"},{"line_number":3573,"context_line":"            try:"},{"line_number":3574,"context_line":"                self.manageLoad()"},{"line_number":3575,"context_line":"            except Exception:"}],"source_content_type":"text/x-python","patch_set":1,"id":"bd3910a5_0054c385","line":3572,"in_reply_to":"a8acd5ef_e5e1fdf8","updated":"2021-10-25 23:24:15.000000000","message":"Yes, we could do that. I actually half expect this change won\u0027t be sufficient which is why I also pushed the followup. But I\u0027ll update this change in case it does become sufficient and we want to merge it as is.","commit_id":"58ecc9c944f4280d74a8f5c13448a7366666bb99"}]}