)]}'
{"swift/obj/reconstructor.py":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"9fd01bf35663fc67de2e172959297050ac1a32af","unresolved":false,"context_lines":[{"line_number":958,"context_line":"                    self.run_pool.spawn(self.process_job, job)"},{"line_number":959,"context_line":"            self.run_pool.waitall()"},{"line_number":960,"context_line":"        except Exception:"},{"line_number":961,"context_line":"            self.logger.exception(_(\"Exception in top-level \""},{"line_number":962,"context_line":"                                    \"reconstruction loop\"))"},{"line_number":963,"context_line":"            self.kill_coros()"},{"line_number":964,"context_line":"        finally:"}],"source_content_type":"text/x-python","patch_set":6,"id":"5f201791_b4a662c0","line":961,"updated":"2017-06-28 05:37:04.000000000","message":"worth it just to fix this missing space!","commit_id":"6f85d635b1ba7ecacf07df472140d303dfa0cb37"}],"swift/obj/replicator.py":[{"author":{"_account_id":6968,"name":"Christian Schwede","email":"cschwede@nvidia.com","username":"cschwede"},"change_message_id":"c9b8890198f2c2e0da594776a97b0b05d9bf261e","unresolved":false,"context_lines":[{"line_number":757,"context_line":"                    if job[\u0027delete\u0027]:"},{"line_number":758,"context_line":"                        self.run_pool.spawn(self.update_deleted, job)"},{"line_number":759,"context_line":"                    else:"},{"line_number":760,"context_line":"                        self.run_pool.spawn(self.update, job)"},{"line_number":761,"context_line":"            current_nodes \u003d None"},{"line_number":762,"context_line":"            with Timeout(self.lockup_timeout):"},{"line_number":763,"context_line":"                self.run_pool.waitall()"}],"source_content_type":"text/x-python","patch_set":2,"id":"1a430d35_edd1cace","line":760,"updated":"2017-02-06 13:09:55.000000000","message":"The timeout is likely not large enough. spawn() blocks if the number of jobs is larger than concurrency. This could result in the following scenario with a concurrency of 2 and 3 jobs:\n\nt \u003d 0: job 1 \u0026 2 will spawn() and run; job 3 should be spawned, but due to 1\u00262 running will block. Time until timeout starts to decrease\nt \u003d lockup_timeout - 1sec: 1\u00262 just finish, no longer blocking the spawn of 3\nt \u003d lockup_timeout: job 3 gets killed, but just run for one second, not for lockup_timeout\n\nI think the Timeout should be 2*lockup_timeout.\n\nThe waitall below runs fine with 1*lockup_timeout, because all jobs are in progress then and each has a grace period of lockup_timeout.","commit_id":"99a645b4ea22ab2da1f5467e33410b52644de5e0"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"63751ac4194fa6fb720bdb5d493a3f283ff499ac","unresolved":false,"context_lines":[{"line_number":757,"context_line":"                    if job[\u0027delete\u0027]:"},{"line_number":758,"context_line":"                        self.run_pool.spawn(self.update_deleted, job)"},{"line_number":759,"context_line":"                    else:"},{"line_number":760,"context_line":"                        self.run_pool.spawn(self.update, job)"},{"line_number":761,"context_line":"            current_nodes \u003d None"},{"line_number":762,"context_line":"            with Timeout(self.lockup_timeout):"},{"line_number":763,"context_line":"                self.run_pool.waitall()"}],"source_content_type":"text/x-python","patch_set":2,"id":"1a430d35_ff59595b","line":760,"in_reply_to":"1a430d35_edd1cace","updated":"2017-02-06 21:31:03.000000000","message":"No, with the timeout inside the loop, each spawn gets it\u0027s own timeout. So\n\n* t0 -- enter timeout, job 1 spawns, leave timeout\n* t1 -- enter timeout, job 2 spawns, leave timeout\n* t2 -- enter timeout, we block waiting for job 3 to spawn\n\n(Noting that t0, t1, and t2 may all be within some fairly small ε depending on whether we do any IO -- probably logging -- on the main thread.)\n\nThen either the timeout pops (at t2 + lockup_timeout) or one of the running jobs completes, the spawn succeeds, and we come down to the waitall.\n\n(Side note: if we were CPU-bound, I could see an argument for making the waitall timeout something like lockup_timeout * concurrency, but since we expect to be IO-bound, just lockup_timeout seems reasonable.)\n\nIf you\u0027re eying the test failures, those are because of a race between the Timeout here and the general lockup_detector.\n\nMakes me wonder: if we get the lockup_detector working as intended, maybe we wouldn\u0027t even need the Timeouts? I\u0027ll go introduce an abort flag so we don\u0027t keep spawning jobs when lockup_detector trips, and we\u0027ll see what that looks like.","commit_id":"99a645b4ea22ab2da1f5467e33410b52644de5e0"}]}
