)]}'
{"/COMMIT_MSG":[{"author":{"_account_id":597,"name":"Pete Zaitcev","email":"zaitcev@kotori.zaitcev.us","username":"zaitcev"},"change_message_id":"4802bbeb5e9c1eca9b9f91ce22ccafe71daad9e9","unresolved":false,"context_lines":[{"line_number":18,"context_line":"Let\u0027s at least clean up the proxy-server connections and greenthreads"},{"line_number":19,"context_line":"when they\u0027re no longer necessary; depending on whether it\u0027s started"},{"line_number":20,"context_line":"processing the requests or not, this may also provide some relief to"},{"line_number":21,"context_line":"the overwhelmed container-server."},{"line_number":22,"context_line":""},{"line_number":23,"context_line":"Here\u0027s my testing setup:"},{"line_number":24,"context_line":""}],"source_content_type":"text/x-gerrit-commit-message","patch_set":2,"id":"1fa4df85_d01fcd58","line":21,"updated":"2020-03-03 17:22:40.000000000","message":"The logic of this statement escapes me. Container server dragging proxy down is well explained above, but how would this help the overloaded container server? It\u0027s not like the number of container requests changes.","commit_id":"280e4a90b53622e1edbf55a17c03a142e88df2a5"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"1e5eea55d9bc92198139bfe117e7ed6e54fd724c","unresolved":false,"context_lines":[{"line_number":18,"context_line":"Let\u0027s at least clean up the proxy-server connections and greenthreads"},{"line_number":19,"context_line":"when they\u0027re no longer necessary; depending on whether it\u0027s started"},{"line_number":20,"context_line":"processing the requests or not, this may also provide some relief to"},{"line_number":21,"context_line":"the overwhelmed container-server."},{"line_number":22,"context_line":""},{"line_number":23,"context_line":"Here\u0027s my testing setup:"},{"line_number":24,"context_line":""}],"source_content_type":"text/x-gerrit-commit-message","patch_set":2,"id":"1fa4df85_d49b4fc6","line":21,"updated":"2020-03-03 16:12:51.000000000","message":"To me that sounds optimistic, I don\u0027t think the container server would notice the proxy has hung up until it goes to write the response back onto the socket...","commit_id":"280e4a90b53622e1edbf55a17c03a142e88df2a5"}],"swift/proxy/controllers/base.py":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"41dbe97a3cd341796b623b0c79c0cef8844411ad","unresolved":false,"context_lines":[{"line_number":1280,"context_line":"                conn.close()"},{"line_number":1281,"context_line":"            raise"},{"line_number":1282,"context_line":"        except (Exception, Timeout):"},{"line_number":1283,"context_line":"            self.app.exception_occurred("},{"line_number":1284,"context_line":"                node, self.server_type,"},{"line_number":1285,"context_line":"                _(\u0027Trying to %(method)s %(path)s\u0027) %"},{"line_number":1286,"context_line":"                {\u0027method\u0027: self.req_method, \u0027path\u0027: self.req_path})"}],"source_content_type":"text/x-python","patch_set":1,"id":"3fa7e38b_36d432ea","line":1283,"updated":"2020-02-06 20:26:13.000000000","message":"Should we just extend this to catch GreenletExit?  I wonder why we don\u0027t have to close/cancel the connection here - does letting it fall out of scope not \"work\"?  We certainly use close_swift_conn a lot below...\n\nI\u0027d guess mostly this is probably where the \"leave it running\" strategy currently normally falls into.  In that case the error limiting is probably healthy.  But I could also imagine some cases where it\u0027s slow but \"works\" and we still process the \"update timing\" code.","commit_id":"07101b18ffaccab9ccd91f12d17300bc785d5d46"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"42f23e0d7ca5deb184884211b2f9cb6b7ad5d8b2","unresolved":false,"context_lines":[{"line_number":1280,"context_line":"                conn.close()"},{"line_number":1281,"context_line":"            raise"},{"line_number":1282,"context_line":"        except (Exception, Timeout):"},{"line_number":1283,"context_line":"            self.app.exception_occurred("},{"line_number":1284,"context_line":"                node, self.server_type,"},{"line_number":1285,"context_line":"                _(\u0027Trying to %(method)s %(path)s\u0027) %"},{"line_number":1286,"context_line":"                {\u0027method\u0027: self.req_method, \u0027path\u0027: self.req_path})"}],"source_content_type":"text/x-python","patch_set":1,"id":"3fa7e38b_5ccf8f53","line":1283,"in_reply_to":"3fa7e38b_36d432ea","updated":"2020-02-06 22:15:37.000000000","message":"So here\u0027s my testing setup:\n\n* Create a new, empty container.\n* Stop all container-servers, then restart just *one* of them.\n* Run nc on one or more of the stopped-container-server ports; something like\n\n ncat -lk --max-conns 1024 localhost 6031 \u003e/dev/null \u0026\n\nIn one window, watch the open connections like\n\n watch -n .1 \u0027lsof -i TCP | grep 6031\u0027\n\nIn another, restart proxies (to make sure nothing\u0027s error-limited) and issue a bunch of requests like\n\n swift-init proxy restart \u0026\u0026 ( for i in {1..100}; do curl http://saio:8090/v1/AUTH_test/c \u003e\u00262 \u0026 done; wait ) 2\u003e /dev/null\n\n(You can repeat this last step a few times, see whether your observation was a one-off or repeatable thing.)\n\nOnce everything joins up again, I\u0027d often see a bunch (~50) of connections sitting in ESTABLISHED without the explicit close. Several minutes later, they\u0027re still hanging around. Eventually they get cleaned up, but only after I issue some reasonably large number of other requests. With the explicit close, they\u0027d clean up immediately. (Though maybe I should be using close_swift_conn? Good pointer!)\n\n---\n\nYou\u0027re spot-on that this is where things were getting logged that lead me down this road.\n\nIDK that I want to bring in error limiting when I don\u0027t know which request this is -- is this the first, sucky request or is this the second request and the first took like concurrency_timeout + 0.1 to respond?","commit_id":"07101b18ffaccab9ccd91f12d17300bc785d5d46"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"a58e0bab9749545a1fff0a70262de24ad50b04f8","unresolved":false,"context_lines":[{"line_number":1280,"context_line":"                conn.close()"},{"line_number":1281,"context_line":"            raise"},{"line_number":1282,"context_line":"        except (Exception, Timeout):"},{"line_number":1283,"context_line":"            self.app.exception_occurred("},{"line_number":1284,"context_line":"                node, self.server_type,"},{"line_number":1285,"context_line":"                _(\u0027Trying to %(method)s %(path)s\u0027) %"},{"line_number":1286,"context_line":"                {\u0027method\u0027: self.req_method, \u0027path\u0027: self.req_path})"}],"source_content_type":"text/x-python","patch_set":1,"id":"3fa7e38b_37442c48","line":1283,"in_reply_to":"3fa7e38b_5ccf8f53","updated":"2020-02-06 22:30:15.000000000","message":"Nope; needs a response, not a connection:\n\n \u0027BufferedHTTPConnection\u0027 object has no attribute \u0027nuke_from_orbit\u0027\n\n(which was harder to figure out than it should have been because of the exception-swallowing in close_swift_conn :-/ )","commit_id":"07101b18ffaccab9ccd91f12d17300bc785d5d46"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"1e5eea55d9bc92198139bfe117e7ed6e54fd724c","unresolved":false,"context_lines":[{"line_number":1275,"context_line":"                # See NOTE: swift_conn at top of file about this."},{"line_number":1276,"context_line":"                possible_source.swift_conn \u003d conn"},{"line_number":1277,"context_line":"        except GreenletExit:"},{"line_number":1278,"context_line":"            # Make sure socket gets cleaned up"},{"line_number":1279,"context_line":"            if conn:"},{"line_number":1280,"context_line":"                conn.close()"},{"line_number":1281,"context_line":"            raise"}],"source_content_type":"text/x-python","patch_set":2,"id":"1fa4df85_f5151b9d","line":1278,"updated":"2020-03-03 16:12:51.000000000","message":"it appears to me that we can\u0027t log from within this block\n\nmaybe worse (or related) any Exceptions I add here vanish - even if I rebase on https://review.opendev.org/#/c/709323/","commit_id":"280e4a90b53622e1edbf55a17c03a142e88df2a5"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"1e5eea55d9bc92198139bfe117e7ed6e54fd724c","unresolved":false,"context_lines":[{"line_number":1283,"context_line":"            self.app.exception_occurred("},{"line_number":1284,"context_line":"                node, self.server_type,"},{"line_number":1285,"context_line":"                _(\u0027Trying to %(method)s %(path)s\u0027) %"},{"line_number":1286,"context_line":"                {\u0027method\u0027: self.req_method, \u0027path\u0027: self.req_path})"},{"line_number":1287,"context_line":"            return False"},{"line_number":1288,"context_line":""},{"line_number":1289,"context_line":"        src_headers \u003d dict("}],"source_content_type":"text/x-python","patch_set":2,"id":"1fa4df85_907f1514","line":1286,"updated":"2020-03-03 16:12:51.000000000","message":"I\u0027m uncomfortable loosing this error limiting, especially in the the connection timeout case which is not long to wait for what would seems some reasonably useful benifit.\n\nI mean... we\u0027ve already made the request let\u0027s get what we can out of it?","commit_id":"280e4a90b53622e1edbf55a17c03a142e88df2a5"}]}