)]}'
{"/COMMIT_MSG":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8615cea0b399558f758e01a14102aaf694569818","unresolved":true,"context_lines":[{"line_number":7,"context_line":"Ignore 404s from handoffs when choosing response code"},{"line_number":8,"context_line":""},{"line_number":9,"context_line":"Previously, when handling an object POST, if the proxy got a 202 and"},{"line_number":10,"context_line":"two Timeouts from the 3 primary backend object servers, it then went"},{"line_number":11,"context_line":"to two handoffs which return 404. The proxy would consider this to be"},{"line_number":12,"context_line":"a quorum of 404s and return the client a 404 response. This is"},{"line_number":13,"context_line":"alarming for the client."}],"source_content_type":"text/x-gerrit-commit-message","patch_set":6,"id":"ea9e6e18_b1299892","line":10,"updated":"2024-08-19 18:36:15.000000000","message":"it would be clarifying to me if this was described as a \"ConnectionTimeout\" specifically since otherwise POST/DELETE don\u0027t normally make \"retry\" connections to handoffs on error/timeout from primary requests.","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"9ac4ae475d0059e7476c2aeb7f617bb4662129b8","unresolved":false,"context_lines":[{"line_number":7,"context_line":"Ignore 404s from handoffs when choosing response code"},{"line_number":8,"context_line":""},{"line_number":9,"context_line":"Previously, when handling an object POST, if the proxy got a 202 and"},{"line_number":10,"context_line":"two Timeouts from the 3 primary backend object servers, it then went"},{"line_number":11,"context_line":"to two handoffs which return 404. The proxy would consider this to be"},{"line_number":12,"context_line":"a quorum of 404s and return the client a 404 response. This is"},{"line_number":13,"context_line":"alarming for the client."}],"source_content_type":"text/x-gerrit-commit-message","patch_set":6,"id":"266954ec_7a3a902b","line":10,"in_reply_to":"ea9e6e18_b1299892","updated":"2024-08-21 15:27:56.000000000","message":"Done","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"}],"/PATCHSET_LEVEL":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"2e3b63a255f0b9baf268beac3c76cebd9841efee","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"baa2c597_47bd2190","updated":"2024-08-12 17:25:07.000000000","message":"\"stop treating 404s from handoffs as authoratative\" seems to be in the vien of similar bugs we\u0027ve fixed.  503s instead of 404s in the face of ambiguity seems to be inline with previous fixes.\n\nWe can probably do better if we can dig into timestamps; but that\u0027s pretty niche - this is probably better than master already.","commit_id":"e05a74ccdc4d145959c33d757860ada0a89e4667"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"0312169bb5ffc047ad54a4953bcd2b5a2d1ade6a","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"3a174287_9dfac050","updated":"2024-08-12 13:17:04.000000000","message":"In light of test_DELETE_found_not_found_and_fail, I\u0027m not sure if the handoff 404 should be ignored for a DELETE?","commit_id":"e05a74ccdc4d145959c33d757860ada0a89e4667"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"63a07daec2d5e7a70e0521d051f987785f4845fd","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"4100f115_782bc6f7","updated":"2024-08-12 16:52:51.000000000","message":"Now I want to write a (maybe controversial?) patch to return 204s on DELETE as long as we got at least one 204 and at least two 204/404 from backends. Maybe even go a step further and drop the \"at least one 204\" condition.","commit_id":"e05a74ccdc4d145959c33d757860ada0a89e4667"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"4cf16b08bbf16eca681f18f9b73d3a9af00ba3f0","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"1548e4e0_534984f4","in_reply_to":"baa2c597_47bd2190","updated":"2024-08-13 16:01:03.000000000","message":"TIL: object server does not return x-backend-timestamp with 202 or 404 response to a POST 😞\n\nWe can fix that of course. But then the proxy currently determines the usefulness of concurrent backend POST requests independently, so we\u0027ll need some relatively heavy lifting to *compare* the 202 and 404 timestamps for precedence before deciding to keep a 404.\n\nWhich we can do, but I think I\u0027d like to take that on as a follow-up so that we can ship a fix for the end-user pain point.","commit_id":"e05a74ccdc4d145959c33d757860ada0a89e4667"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"c45e0ea3f6d2219ddb1b6e7bd4364868daaa052c","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":4,"id":"094c08ca_49214b04","updated":"2024-08-13 16:01:49.000000000","message":"added a follow up to include x-backend-timestamp to POST 404s https://review.opendev.org/c/openstack/swift/+/926220","commit_id":"9e9585e714f5642ad680eff0215656a092823bf5"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8615cea0b399558f758e01a14102aaf694569818","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":6,"id":"fa807bc2_296be34c","updated":"2024-08-19 18:36:15.000000000","message":"I\u0027m a little nervous about the 204 DELETE behavior changing; the overrides stuff sure looks like it was on purpose and Tim has already been trying to make the point that a 404 response from a handoff will write down a tombstone anyway so I worry maybe this change should *really* just try and focus on fixing the bug with POST\n\n926569: wip: keep the 204 on DELETE | https://review.opendev.org/c/openstack/swift/+/926569\n\nIt\u0027s annoying that we can\u0027t easily demonstrate the bug with POST on a vsaio because we don\u0027t have enough devices; but I think it\u0027s going to be a pretty serious endeavor to rework the existing probe suite and vsaio docs to have 8 replicated devices with at least 2 handoffs for returning 404s when all primaries are down.  To get *started* we could invest in trying to develop a \"FlexibleReplProbeTest\" that will pass on *either* a 4 device or 8 device configured vsaio?  And then just get it working on *this one test* to start?\n\nI still need to consider the implicatoins of making extra requests when we get 404s from handoffs and I haven\u0027t worked through how EC reacts to all this.","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"9ac4ae475d0059e7476c2aeb7f617bb4662129b8","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":6,"id":"7faf9a9a_3c430a95","updated":"2024-08-21 15:27:56.000000000","message":"OK, so that last patchset was below par! But, the more I work with these tests the more I get confused/annoyed :( \n\noff-topic but noteworthy: set_http_connect does not check for unexpected requests in the way that mocked_http_conn does, so it turns out there\u0027s a bunch of passing tests that actually run out of mock responses. IDK yet if they\u0027d fail if they had the correct response set, will try to pin them down in a follow-up.\n\nOn the topic of continuing to try handoffs after a 404...I think it is sensible to break on the first 404, and the next patchset will. But I also think that it\u0027s possible for a .data file to be on the *next* handoff that isn\u0027t tried, and so the POST wouldn\u0027t find it (when a HEAD would). But it\u0027s less likely, and we\u0027ll now return a 503 so the client should retry, and that\u0027s probably better than iterating over all the handoffs every time.","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"a9e71c469baeac283781f187b1f83910fd3414b5","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":7,"id":"c36ec594_70008e01","updated":"2024-08-21 15:29:23.000000000","message":"I have moved the unit tests to a pre-patch https://review.opendev.org/c/openstack/swift/+/926805","commit_id":"cfd6a841ce4b82146feb0d37291c90f6b52e3d50"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"4cbc6e115a0dfb81a85189e04e2957e0a0164ca3","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":8,"id":"167895ac_1dbeaa97","updated":"2024-08-23 17:48:10.000000000","message":"recheck\n\nprobe test timed out","commit_id":"d0f6a0ce5624e271bfc3e2cca7d328a18df546f4"}],"swift/proxy/controllers/base.py":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"2e3b63a255f0b9baf268beac3c76cebd9841efee","unresolved":true,"context_lines":[{"line_number":2050,"context_line":"            return False"},{"line_number":2051,"context_line":"        if is_informational(resp.status):"},{"line_number":2052,"context_line":"            return False"},{"line_number":2053,"context_line":"        if \u0027handoff_index\u0027 in node:"},{"line_number":2054,"context_line":"            return resp.status !\u003d 404"},{"line_number":2055,"context_line":"        return True"},{"line_number":2056,"context_line":""}],"source_content_type":"text/x-python","patch_set":3,"id":"5bde454b_5ab16e4a","line":2053,"updated":"2024-08-12 17:25:07.000000000","message":"can we consider if the handoff has a x-backend-timestamp (i.e. *is* a tombstone) before we throw out the 404?\n\nThe reason we talk to handoffs is in-case the primaries are offline we hope that a POST made at t2 would talk to the same nodes as a DELEETE made at t1.","commit_id":"e05a74ccdc4d145959c33d757860ada0a89e4667"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"aea00d48a75e8f188add5671f69cafa7b1925be2","unresolved":true,"context_lines":[{"line_number":2050,"context_line":"            return False"},{"line_number":2051,"context_line":"        if is_informational(resp.status):"},{"line_number":2052,"context_line":"            return False"},{"line_number":2053,"context_line":"        if \u0027handoff_index\u0027 in node:"},{"line_number":2054,"context_line":"            return resp.status !\u003d 404"},{"line_number":2055,"context_line":"        return True"},{"line_number":2056,"context_line":""}],"source_content_type":"text/x-python","patch_set":3,"id":"63b10e3c_c54cbfe8","line":2053,"in_reply_to":"5bde454b_5ab16e4a","updated":"2024-08-13 03:33:30.000000000","message":"oh that\u0027s a cool idea Clay!","commit_id":"e05a74ccdc4d145959c33d757860ada0a89e4667"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"4cf16b08bbf16eca681f18f9b73d3a9af00ba3f0","unresolved":true,"context_lines":[{"line_number":2050,"context_line":"            return False"},{"line_number":2051,"context_line":"        if is_informational(resp.status):"},{"line_number":2052,"context_line":"            return False"},{"line_number":2053,"context_line":"        if \u0027handoff_index\u0027 in node:"},{"line_number":2054,"context_line":"            return resp.status !\u003d 404"},{"line_number":2055,"context_line":"        return True"},{"line_number":2056,"context_line":""}],"source_content_type":"text/x-python","patch_set":3,"id":"a505efa4_cca5844c","line":2053,"in_reply_to":"63b10e3c_c54cbfe8","updated":"2024-08-13 16:01:03.000000000","message":"we can if we also make the object server return an x-backend-timestamp with  404 response to a POST.\n\nWhat we don\u0027t have so easily here is access to any timestamp that a 202 response might have returned. So we\u0027d have to do any comparison of timestamps later in compute_quorum_response or w/e. And we\u0027d also need to have the object-server give us an x-backend-timestamp with the 202 response.","commit_id":"e05a74ccdc4d145959c33d757860ada0a89e4667"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8615cea0b399558f758e01a14102aaf694569818","unresolved":false,"context_lines":[{"line_number":2050,"context_line":"            return False"},{"line_number":2051,"context_line":"        if is_informational(resp.status):"},{"line_number":2052,"context_line":"            return False"},{"line_number":2053,"context_line":"        if \u0027handoff_index\u0027 in node:"},{"line_number":2054,"context_line":"            return resp.status !\u003d 404"},{"line_number":2055,"context_line":"        return True"},{"line_number":2056,"context_line":""}],"source_content_type":"text/x-python","patch_set":3,"id":"24e73d51_dbd686b8","line":2053,"in_reply_to":"a505efa4_cca5844c","updated":"2024-08-19 18:36:15.000000000","message":"Acknowledged","commit_id":"e05a74ccdc4d145959c33d757860ada0a89e4667"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8615cea0b399558f758e01a14102aaf694569818","unresolved":true,"context_lines":[{"line_number":2044,"context_line":"        return info"},{"line_number":2045,"context_line":""},{"line_number":2046,"context_line":"    def _is_useful_response(self, node, method, path, resp):"},{"line_number":2047,"context_line":"        assert (\u0027index\u0027 in node or \u0027handoff_index\u0027 in node)"},{"line_number":2048,"context_line":"        if not self.app.check_response(node, self.server_type, resp,"},{"line_number":2049,"context_line":"                                       method, path):"},{"line_number":2050,"context_line":"            return False"}],"source_content_type":"text/x-python","patch_set":6,"id":"f055f24c_d9852731","line":2047,"updated":"2024-08-19 18:36:15.000000000","message":"I could imagine this possibly useful during testing/development; but probably not needed or typically optimized out (-o) when running proxies in production:\n\nhttps://docs.python.org/3/reference/simple_stmts.html#the-assert-statement","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"9ac4ae475d0059e7476c2aeb7f617bb4662129b8","unresolved":false,"context_lines":[{"line_number":2044,"context_line":"        return info"},{"line_number":2045,"context_line":""},{"line_number":2046,"context_line":"    def _is_useful_response(self, node, method, path, resp):"},{"line_number":2047,"context_line":"        assert (\u0027index\u0027 in node or \u0027handoff_index\u0027 in node)"},{"line_number":2048,"context_line":"        if not self.app.check_response(node, self.server_type, resp,"},{"line_number":2049,"context_line":"                                       method, path):"},{"line_number":2050,"context_line":"            return False"}],"source_content_type":"text/x-python","patch_set":6,"id":"0df0bcfe_aaef23bc","line":2047,"in_reply_to":"f055f24c_d9852731","updated":"2024-08-21 15:27:56.000000000","message":"Done","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8615cea0b399558f758e01a14102aaf694569818","unresolved":true,"context_lines":[{"line_number":2046,"context_line":"    def _is_useful_response(self, node, method, path, resp):"},{"line_number":2047,"context_line":"        assert (\u0027index\u0027 in node or \u0027handoff_index\u0027 in node)"},{"line_number":2048,"context_line":"        if not self.app.check_response(node, self.server_type, resp,"},{"line_number":2049,"context_line":"                                       method, path):"},{"line_number":2050,"context_line":"            return False"},{"line_number":2051,"context_line":"        if is_informational(resp.status):"},{"line_number":2052,"context_line":"            return False"}],"source_content_type":"text/x-python","patch_set":6,"id":"b704d5f8_cc6f4203","line":2049,"updated":"2024-08-19 18:36:15.000000000","message":"this was an existing call that was relocated as part of the refactor; it\u0027s doing error limiting for 507/5XX","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"9ac4ae475d0059e7476c2aeb7f617bb4662129b8","unresolved":false,"context_lines":[{"line_number":2046,"context_line":"    def _is_useful_response(self, node, method, path, resp):"},{"line_number":2047,"context_line":"        assert (\u0027index\u0027 in node or \u0027handoff_index\u0027 in node)"},{"line_number":2048,"context_line":"        if not self.app.check_response(node, self.server_type, resp,"},{"line_number":2049,"context_line":"                                       method, path):"},{"line_number":2050,"context_line":"            return False"},{"line_number":2051,"context_line":"        if is_informational(resp.status):"},{"line_number":2052,"context_line":"            return False"}],"source_content_type":"text/x-python","patch_set":6,"id":"80f7f395_6f8d4f76","line":2049,"in_reply_to":"b704d5f8_cc6f4203","updated":"2024-08-21 15:27:56.000000000","message":"Acknowledged","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8615cea0b399558f758e01a14102aaf694569818","unresolved":true,"context_lines":[{"line_number":2052,"context_line":"            return False"},{"line_number":2053,"context_line":"        if (\u0027handoff_index\u0027 in node"},{"line_number":2054,"context_line":"                and resp.status \u003d\u003d 404"},{"line_number":2055,"context_line":"                and resp.headers.get(\u0027x-backend-timestamp\u0027) is None):"},{"line_number":2056,"context_line":"            # 404s from handoffs used to be treated as authoritative, but now"},{"line_number":2057,"context_line":"            # they are only considered useful when they have an"},{"line_number":2058,"context_line":"            # x-backend-timestamp header that indicates there\u0027s a tombstone"}],"source_content_type":"text/x-python","patch_set":6,"id":"c118fbfd_ba7c63c5","line":2055,"updated":"2024-08-19 18:36:15.000000000","message":"oh wow, in tests resp.headers is very different from the resp.getheaders() we return from `_make_request`\n\n```\n(Pdb) !resp.headers\n{}\n(Pdb) !resp.getheaders()\ndict_items([(\u0027Content-Length\u0027, \u00270\u0027), (\u0027Content-Type\u0027, \u0027x-application/test\u0027), (\u0027X-Timestamp\u0027, \u00271\u0027), (\u0027X-Backend-Timestamp\u0027, \u00271\u0027), (\u0027Last-Modified\u0027, \u00271\u0027), (\u0027X-Object-Meta-Test\u0027, \u0027testing\u0027), (\u0027X-Delete-At\u0027, \u00279876543210\u0027), (\u0027Etag\u0027, \u0027\"d41d8cd98f00b204e9800998ecf8427e\"\u0027), (\u0027X-Works\u0027, \u0027yes\u0027), (\u0027X-Account-Container-Count\u0027, \u002712345\u0027), (\u0027X-Container-Timestamp\u0027, \u00271\u0027)])\n```","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"9ac4ae475d0059e7476c2aeb7f617bb4662129b8","unresolved":false,"context_lines":[{"line_number":2052,"context_line":"            return False"},{"line_number":2053,"context_line":"        if (\u0027handoff_index\u0027 in node"},{"line_number":2054,"context_line":"                and resp.status \u003d\u003d 404"},{"line_number":2055,"context_line":"                and resp.headers.get(\u0027x-backend-timestamp\u0027) is None):"},{"line_number":2056,"context_line":"            # 404s from handoffs used to be treated as authoritative, but now"},{"line_number":2057,"context_line":"            # they are only considered useful when they have an"},{"line_number":2058,"context_line":"            # x-backend-timestamp header that indicates there\u0027s a tombstone"}],"source_content_type":"text/x-python","patch_set":6,"id":"00695f86_b7195b4f","line":2055,"in_reply_to":"c118fbfd_ba7c63c5","updated":"2024-08-21 15:27:56.000000000","message":"Acknowledged","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8615cea0b399558f758e01a14102aaf694569818","unresolved":true,"context_lines":[{"line_number":2055,"context_line":"                and resp.headers.get(\u0027x-backend-timestamp\u0027) is None):"},{"line_number":2056,"context_line":"            # 404s from handoffs used to be treated as authoritative, but now"},{"line_number":2057,"context_line":"            # they are only considered useful when they have an"},{"line_number":2058,"context_line":"            # x-backend-timestamp header that indicates there\u0027s a tombstone"},{"line_number":2059,"context_line":"            return False"},{"line_number":2060,"context_line":"        return True"},{"line_number":2061,"context_line":""}],"source_content_type":"text/x-python","patch_set":6,"id":"6e61f978_f97f1794","line":2058,"updated":"2024-08-19 18:36:15.000000000","message":"this is the new condition and I think it makes sense; the comment is helpful - but could be phrased in the current tense:\n\n404s from handoffs that don\u0027t include an x-backend-timestamp which would indicate a tombstone are not considered useful or authoratative because they don\u0027t add any information to the response set.","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8615cea0b399558f758e01a14102aaf694569818","unresolved":true,"context_lines":[{"line_number":2084,"context_line":"        self.logger.thread_locals \u003d logger_thread_locals"},{"line_number":2085,"context_line":"        if body:"},{"line_number":2086,"context_line":"            if not isinstance(body, bytes):"},{"line_number":2087,"context_line":"                raise TypeError(\u0027body must be bytes, not %s\u0027 % type(body))"},{"line_number":2088,"context_line":"            headers[\u0027Content-Length\u0027] \u003d str(len(body))"},{"line_number":2089,"context_line":"        for node in nodes:"},{"line_number":2090,"context_line":"            try:"}],"source_content_type":"text/x-python","patch_set":6,"id":"fb2e4169_b50ea0a8","line":2087,"updated":"2024-08-19 18:36:15.000000000","message":"ok, body is definately bytes","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"9ac4ae475d0059e7476c2aeb7f617bb4662129b8","unresolved":false,"context_lines":[{"line_number":2084,"context_line":"        self.logger.thread_locals \u003d logger_thread_locals"},{"line_number":2085,"context_line":"        if body:"},{"line_number":2086,"context_line":"            if not isinstance(body, bytes):"},{"line_number":2087,"context_line":"                raise TypeError(\u0027body must be bytes, not %s\u0027 % type(body))"},{"line_number":2088,"context_line":"            headers[\u0027Content-Length\u0027] \u003d str(len(body))"},{"line_number":2089,"context_line":"        for node in nodes:"},{"line_number":2090,"context_line":"            try:"}],"source_content_type":"text/x-python","patch_set":6,"id":"ca15375d_c1f94f79","line":2087,"in_reply_to":"fb2e4169_b50ea0a8","updated":"2024-08-21 15:27:56.000000000","message":"Acknowledged","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8615cea0b399558f758e01a14102aaf694569818","unresolved":true,"context_lines":[{"line_number":2101,"context_line":"                        conn.send(body)"},{"line_number":2102,"context_line":"                with Timeout(self.app.node_timeout):"},{"line_number":2103,"context_line":"                    resp \u003d conn.getresponse()"},{"line_number":2104,"context_line":"                    if self._is_useful_response(node, method, path, resp):"},{"line_number":2105,"context_line":"                        return resp.status, resp.reason, resp.getheaders(), \\"},{"line_number":2106,"context_line":"                            resp.read()"},{"line_number":2107,"context_line":""}],"source_content_type":"text/x-python","patch_set":6,"id":"5b436800_6e6853c2","line":2104,"updated":"2024-08-19 18:36:15.000000000","message":"yikes; we don\u0027t have access to overrides in here; maybe this isn\u0027t the best place to \"throw out\" the responses?\n\nAlso; and I *totally* did not understand this - when we *don\u0027t* return here - we\u0027re actually telling this \"loop\" over \"for node in nodes\" to continue again to another handoff and make another request.  I think that makes sense on a 507 but for \"just\" a 404 from a handoff maybe we don\u0027t need to try *another* handoff?","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"3c944ceab93a981d901c1e9c9d9907c92137b876","unresolved":true,"context_lines":[{"line_number":2101,"context_line":"                        conn.send(body)"},{"line_number":2102,"context_line":"                with Timeout(self.app.node_timeout):"},{"line_number":2103,"context_line":"                    resp \u003d conn.getresponse()"},{"line_number":2104,"context_line":"                    if self._is_useful_response(node, method, path, resp):"},{"line_number":2105,"context_line":"                        return resp.status, resp.reason, resp.getheaders(), \\"},{"line_number":2106,"context_line":"                            resp.read()"},{"line_number":2107,"context_line":""}],"source_content_type":"text/x-python","patch_set":6,"id":"d004e27e_17304a04","line":2104,"in_reply_to":"5b436800_6e6853c2","updated":"2024-08-20 03:59:03.000000000","message":"I think we shouldn\u0027t return here if just one handoff node return 404 without ``x-backend-timestamp``, the loop should go on to reach other handoff nodes, since there may be only two handoff nodes with the delete tombstone.","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"cabdc6cd68e4da524d2d0f9cbd342fa670520120","unresolved":true,"context_lines":[{"line_number":2101,"context_line":"                        conn.send(body)"},{"line_number":2102,"context_line":"                with Timeout(self.app.node_timeout):"},{"line_number":2103,"context_line":"                    resp \u003d conn.getresponse()"},{"line_number":2104,"context_line":"                    if self._is_useful_response(node, method, path, resp):"},{"line_number":2105,"context_line":"                        return resp.status, resp.reason, resp.getheaders(), \\"},{"line_number":2106,"context_line":"                            resp.read()"},{"line_number":2107,"context_line":""}],"source_content_type":"text/x-python","patch_set":6,"id":"66428540_e3d014dc","line":2104,"in_reply_to":"d004e27e_17304a04","updated":"2024-08-20 12:42:19.000000000","message":"well, except some other thread/loop probably already went to the second and third handoff because the primary hit a connection timeout.  I think a continuation here is most likely to reach out to a 4th, 5th, or 6th handoff node!  Maybe reasonable for a read operatoin; but every handoff we talk to on DELETE has to write a tombstone.  That\u0027s a lot of writing down zero-byte files for an object that might not even exist...","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8615cea0b399558f758e01a14102aaf694569818","unresolved":true,"context_lines":[{"line_number":2133,"context_line":"        :param node_count: optional number of nodes to send request to."},{"line_number":2134,"context_line":"        :param node_iterator: optional node iterator."},{"line_number":2135,"context_line":"        :param body: byte string to use as the request body."},{"line_number":2136,"context_line":"                     Try to keep it small."},{"line_number":2137,"context_line":"        :returns: a swob.Response object"},{"line_number":2138,"context_line":"        \"\"\""},{"line_number":2139,"context_line":"        nodes \u003d GreenthreadSafeIterator(node_iterator or NodeIter("}],"source_content_type":"text/x-python","patch_set":6,"id":"00b566d6_998da1fb","line":2136,"updated":"2024-08-19 18:36:15.000000000","message":"this is helpful - kudos.","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"9ac4ae475d0059e7476c2aeb7f617bb4662129b8","unresolved":false,"context_lines":[{"line_number":2133,"context_line":"        :param node_count: optional number of nodes to send request to."},{"line_number":2134,"context_line":"        :param node_iterator: optional node iterator."},{"line_number":2135,"context_line":"        :param body: byte string to use as the request body."},{"line_number":2136,"context_line":"                     Try to keep it small."},{"line_number":2137,"context_line":"        :returns: a swob.Response object"},{"line_number":2138,"context_line":"        \"\"\""},{"line_number":2139,"context_line":"        nodes \u003d GreenthreadSafeIterator(node_iterator or NodeIter("}],"source_content_type":"text/x-python","patch_set":6,"id":"ccd6a0f9_0e9e74a8","line":2136,"in_reply_to":"00b566d6_998da1fb","updated":"2024-08-21 15:27:56.000000000","message":"Acknowledged","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"d831f04c5d6c54624ec1b7a5c8774a2b12674a9c","unresolved":true,"context_lines":[{"line_number":2131,"context_line":"        response \u003d []"},{"line_number":2132,"context_line":"        statuses \u003d []"},{"line_number":2133,"context_line":"        for resp in pile:"},{"line_number":2134,"context_line":"            if not resp:"},{"line_number":2135,"context_line":"                continue"},{"line_number":2136,"context_line":"            response.append(resp)"},{"line_number":2137,"context_line":"            statuses.append(resp[0])"}],"source_content_type":"text/x-python","patch_set":7,"id":"4c681df6_344f1015","side":"PARENT","line":2134,"range":{"start_line":2134,"start_character":19,"end_line":2134,"end_character":23},"updated":"2024-08-21 16:01:39.000000000","message":"I grew to dislike how this ``resp`` variable is actually a tuple","commit_id":"dec9db245d190731f652a7a85f2c0d5a02fe8a84"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"d831f04c5d6c54624ec1b7a5c8774a2b12674a9c","unresolved":true,"context_lines":[{"line_number":2145,"context_line":"        results \u003d []"},{"line_number":2146,"context_line":"        statuses \u003d []"},{"line_number":2147,"context_line":"        for resp, body, node in pile:"},{"line_number":2148,"context_line":"            if not is_useful_response(resp, node):"},{"line_number":2149,"context_line":"                continue"},{"line_number":2150,"context_line":"            results.append((resp.status, resp.reason, resp.getheaders(), body))"},{"line_number":2151,"context_line":"            statuses.append(resp.status)"}],"source_content_type":"text/x-python","patch_set":7,"id":"79079208_cb54633e","line":2148,"updated":"2024-08-21 16:01:39.000000000","message":"I think this could have stayed in _make_request, and that would have avoided repeating at line 2157, but it felt like it belonged here more...``_make_requests`` returns anything but errors, and ``make_requests`` is more nuanced.","commit_id":"cfd6a841ce4b82146feb0d37291c90f6b52e3d50"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"571a135a404e47e749579db100811a327934e674","unresolved":true,"context_lines":[{"line_number":2147,"context_line":"        for resp, body, node in pile:"},{"line_number":2148,"context_line":"            if not is_useful_response(resp, node):"},{"line_number":2149,"context_line":"                continue"},{"line_number":2150,"context_line":"            results.append((resp.status, resp.reason, resp.getheaders(), body))"},{"line_number":2151,"context_line":"            statuses.append(resp.status)"},{"line_number":2152,"context_line":"            if self.have_quorum(statuses, node_number):"},{"line_number":2153,"context_line":"                break"}],"source_content_type":"text/x-python","patch_set":7,"id":"30caf477_94a99cef","line":2150,"updated":"2024-08-21 16:50:12.000000000","message":"FWIW we introduced some encapsulating helper classes on feature/mpu that cleans this code path up a little\n\nhttps://github.com/openstack/swift/blob/feature/mpu/swift/proxy/controllers/base.py#L2159\n\nI\u0027d pull it back into master if I could be confident it wouldn\u0027t stall this bug fix","commit_id":"cfd6a841ce4b82146feb0d37291c90f6b52e3d50"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"332f493620c311a13cb66c1359308cfa829f36ef","unresolved":true,"context_lines":[{"line_number":2159,"context_line":"            results.append((resp.status, resp.reason, resp.getheaders(), body))"},{"line_number":2160,"context_line":"            statuses.append(resp.status)"},{"line_number":2161,"context_line":"        while len(results) \u003c node_number:"},{"line_number":2162,"context_line":"            results.append((HTTP_SERVICE_UNAVAILABLE, \u0027\u0027, \u0027\u0027, b\u0027\u0027))"},{"line_number":2163,"context_line":"        statuses, reasons, resp_headers, bodies \u003d zip(*results)"},{"line_number":2164,"context_line":"        return self.best_response(req, statuses, reasons, bodies,"},{"line_number":2165,"context_line":"                                  \u0027%s %s\u0027 % (self.server_type, req.method),"}],"source_content_type":"text/x-python","patch_set":8,"id":"e399ae40_55325726","line":2162,"range":{"start_line":2162,"start_character":58,"end_line":2162,"end_character":60},"updated":"2024-09-04 16:32:23.000000000","message":"Off-topic: I feel like we should be more careful about types here...","commit_id":"d0f6a0ce5624e271bfc3e2cca7d328a18df546f4"}],"test/probe/test_object_handoff.py":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8615cea0b399558f758e01a14102aaf694569818","unresolved":true,"context_lines":[{"line_number":314,"context_line":"                                      self.policy.name})"},{"line_number":315,"context_line":""},{"line_number":316,"context_line":"        # Create container/obj (goes to all three primaries)"},{"line_number":317,"context_line":"        obj \u003d \u0027object-%s\u0027 % uuid4()"},{"line_number":318,"context_line":"        client.put_object(self.url, self.token, container, obj, b\u0027VERIFY\u0027)"},{"line_number":319,"context_line":"        odata \u003d client.get_object(self.url, self.token, container, obj)[-1]"},{"line_number":320,"context_line":"        if odata !\u003d b\u0027VERIFY\u0027:"}],"source_content_type":"text/x-python","patch_set":6,"id":"6e045ba5_a492e214","line":317,"updated":"2024-08-19 18:36:15.000000000","message":"i don\u0027t really understand what this object is for.","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"9ac4ae475d0059e7476c2aeb7f617bb4662129b8","unresolved":false,"context_lines":[{"line_number":314,"context_line":"                                      self.policy.name})"},{"line_number":315,"context_line":""},{"line_number":316,"context_line":"        # Create container/obj (goes to all three primaries)"},{"line_number":317,"context_line":"        obj \u003d \u0027object-%s\u0027 % uuid4()"},{"line_number":318,"context_line":"        client.put_object(self.url, self.token, container, obj, b\u0027VERIFY\u0027)"},{"line_number":319,"context_line":"        odata \u003d client.get_object(self.url, self.token, container, obj)[-1]"},{"line_number":320,"context_line":"        if odata !\u003d b\u0027VERIFY\u0027:"}],"source_content_type":"text/x-python","patch_set":6,"id":"ce6a30b3_9b81de28","line":317,"in_reply_to":"6e045ba5_a492e214","updated":"2024-08-21 15:27:56.000000000","message":"Acknowledged","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8615cea0b399558f758e01a14102aaf694569818","unresolved":true,"context_lines":[{"line_number":322,"context_line":"                            \u0027returned: %s\u0027 % repr(odata))"},{"line_number":323,"context_line":""},{"line_number":324,"context_line":"        # Kill all primaries obj server"},{"line_number":325,"context_line":"        obj \u003d \u0027object-%s\u0027 % uuid4()"},{"line_number":326,"context_line":"        opart, onodes \u003d self.object_ring.get_nodes("},{"line_number":327,"context_line":"            self.account, container, obj)"},{"line_number":328,"context_line":"        for onode in onodes:"}],"source_content_type":"text/x-python","patch_set":6,"id":"9cfe2c85_ad782738","line":325,"updated":"2024-08-19 18:36:15.000000000","message":"ok, so this object is never uploaded.","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"9ac4ae475d0059e7476c2aeb7f617bb4662129b8","unresolved":false,"context_lines":[{"line_number":322,"context_line":"                            \u0027returned: %s\u0027 % repr(odata))"},{"line_number":323,"context_line":""},{"line_number":324,"context_line":"        # Kill all primaries obj server"},{"line_number":325,"context_line":"        obj \u003d \u0027object-%s\u0027 % uuid4()"},{"line_number":326,"context_line":"        opart, onodes \u003d self.object_ring.get_nodes("},{"line_number":327,"context_line":"            self.account, container, obj)"},{"line_number":328,"context_line":"        for onode in onodes:"}],"source_content_type":"text/x-python","patch_set":6,"id":"9d1e2d3e_f23fcf51","line":325,"in_reply_to":"9cfe2c85_ad782738","updated":"2024-08-21 15:27:56.000000000","message":"Acknowledged","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8615cea0b399558f758e01a14102aaf694569818","unresolved":true,"context_lines":[{"line_number":343,"context_line":"        except client.ClientException as err:"},{"line_number":344,"context_line":"            self.assertEqual(err.http_status, 503)"},{"line_number":345,"context_line":"        else:"},{"line_number":346,"context_line":"            self.fail(\"Expected ClientException but didn\u0027t get it\")"},{"line_number":347,"context_line":""},{"line_number":348,"context_line":"        # Restart the first container/obj primary server again"},{"line_number":349,"context_line":"        onode \u003d onodes[0]"}],"source_content_type":"text/x-python","patch_set":6,"id":"69f1d218_deaa297c","line":346,"updated":"2024-08-19 18:36:15.000000000","message":"when I revert the changes in base and pause in a debugger this \"obj\" is not in the listing or on disk; since *all* primaries are shutdown (and there\u0027s only 4 disks in a vsaio) I get:\n\n```\nAug 19 15:03:15 saio proxy-server: Object POST returning 503 for (404, 503, 503) (txn: txefef7d95f2c942d995520-0066c35eb3) (client_ip: 127.0.0.1)\n```\n\nBut I can configure my vsaio with more disks:\n\nhttps://github.com/NVIDIA/vagrant-swift-all-in-one/blob/master/localrc-template#L24\n\nThen if I update the probe test a little:\n\n```\ndiff --git a/test/probe/test_object_handoff.py b/test/probe/test_object_handoff.py\nindex a2934ed70..ec52db821 100644\n--- a/test/probe/test_object_handoff.py\n+++ b/test/probe/test_object_handoff.py\n@@ -37,6 +37,9 @@ from test.probe.common import (\n \n class TestObjectHandoff(ReplProbeTest):\n \n+    acct_cont_required_devices \u003d 8\n+    obj_required_devices \u003d 8\n+\n     def test_main(self):\n         # Create container\n         container \u003d \u0027container-%s\u0027 % uuid4()\n```\n\nI can pause the test in a debugger and manually verify the 404:\n\n```\nvagrant@saio:~$ swift -R1 post -v container-a7526ab0-e7b9-4327-84b1-a650026f8d5b object-66c23a6b-1cf8-4d23-b3c7-dd46a078eaa9\nObject POST failed: http://saio:8080/v1/AUTH_test/container-a7526ab0-e7b9-4327-84b1-a650026f8d5b/object-66c23a6b-1cf8-4d23-b3c7-dd46a078eaa9 404 Not Found  [first 60 chars of response] b\u0027\u003chtml\u003e\u003ch1\u003eNot Found\u003c/h1\u003e\u003cp\u003eThe resource could not be found.\u003c\u0027\nFailed Transaction ID: tx7d475d9c7a66424eb1692-0066c38d9b\n```\n\nthe logs show the two extra 404 responses are sufficient to return 404 to the client despite all the primaries and one of the handoffs returning Connection refused (yikes!)\n\n```\nAug 19 18:23:23 saio proxy-server: ERROR with Object server 127.0.0.4:6046/sdb4 re: Trying to POST /AUTH_test/container-a7526ab0-e7b9-4327-84b1-a650026f8d5b/object-66c23a6b-1cf8-4d23-b3c7-dd46a078eaa9: Connection refused (txn: tx7d475d9c7a66424eb1692-0066c38d9b) (client_ip: 127.0.0.1)\nAug 19 18:23:23 saio proxy-server: ERROR with Object server 127.0.0.3:6037/sdb7 re: Trying to POST /AUTH_test/container-a7526ab0-e7b9-4327-84b1-a650026f8d5b/object-66c23a6b-1cf8-4d23-b3c7-dd46a078eaa9: Connection refused (txn: tx7d475d9c7a66424eb1692-0066c38d9b) (client_ip: 127.0.0.1)\nAug 19 18:23:23 saio proxy-server: ERROR with Object server 127.0.0.2:6026/sdb2 re: Trying to POST /AUTH_test/container-a7526ab0-e7b9-4327-84b1-a650026f8d5b/object-66c23a6b-1cf8-4d23-b3c7-dd46a078eaa9: Connection refused (txn: tx7d475d9c7a66424eb1692-0066c38d9b) (client_ip: 127.0.0.1)\nAug 19 18:23:23 saio object-server-6016: 127.0.0.1 - - [19/Aug/2024:18:23:23 +0000] \"POST /sdb1/56/AUTH_test/container-a7526ab0-e7b9-4327-84b1-a650026f8d5b/object-66c23a6b-1cf8-4d23-b3c7-dd46a078eaa9\" 404 70 \"POST http://saio:8080/v1/AUTH_test/container-a7526ab0-e7b9-4327-84b1-a650026f8d5b/object-66c23a6b-1cf8-4d23-b3c7-dd46a078eaa9\" \"tx7d475d9c7a66424eb1692-0066c38d9b\" \"proxy-server 26985\" 0.0003 \"-\" 26972 0\nAug 19 18:23:23 saio object-server-6016: 127.0.0.1 - - [19/Aug/2024:18:23:23 +0000] \"POST /sdb5/56/AUTH_test/container-a7526ab0-e7b9-4327-84b1-a650026f8d5b/object-66c23a6b-1cf8-4d23-b3c7-dd46a078eaa9\" 404 70 \"POST http://saio:8080/v1/AUTH_test/container-a7526ab0-e7b9-4327-84b1-a650026f8d5b/object-66c23a6b-1cf8-4d23-b3c7-dd46a078eaa9\" \"tx7d475d9c7a66424eb1692-0066c38d9b\" \"proxy-server 26985\" 0.0003 \"-\" 26974 0\nAug 19 18:23:23 saio proxy-server: ERROR with Object server 127.0.0.2:6027/sdb6 re: Trying to POST /AUTH_test/container-a7526ab0-e7b9-4327-84b1-a650026f8d5b/object-66c23a6b-1cf8-4d23-b3c7-dd46a078eaa9: Connection refused (txn: tx7d475d9c7a66424eb1692-0066c38d9b) (client_ip: 127.0.0.1)\nAug 19 18:23:23 saio proxy-server: 127.0.0.1 127.0.0.1 19/Aug/2024/18/23/23 POST /v1/AUTH_test/container-a7526ab0-e7b9-4327-84b1-a650026f8d5b/object-66c23a6b-1cf8-4d23-b3c7-dd46a078eaa9 HTTP/1.0 404 - python-swiftclient-4.6.1.dev2 AUTH_tkee727235c... - 70 - tx7d475d9c7a66424eb1692-0066c38d9b - 0.0145 - - 1724091803.926758051 1724091803.941242218 0\n```\n\nand the test breaks as expected:\n\n```\n        except client.ClientException as err:\n\u003e           self.assertEqual(err.http_status, 503)\nE           AssertionError: 404 !\u003d 503\n\nswift/test/probe/test_object_handoff.py:349: AssertionError\n```","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"9ac4ae475d0059e7476c2aeb7f617bb4662129b8","unresolved":false,"context_lines":[{"line_number":343,"context_line":"        except client.ClientException as err:"},{"line_number":344,"context_line":"            self.assertEqual(err.http_status, 503)"},{"line_number":345,"context_line":"        else:"},{"line_number":346,"context_line":"            self.fail(\"Expected ClientException but didn\u0027t get it\")"},{"line_number":347,"context_line":""},{"line_number":348,"context_line":"        # Restart the first container/obj primary server again"},{"line_number":349,"context_line":"        onode \u003d onodes[0]"}],"source_content_type":"text/x-python","patch_set":6,"id":"261e0a46_752d00fb","line":346,"in_reply_to":"69f1d218_deaa297c","updated":"2024-08-21 15:27:56.000000000","message":"Acknowledged","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8615cea0b399558f758e01a14102aaf694569818","unresolved":true,"context_lines":[{"line_number":377,"context_line":"        except client.ClientException as err:"},{"line_number":378,"context_line":"            self.assertEqual(err.http_status, 503)"},{"line_number":379,"context_line":"        else:"},{"line_number":380,"context_line":"            self.fail(\"Expected ClientException but didn\u0027t get it\")"},{"line_number":381,"context_line":""},{"line_number":382,"context_line":""},{"line_number":383,"context_line":"class TestECObjectHandoff(ECProbeTest):"}],"source_content_type":"text/x-python","patch_set":6,"id":"aee39704_fc1d36fb","line":380,"updated":"2024-08-19 18:36:15.000000000","message":"at this point only 1.conf obj-server is running, primaries for this object in my case are 2,4,3; a tombstone eixsts for this object on 1 (handoff) and 2.\n\nagain; because of the lack of disks I see:\n\n```\nAug 19 15:08:17 saio proxy-server: Object POST returning 503 for (404, 503, 503) (txn: txc26aa1a701cd4fcd97596-0066c35fe1) (client_ip: 127.0.0.1)\n```\n\n... so both of these assertions pass on msater; I think they\u0027re reasonable but not actually demonstrating this fix very well.","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"9ac4ae475d0059e7476c2aeb7f617bb4662129b8","unresolved":true,"context_lines":[{"line_number":377,"context_line":"        except client.ClientException as err:"},{"line_number":378,"context_line":"            self.assertEqual(err.http_status, 503)"},{"line_number":379,"context_line":"        else:"},{"line_number":380,"context_line":"            self.fail(\"Expected ClientException but didn\u0027t get it\")"},{"line_number":381,"context_line":""},{"line_number":382,"context_line":""},{"line_number":383,"context_line":"class TestECObjectHandoff(ECProbeTest):"}],"source_content_type":"text/x-python","patch_set":6,"id":"a76d2e5f_5157e897","line":380,"in_reply_to":"aee39704_fc1d36fb","updated":"2024-08-21 15:27:56.000000000","message":"backed out these changes. thanks for pointing out their weakness.","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"}],"test/unit/proxy/controllers/test_obj.py":[{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"63a07daec2d5e7a70e0521d051f987785f4845fd","unresolved":true,"context_lines":[{"line_number":574,"context_line":"        self.assertEqual(len(codes), self.replicas())"},{"line_number":575,"context_line":"        with set_http_connect(*codes):"},{"line_number":576,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":577,"context_line":"        self.assertEqual(resp.status_int, 404)"},{"line_number":578,"context_line":""},{"line_number":579,"context_line":"    def test_DELETE_found_not_found_and_fail(self):"},{"line_number":580,"context_line":"        self.obj_ring.set_replicas(3)"}],"source_content_type":"text/x-python","patch_set":3,"id":"c2a3bf7f_544fda43","line":577,"updated":"2024-08-12 16:52:51.000000000","message":"This feels a little weird to me. Below you show that [404, 204, 507] \u003d\u003e 204 because of `status_overrides` -- and that makes some sense to me; DELETEs write `.ts` even on 404, so we\u0027ve written a durable number of tombstones, so 204. But then why wouldn\u0027t we use the same logic for [404, 404, 204]? We\u0027ve written `.ts` to all three now!\n\nMaybe I\u0027ve just gotten too used to S3, where you\u0027d get back a 2xx even if you\u0027d **never** written *anything* to the name.","commit_id":"e05a74ccdc4d145959c33d757860ada0a89e4667"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8615cea0b399558f758e01a14102aaf694569818","unresolved":false,"context_lines":[{"line_number":574,"context_line":"        self.assertEqual(len(codes), self.replicas())"},{"line_number":575,"context_line":"        with set_http_connect(*codes):"},{"line_number":576,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":577,"context_line":"        self.assertEqual(resp.status_int, 404)"},{"line_number":578,"context_line":""},{"line_number":579,"context_line":"    def test_DELETE_found_not_found_and_fail(self):"},{"line_number":580,"context_line":"        self.obj_ring.set_replicas(3)"}],"source_content_type":"text/x-python","patch_set":3,"id":"76e3f7d2_5af0719e","line":577,"in_reply_to":"c2a3bf7f_544fda43","updated":"2024-08-19 18:36:15.000000000","message":"I think the idea is that the handoffs might have had an older .data and the primaries probably had a .ts already\n\nof course if the primaries had only *just* come back online maybe their timestamps were older and the 204s should be authoratative; we can\u0027t know w/o checking timestamps\n\nI know that s3 does this different; and for a DELETE client libraries code can mostly treat 204 or 404 as successful; but on the CLI or when debugging with curl swift has always made the decision to return 404 when deleting an object that doesn\u0027t appear to exist - this is existing behavior and should be out of scope for this bug fix.","commit_id":"e05a74ccdc4d145959c33d757860ada0a89e4667"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"0312169bb5ffc047ad54a4953bcd2b5a2d1ade6a","unresolved":true,"context_lines":[{"line_number":590,"context_line":"        with set_http_connect(204, 507, Timeout(), 404):"},{"line_number":591,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":592,"context_line":"        # status_overrides passed to make_requests converts the 404 to a 204..."},{"line_number":593,"context_line":"        self.assertEqual(resp.status_int, 503)"},{"line_number":594,"context_line":""},{"line_number":595,"context_line":"    def test_DELETE_half_not_found_statuses(self):"},{"line_number":596,"context_line":"        self.obj_ring.set_replicas(4)"}],"source_content_type":"text/x-python","patch_set":3,"id":"05af889a_3be2b4d4","line":593,"updated":"2024-08-12 13:17:04.000000000","message":"I\u0027m not sure whether this previous patch https://review.opendev.org/c/openstack/swift/+/114120 intended this scenario to return a 204, similar to test_DELETE_found_not_found_and_fail ... but I\u0027ve not broken any existing unit test with the change, just a probe test which didn\u0027t come with the patch 114120 😕","commit_id":"e05a74ccdc4d145959c33d757860ada0a89e4667"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"63a07daec2d5e7a70e0521d051f987785f4845fd","unresolved":true,"context_lines":[{"line_number":590,"context_line":"        with set_http_connect(204, 507, Timeout(), 404):"},{"line_number":591,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":592,"context_line":"        # status_overrides passed to make_requests converts the 404 to a 204..."},{"line_number":593,"context_line":"        self.assertEqual(resp.status_int, 503)"},{"line_number":594,"context_line":""},{"line_number":595,"context_line":"    def test_DELETE_half_not_found_statuses(self):"},{"line_number":596,"context_line":"        self.obj_ring.set_replicas(4)"}],"source_content_type":"text/x-python","patch_set":3,"id":"c177c8c5_c94f32ad","line":593,"in_reply_to":"05af889a_3be2b4d4","updated":"2024-08-12 16:52:51.000000000","message":"Oh, hmm... I think the other day when I was thinking about 404s from handoffs, I definitely *wasn\u0027t* thinking about DELETEs...\n\nWe still would have written down a `.ts` on the handoff, right? I could see us justifying 204 here if we do. Otherwise, we\u0027re just pushing clients harder to treat 404 the same as 204, in which case why are we separating the statuses?\n\nIn the POST case, of course, we *wouldn\u0027t* write anything down, so 503 seems best.","commit_id":"e05a74ccdc4d145959c33d757860ada0a89e4667"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"9ac4ae475d0059e7476c2aeb7f617bb4662129b8","unresolved":true,"context_lines":[{"line_number":590,"context_line":"        with set_http_connect(204, 507, Timeout(), 404):"},{"line_number":591,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":592,"context_line":"        # status_overrides passed to make_requests converts the 404 to a 204..."},{"line_number":593,"context_line":"        self.assertEqual(resp.status_int, 503)"},{"line_number":594,"context_line":""},{"line_number":595,"context_line":"    def test_DELETE_half_not_found_statuses(self):"},{"line_number":596,"context_line":"        self.obj_ring.set_replicas(4)"}],"source_content_type":"text/x-python","patch_set":3,"id":"ab2311ed_2d08fb88","line":593,"in_reply_to":"319339e1_70bdecf3","updated":"2024-08-21 15:27:56.000000000","message":"This would have returned success with this patch too if I had thought to add the backend timestamp to the mocked 404 response :( , my bad","commit_id":"e05a74ccdc4d145959c33d757860ada0a89e4667"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8615cea0b399558f758e01a14102aaf694569818","unresolved":true,"context_lines":[{"line_number":590,"context_line":"        with set_http_connect(204, 507, Timeout(), 404):"},{"line_number":591,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":592,"context_line":"        # status_overrides passed to make_requests converts the 404 to a 204..."},{"line_number":593,"context_line":"        self.assertEqual(resp.status_int, 503)"},{"line_number":594,"context_line":""},{"line_number":595,"context_line":"    def test_DELETE_half_not_found_statuses(self):"},{"line_number":596,"context_line":"        self.obj_ring.set_replicas(4)"}],"source_content_type":"text/x-python","patch_set":3,"id":"319339e1_70bdecf3","line":593,"in_reply_to":"c177c8c5_c94f32ad","updated":"2024-08-19 18:36:15.000000000","message":"this is returning success on master:\n\n```\nFAILED swift/test/unit/proxy/controllers/test_obj.py::TestReplicatedObjController::test_DELETE_found_fail_and_timeout - AssertionError: 204 !\u003d 503\n```\n\nI think previously the 404s would come out of resp pile and make their way into best_response where the \"status_overrides \u003d {404: 204}` would turn this into success:\n\nhttps://github.com/NVIDIA/swift/blob/master/swift/proxy/controllers/obj.py#L815","commit_id":"e05a74ccdc4d145959c33d757860ada0a89e4667"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"2e3b63a255f0b9baf268beac3c76cebd9841efee","unresolved":true,"context_lines":[{"line_number":1389,"context_line":"        with set_http_connect(*(primary_codes + handoff_codes)):"},{"line_number":1390,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":1391,"context_line":"        self.assertEqual(503, resp.status_int,"},{"line_number":1392,"context_line":"                         \u0027replicas \u003d %s\u0027 % self.replicas())"},{"line_number":1393,"context_line":""},{"line_number":1394,"context_line":"    def test_POST_all_primaries_fail_below_quorum_handoff_succeeds(self):"},{"line_number":1395,"context_line":"        req \u003d swift.common.swob.Request.blank(\u0027/v1/a/c/o\u0027, method\u003d\u0027POST\u0027)"}],"source_content_type":"text/x-python","patch_set":3,"id":"53796141_d03c9d66","line":1392,"updated":"2024-08-12 17:25:07.000000000","message":"if any of those handoff_codes\u0027 404s have an x-backend-timestamp (i.e. a tombstone, especially one that\u0027s *newer* than te 202 datafile timestamp) I\u0027d prefer not to have to return 503 here.\n\nThose responses should just be telling us the user is trying to POST to an object they just DELETEd while the primaries were offline; the new .meta on the 202 should eventually get washed out by the tombstones.\n\nBut if we can\u0027t inspect the 202/404 headers we probably can\u0027t know if they just did  a DELETE and a 503 is probably better than 404.","commit_id":"e05a74ccdc4d145959c33d757860ada0a89e4667"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8615cea0b399558f758e01a14102aaf694569818","unresolved":false,"context_lines":[{"line_number":1389,"context_line":"        with set_http_connect(*(primary_codes + handoff_codes)):"},{"line_number":1390,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":1391,"context_line":"        self.assertEqual(503, resp.status_int,"},{"line_number":1392,"context_line":"                         \u0027replicas \u003d %s\u0027 % self.replicas())"},{"line_number":1393,"context_line":""},{"line_number":1394,"context_line":"    def test_POST_all_primaries_fail_below_quorum_handoff_succeeds(self):"},{"line_number":1395,"context_line":"        req \u003d swift.common.swob.Request.blank(\u0027/v1/a/c/o\u0027, method\u003d\u0027POST\u0027)"}],"source_content_type":"text/x-python","patch_set":3,"id":"bff1b59b_de592c4d","line":1392,"in_reply_to":"53796141_d03c9d66","updated":"2024-08-19 18:36:15.000000000","message":"because POST doesn\u0027t return x-backend-timestamp (yet) we should try to merge this improvement and refine further with x-backend-timestamp in a follow-up.\n\nThe big win with this change is this assertion:\n\n```\nswift/test/unit/proxy/controllers/test_obj.py::TestReplicatedObjController::test_POST_insufficient_primaries_handoffs_404 - AssertionError: 503 !\u003d 404 : replicas \u003d 3\n```\n\n^ shows up when you disable the left-over-status check","commit_id":"e05a74ccdc4d145959c33d757860ada0a89e4667"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8615cea0b399558f758e01a14102aaf694569818","unresolved":true,"context_lines":[{"line_number":533,"context_line":"        random.shuffle(codes)"},{"line_number":534,"context_line":"        with set_http_connect(*codes):"},{"line_number":535,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":536,"context_line":"        self.assertEqual(resp.status_int, 404)"},{"line_number":537,"context_line":""},{"line_number":538,"context_line":"    def test_DELETE_missing_one(self):"},{"line_number":539,"context_line":"        # Obviously this test doesn\u0027t work if we\u0027re testing 1 replica."}],"source_content_type":"text/x-python","patch_set":6,"id":"1646adf2_7bafa618","line":536,"updated":"2024-08-19 18:36:15.000000000","message":"sanity; both pass on master","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"9ac4ae475d0059e7476c2aeb7f617bb4662129b8","unresolved":false,"context_lines":[{"line_number":533,"context_line":"        random.shuffle(codes)"},{"line_number":534,"context_line":"        with set_http_connect(*codes):"},{"line_number":535,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":536,"context_line":"        self.assertEqual(resp.status_int, 404)"},{"line_number":537,"context_line":""},{"line_number":538,"context_line":"    def test_DELETE_missing_one(self):"},{"line_number":539,"context_line":"        # Obviously this test doesn\u0027t work if we\u0027re testing 1 replica."}],"source_content_type":"text/x-python","patch_set":6,"id":"275605d5_91286207","line":536,"in_reply_to":"1646adf2_7bafa618","updated":"2024-08-21 15:27:56.000000000","message":"Acknowledged","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8615cea0b399558f758e01a14102aaf694569818","unresolved":true,"context_lines":[{"line_number":582,"context_line":"        with set_http_connect(404, 204, 507):"},{"line_number":583,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":584,"context_line":"        # status_overrides passed to make_requests converts the 404 to a 204..."},{"line_number":585,"context_line":"        self.assertEqual(resp.status_int, 204)"},{"line_number":586,"context_line":""},{"line_number":587,"context_line":"    def test_DELETE_found_fail_and_timeout(self):"},{"line_number":588,"context_line":"        self.obj_ring.set_replicas(3)"}],"source_content_type":"text/x-python","patch_set":6,"id":"2567d56a_245d2fbe","line":585,"updated":"2024-08-19 18:36:15.000000000","message":"sanity; passes on master","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"9ac4ae475d0059e7476c2aeb7f617bb4662129b8","unresolved":false,"context_lines":[{"line_number":582,"context_line":"        with set_http_connect(404, 204, 507):"},{"line_number":583,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":584,"context_line":"        # status_overrides passed to make_requests converts the 404 to a 204..."},{"line_number":585,"context_line":"        self.assertEqual(resp.status_int, 204)"},{"line_number":586,"context_line":""},{"line_number":587,"context_line":"    def test_DELETE_found_fail_and_timeout(self):"},{"line_number":588,"context_line":"        self.obj_ring.set_replicas(3)"}],"source_content_type":"text/x-python","patch_set":6,"id":"bae22c02_65f03b8b","line":585,"in_reply_to":"2567d56a_245d2fbe","updated":"2024-08-21 15:27:56.000000000","message":"Acknowledged","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8615cea0b399558f758e01a14102aaf694569818","unresolved":true,"context_lines":[{"line_number":634,"context_line":"        with set_http_connect(*(primary_codes + handoff_codes)):"},{"line_number":635,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":636,"context_line":"        self.assertEqual(503, resp.status_int,"},{"line_number":637,"context_line":"                         \u0027replicas \u003d %s\u0027 % self.replicas())"},{"line_number":638,"context_line":""},{"line_number":639,"context_line":"    def test_DELETE_limits_expirer_queue_updates(self):"},{"line_number":640,"context_line":"        req \u003d swift.common.swob.Request.blank(\u0027/v1/a/c/o\u0027, method\u003d\u0027DELETE\u0027)"}],"source_content_type":"text/x-python","patch_set":6,"id":"1b5b32ff_b4880237","line":637,"updated":"2024-08-19 18:36:15.000000000","message":"this was 404 on master\n\n```\nFAILED swift/test/unit/proxy/controllers/test_obj.py::TestReplicatedObjController::test_DELETE_below_quorum_of_primaries_succeed - AssertionError: 503 !\u003d 404 : replicas \u003d 3\n```","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"9ac4ae475d0059e7476c2aeb7f617bb4662129b8","unresolved":true,"context_lines":[{"line_number":634,"context_line":"        with set_http_connect(*(primary_codes + handoff_codes)):"},{"line_number":635,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":636,"context_line":"        self.assertEqual(503, resp.status_int,"},{"line_number":637,"context_line":"                         \u0027replicas \u003d %s\u0027 % self.replicas())"},{"line_number":638,"context_line":""},{"line_number":639,"context_line":"    def test_DELETE_limits_expirer_queue_updates(self):"},{"line_number":640,"context_line":"        req \u003d swift.common.swob.Request.blank(\u0027/v1/a/c/o\u0027, method\u003d\u0027DELETE\u0027)"}],"source_content_type":"text/x-python","patch_set":6,"id":"255a483b_883e8a59","line":637,"in_reply_to":"1b5b32ff_b4880237","updated":"2024-08-21 15:27:56.000000000","message":"as before, I think This would have returned success with this patch too if I had thought to add the backend timestamp to the mocked 404 response :( , my bad","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8615cea0b399558f758e01a14102aaf694569818","unresolved":true,"context_lines":[{"line_number":1343,"context_line":"        with set_http_connect(*primary_codes):"},{"line_number":1344,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":1345,"context_line":"        self.assertEqual(202, resp.status_int,"},{"line_number":1346,"context_line":"                         \u0027replicas \u003d %s\u0027 % self.replicas())"},{"line_number":1347,"context_line":""},{"line_number":1348,"context_line":"    def test_POST_sufficient_primaries_succeed_others_404(self):"},{"line_number":1349,"context_line":"        req \u003d swift.common.swob.Request.blank(\u0027/v1/a/c/o\u0027, method\u003d\u0027POST\u0027)"}],"source_content_type":"text/x-python","patch_set":6,"id":"6e74fe43_54f52edf","line":1346,"updated":"2024-08-19 18:36:15.000000000","message":"sanity; passes on master","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"9ac4ae475d0059e7476c2aeb7f617bb4662129b8","unresolved":false,"context_lines":[{"line_number":1343,"context_line":"        with set_http_connect(*primary_codes):"},{"line_number":1344,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":1345,"context_line":"        self.assertEqual(202, resp.status_int,"},{"line_number":1346,"context_line":"                         \u0027replicas \u003d %s\u0027 % self.replicas())"},{"line_number":1347,"context_line":""},{"line_number":1348,"context_line":"    def test_POST_sufficient_primaries_succeed_others_404(self):"},{"line_number":1349,"context_line":"        req \u003d swift.common.swob.Request.blank(\u0027/v1/a/c/o\u0027, method\u003d\u0027POST\u0027)"}],"source_content_type":"text/x-python","patch_set":6,"id":"2a74afb3_e9946097","line":1346,"in_reply_to":"6e74fe43_54f52edf","updated":"2024-08-21 15:27:56.000000000","message":"Acknowledged","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8615cea0b399558f758e01a14102aaf694569818","unresolved":true,"context_lines":[{"line_number":1355,"context_line":"        with set_http_connect(*primary_codes):"},{"line_number":1356,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":1357,"context_line":"        self.assertEqual(202, resp.status_int,"},{"line_number":1358,"context_line":"                         \u0027replicas \u003d %s\u0027 % self.replicas())"},{"line_number":1359,"context_line":""},{"line_number":1360,"context_line":"    def test_POST_insufficient_primaries_succeed_others_404(self):"},{"line_number":1361,"context_line":"        req \u003d swift.common.swob.Request.blank(\u0027/v1/a/c/o\u0027, method\u003d\u0027POST\u0027)"}],"source_content_type":"text/x-python","patch_set":6,"id":"7d77caa8_3ced0dff","line":1358,"updated":"2024-08-19 18:36:15.000000000","message":"interestingly *this* one passes on master; but ONLY for the \"TestReplicatedObjControllerVariousReplicas\" TestCase; all others\n\n * TestReplicatedObjController\n * TestECObjController\n * TestECDuplicationObjController\n \nhave 5 failures:\n\n * test_DELETE_below_quorum_of_primaries_succeed\n * test_DELETE_found_fail_and_timeout\n * test_POST_all_primaries_fail_insufficient_handoff_succeeds\n * test_POST_insufficient_primaries_handoffs_404\n * test_POST_sufficient_primaries_succeed_others_fail","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"9ac4ae475d0059e7476c2aeb7f617bb4662129b8","unresolved":true,"context_lines":[{"line_number":1355,"context_line":"        with set_http_connect(*primary_codes):"},{"line_number":1356,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":1357,"context_line":"        self.assertEqual(202, resp.status_int,"},{"line_number":1358,"context_line":"                         \u0027replicas \u003d %s\u0027 % self.replicas())"},{"line_number":1359,"context_line":""},{"line_number":1360,"context_line":"    def test_POST_insufficient_primaries_succeed_others_404(self):"},{"line_number":1361,"context_line":"        req \u003d swift.common.swob.Request.blank(\u0027/v1/a/c/o\u0027, method\u003d\u0027POST\u0027)"}],"source_content_type":"text/x-python","patch_set":6,"id":"8690088c_1d662a11","line":1358,"in_reply_to":"7d77caa8_3ced0dff","updated":"2024-08-21 15:27:56.000000000","message":"IIUC TestReplicatedObjControllerVariousReplicas actually just sets up a bunch of rings and the default happens to be replicas\u003d1, so there would be just the one 204 response in that case.","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8615cea0b399558f758e01a14102aaf694569818","unresolved":true,"context_lines":[{"line_number":1365,"context_line":"        with set_http_connect(*primary_codes):"},{"line_number":1366,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":1367,"context_line":"        self.assertEqual(404, resp.status_int,"},{"line_number":1368,"context_line":"                         \u0027replicas \u003d %s\u0027 % self.replicas())"},{"line_number":1369,"context_line":""},{"line_number":1370,"context_line":"    def test_POST_sufficient_primaries_succeed_others_fail(self):"},{"line_number":1371,"context_line":"        req \u003d swift.common.swob.Request.blank(\u0027/v1/a/c/o\u0027, method\u003d\u0027POST\u0027)"}],"source_content_type":"text/x-python","patch_set":6,"id":"d9f6fb10_9448b9f4","line":1368,"updated":"2024-08-19 18:36:15.000000000","message":"sanity; passes on master","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"9ac4ae475d0059e7476c2aeb7f617bb4662129b8","unresolved":false,"context_lines":[{"line_number":1365,"context_line":"        with set_http_connect(*primary_codes):"},{"line_number":1366,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":1367,"context_line":"        self.assertEqual(404, resp.status_int,"},{"line_number":1368,"context_line":"                         \u0027replicas \u003d %s\u0027 % self.replicas())"},{"line_number":1369,"context_line":""},{"line_number":1370,"context_line":"    def test_POST_sufficient_primaries_succeed_others_fail(self):"},{"line_number":1371,"context_line":"        req \u003d swift.common.swob.Request.blank(\u0027/v1/a/c/o\u0027, method\u003d\u0027POST\u0027)"}],"source_content_type":"text/x-python","patch_set":6,"id":"7f4d9bdd_0c0dfa47","line":1368,"in_reply_to":"d9f6fb10_9448b9f4","updated":"2024-08-21 15:27:56.000000000","message":"Acknowledged","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8615cea0b399558f758e01a14102aaf694569818","unresolved":true,"context_lines":[{"line_number":1376,"context_line":"        primary_codes \u003d [202] * primary_success + [Timeout()] * primary_failure"},{"line_number":1377,"context_line":"        # if replicas\u003d\u003d1 there won\u0027t be any failures..."},{"line_number":1378,"context_line":"        handoff_codes \u003d [404] * self.replicas() if primary_failure else []"},{"line_number":1379,"context_line":"        print(\u0027replicas\u0027, self.replicas())"},{"line_number":1380,"context_line":"        with set_http_connect(*(primary_codes + handoff_codes)):"},{"line_number":1381,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":1382,"context_line":"        self.assertEqual(202, resp.status_int,"}],"source_content_type":"text/x-python","patch_set":6,"id":"66002460_a837ee8d","line":1379,"updated":"2024-08-19 18:36:15.000000000","message":"i like the `replicas \u003d` in the assertion error message; this print can probably go","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"9ac4ae475d0059e7476c2aeb7f617bb4662129b8","unresolved":false,"context_lines":[{"line_number":1376,"context_line":"        primary_codes \u003d [202] * primary_success + [Timeout()] * primary_failure"},{"line_number":1377,"context_line":"        # if replicas\u003d\u003d1 there won\u0027t be any failures..."},{"line_number":1378,"context_line":"        handoff_codes \u003d [404] * self.replicas() if primary_failure else []"},{"line_number":1379,"context_line":"        print(\u0027replicas\u0027, self.replicas())"},{"line_number":1380,"context_line":"        with set_http_connect(*(primary_codes + handoff_codes)):"},{"line_number":1381,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":1382,"context_line":"        self.assertEqual(202, resp.status_int,"}],"source_content_type":"text/x-python","patch_set":6,"id":"36b957b1_8a6c75ae","line":1379,"in_reply_to":"66002460_a837ee8d","updated":"2024-08-21 15:27:56.000000000","message":"Done","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8615cea0b399558f758e01a14102aaf694569818","unresolved":true,"context_lines":[{"line_number":1380,"context_line":"        with set_http_connect(*(primary_codes + handoff_codes)):"},{"line_number":1381,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":1382,"context_line":"        self.assertEqual(202, resp.status_int,"},{"line_number":1383,"context_line":"                         \u0027replicas \u003d %s\u0027 % self.replicas())"},{"line_number":1384,"context_line":""},{"line_number":1385,"context_line":"    def test_POST_insufficient_primaries_handoffs_404(self):"},{"line_number":1386,"context_line":"        req \u003d swift.common.swob.Request.blank(\u0027/v1/a/c/o\u0027, method\u003d\u0027POST\u0027)"}],"source_content_type":"text/x-python","patch_set":6,"id":"f75fd9b7_e4eb4a32","line":1383,"updated":"2024-08-19 18:36:15.000000000","message":"bah, AssertionError: left over status [404, 404]\n\nWhen I disable that check I actaully get the 202 here\n\nI don\u0027t think I understand why master requires less requests to return the same response...","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"9ac4ae475d0059e7476c2aeb7f617bb4662129b8","unresolved":true,"context_lines":[{"line_number":1380,"context_line":"        with set_http_connect(*(primary_codes + handoff_codes)):"},{"line_number":1381,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":1382,"context_line":"        self.assertEqual(202, resp.status_int,"},{"line_number":1383,"context_line":"                         \u0027replicas \u003d %s\u0027 % self.replicas())"},{"line_number":1384,"context_line":""},{"line_number":1385,"context_line":"    def test_POST_insufficient_primaries_handoffs_404(self):"},{"line_number":1386,"context_line":"        req \u003d swift.common.swob.Request.blank(\u0027/v1/a/c/o\u0027, method\u003d\u0027POST\u0027)"}],"source_content_type":"text/x-python","patch_set":6,"id":"669488a6_7a992d42","line":1383,"in_reply_to":"f75fd9b7_e4eb4a32","updated":"2024-08-21 15:27:56.000000000","message":"\u003e AssertionError: left over status [404, 404]\n on master I assume?\n \n on master the iterator would have returned after the first 404 but with this patchset it keeps chugging through the handoffs","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8615cea0b399558f758e01a14102aaf694569818","unresolved":true,"context_lines":[{"line_number":1402,"context_line":"        with set_http_connect(*(primary_codes + handoff_codes)):"},{"line_number":1403,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":1404,"context_line":"        self.assertEqual(503, resp.status_int,"},{"line_number":1405,"context_line":"                         \u0027replicas \u003d %s\u0027 % self.replicas())"},{"line_number":1406,"context_line":""},{"line_number":1407,"context_line":"    def test_POST_all_primaries_fail_sufficient_handoff_succeeds(self):"},{"line_number":1408,"context_line":"        req \u003d swift.common.swob.Request.blank(\u0027/v1/a/c/o\u0027, method\u003d\u0027POST\u0027)"}],"source_content_type":"text/x-python","patch_set":6,"id":"85a49b7f_70604c64","line":1405,"updated":"2024-08-19 18:36:15.000000000","message":"I think this is similar to `test_POST_insufficient_primaries_handoffs_404`\n\n```\nAssertionError: 503 !\u003d 404 : replicas \u003d 3\n```\n\nThe idea here is all primaries 503, and the handoffs are 1-shy of a quorum; the 404s from handoffs don\u0027t help.\n\nOn master the error is:\n\n```\nFAILED swift/test/unit/proxy/controllers/test_obj.py::TestReplicatedObjController::test_POST_insufficient_primaries_handoffs_404 - AssertionError: left over status [404]\n```\n\nagain i\u0027m totally confused how master is making less requests; I assume the expected response pattern is:\n\n```\n[Timeout, Timeout, Timeout, 202, 404, 404]\n```\n\n... and where as master decides that looks like \"mostly 404\" this change correctly identifies it as ambigous response and returns error to encourage a retry.","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"9ac4ae475d0059e7476c2aeb7f617bb4662129b8","unresolved":false,"context_lines":[{"line_number":1402,"context_line":"        with set_http_connect(*(primary_codes + handoff_codes)):"},{"line_number":1403,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":1404,"context_line":"        self.assertEqual(503, resp.status_int,"},{"line_number":1405,"context_line":"                         \u0027replicas \u003d %s\u0027 % self.replicas())"},{"line_number":1406,"context_line":""},{"line_number":1407,"context_line":"    def test_POST_all_primaries_fail_sufficient_handoff_succeeds(self):"},{"line_number":1408,"context_line":"        req \u003d swift.common.swob.Request.blank(\u0027/v1/a/c/o\u0027, method\u003d\u0027POST\u0027)"}],"source_content_type":"text/x-python","patch_set":6,"id":"7b818269_d7155b50","line":1405,"in_reply_to":"85a49b7f_70604c64","updated":"2024-08-21 15:27:56.000000000","message":"Acknowledged","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8615cea0b399558f758e01a14102aaf694569818","unresolved":true,"context_lines":[{"line_number":1413,"context_line":"        with set_http_connect(*(primary_codes + handoff_codes)):"},{"line_number":1414,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":1415,"context_line":"        self.assertEqual(202, resp.status_int,"},{"line_number":1416,"context_line":"                         \u0027replicas \u003d %s\u0027 % self.replicas())"},{"line_number":1417,"context_line":""},{"line_number":1418,"context_line":"# end of CommonObjectControllerMixin"},{"line_number":1419,"context_line":""}],"source_content_type":"text/x-python","patch_set":6,"id":"da0293fb_fdf84bc2","line":1416,"updated":"2024-08-19 18:36:15.000000000","message":"sanity, passes on master","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"9ac4ae475d0059e7476c2aeb7f617bb4662129b8","unresolved":false,"context_lines":[{"line_number":1413,"context_line":"        with set_http_connect(*(primary_codes + handoff_codes)):"},{"line_number":1414,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":1415,"context_line":"        self.assertEqual(202, resp.status_int,"},{"line_number":1416,"context_line":"                         \u0027replicas \u003d %s\u0027 % self.replicas())"},{"line_number":1417,"context_line":""},{"line_number":1418,"context_line":"# end of CommonObjectControllerMixin"},{"line_number":1419,"context_line":""}],"source_content_type":"text/x-python","patch_set":6,"id":"7bfd340e_3710f6ca","line":1416,"in_reply_to":"da0293fb_fdf84bc2","updated":"2024-08-21 15:27:56.000000000","message":"Acknowledged","commit_id":"ff76290e67bb2c38d79a9a9f014769e08fb09972"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"d831f04c5d6c54624ec1b7a5c8774a2b12674a9c","unresolved":true,"context_lines":[{"line_number":1445,"context_line":"        primary_codes \u003d [404] * primary_failure + [202] * primary_success"},{"line_number":1446,"context_line":"        with mocked_http_conn(*primary_codes):"},{"line_number":1447,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":1448,"context_line":"        # TODO: should this be a 503?"},{"line_number":1449,"context_line":"        self.assertEqual(404, resp.status_int,"},{"line_number":1450,"context_line":"                         \u0027replicas \u003d %s\u0027 % self.replicas())"},{"line_number":1451,"context_line":""}],"source_content_type":"text/x-python","patch_set":7,"id":"66581fe9_0d477735","line":1448,"updated":"2024-08-21 16:01:39.000000000","message":"still this case to think about","commit_id":"cfd6a841ce4b82146feb0d37291c90f6b52e3d50"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"af43962b54539c2e1e437c062407042e2f093960","unresolved":true,"context_lines":[{"line_number":1445,"context_line":"        primary_codes \u003d [404] * primary_failure + [202] * primary_success"},{"line_number":1446,"context_line":"        with mocked_http_conn(*primary_codes):"},{"line_number":1447,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":1448,"context_line":"        # TODO: should this be a 503?"},{"line_number":1449,"context_line":"        self.assertEqual(404, resp.status_int,"},{"line_number":1450,"context_line":"                         \u0027replicas \u003d %s\u0027 % self.replicas())"},{"line_number":1451,"context_line":""}],"source_content_type":"text/x-python","patch_set":7,"id":"33c2f080_d8c99831","line":1448,"in_reply_to":"66581fe9_0d477735","updated":"2024-08-23 15:01:23.000000000","message":"I think this case is interesting but I\u0027d like to consider it as a follow-up. The scope of this patch, handoffs responses, already fixes a significant bug for end-users.","commit_id":"cfd6a841ce4b82146feb0d37291c90f6b52e3d50"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"332f493620c311a13cb66c1359308cfa829f36ef","unresolved":true,"context_lines":[{"line_number":629,"context_line":"        with mocked_http_conn(*(primary_codes + handoff_codes),"},{"line_number":630,"context_line":"                              headers\u003dheaders):"},{"line_number":631,"context_line":"            resp \u003d req.get_response(self.app)"},{"line_number":632,"context_line":"        # handoff 404s form a quorum..."},{"line_number":633,"context_line":"        self.assertEqual(404, resp.status_int,"},{"line_number":634,"context_line":"                         \u0027replicas \u003d %s\u0027 % self.replicas())"},{"line_number":635,"context_line":"        self.assertEqual(ts.internal, resp.headers.get(\u0027X-Backend-Timestamp\u0027))"}],"source_content_type":"text/x-python","patch_set":8,"id":"075dfecb_f89fb128","line":632,"updated":"2024-09-04 16:32:23.000000000","message":"I\u0027m surprised this didn\u0027t need to change... Oh, but the 404s all have timestamps!","commit_id":"d0f6a0ce5624e271bfc3e2cca7d328a18df546f4"}]}