)]}'
{"/PATCHSET_LEVEL":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"c0d4119acc2532697eb77bc21eb6e1d048615b9e","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"36b38994_a452f6fd","updated":"2026-08-28 23:16:18.000000000","message":"I\u0027m growing in confidence that we can safely target the waiting lagard condition for the 4xx response case\n\nI do not think we should fix it once for non-conditional gets and then go back and fix it again for GETs with Range - it should be the same logic; but there\u0027s something fishy going on that I we could probably nail down if we put our heads together:\n\n1002841: WIP: key EC error evidence by decision identity | https://review.opendev.org/c/openstack/swift/+/1002841\n\nI think ultimately it\u0027s not worth it to avoid the problem here; a half solution could cause more problems later.\n\nFWIW tho I *do* think there is a *real* operator/metrics preference here - the answer \"1\" is conservative, but even that may depend on the cluster and operations team - I think WE will want to push it further (my gut says ~3 on our 2 * (8+4) default node_requests - but I\u0027ll want to be able to do some testing).","commit_id":"58567198339842c43924bd35ffee5c513f1754d8"}],"swift/proxy/controllers/obj.py":[{"author":{"_account_id":6968,"name":"Christian Schwede","email":"cschwede@nvidia.com","username":"cschwede"},"change_message_id":"104b6614d6b7e68845d6bc76b57d41e759933100","unresolved":true,"context_lines":[{"line_number":3018,"context_line":"                    extra_requests +\u003d 1"},{"line_number":3019,"context_line":"                    pile.spawn(self._fragment_GET_request, req, safe_iter,"},{"line_number":3020,"context_line":"                               partition, policy, buckets.get_extra_headers,"},{"line_number":3021,"context_line":"                               self.logger.thread_locals)"},{"line_number":3022,"context_line":"                elif not requests_available and not buckets.buckets and \\"},{"line_number":3023,"context_line":"                        pile._pending \u003c policy.ec_ndata:"},{"line_number":3024,"context_line":"                    # no frags, and too few replies left to ever reach"}],"source_content_type":"text/x-python","patch_set":1,"id":"116d2d70_c3104bd2","line":3021,"updated":"2026-08-05 11:31:47.000000000","message":"Note: buckets.buckets is the dict of good responses populted by add_good_response (probably should add this to the inline comment)","commit_id":"785afed926a83b09d09635c07337b776c53b3ef4"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"c0d4119acc2532697eb77bc21eb6e1d048615b9e","unresolved":true,"context_lines":[{"line_number":2552,"context_line":"        # are gone, and the reply in flight is something other than a 404."},{"line_number":2553,"context_line":"        # a negative suppression count is configurable but means no"},{"line_number":2554,"context_line":"        # suppression, as elsewhere"},{"line_number":2555,"context_line":"        trusted_404s \u003d len(not_found.get_responses()) - max(0, suppressed)"},{"line_number":2556,"context_line":"        return trusted_404s \u003e\u003d quorum_size(status_count + outstanding)"},{"line_number":2557,"context_line":""},{"line_number":2558,"context_line":"    @property"}],"source_content_type":"text/x-python","patch_set":3,"id":"28c96aa3_e9433a0a","line":2555,"updated":"2026-08-28 23:16:18.000000000","message":"I\u0027m pretty sure the magic \"oustanding \u003d 1\" is kind of *stacking* policy ontop of supressed in a implicit way that is going to be opaque to operations\n\nthere should be a single confidence knob - there\u0027s nothing special about oustanding\u003d1 other than it\u0027s quite conservative:\n\nhttps://review.opendev.org/c/openstack/swift/+/1002831/1/etc/proxy-server.conf-sample","commit_id":"58567198339842c43924bd35ffee5c513f1754d8"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"c0d4119acc2532697eb77bc21eb6e1d048615b9e","unresolved":true,"context_lines":[{"line_number":3048,"context_line":"            # best_response() ranks above the 404s"},{"line_number":3049,"context_line":"            unconditional \u003d not any(("},{"line_number":3050,"context_line":"                orig_range, req.if_match, req.if_none_match,"},{"line_number":3051,"context_line":"                req.if_modified_since, req.if_unmodified_since))"},{"line_number":3052,"context_line":"            for get, parts_iter in pile:"},{"line_number":3053,"context_line":"                try:"},{"line_number":3054,"context_line":"                    buckets.add_response(get, parts_iter)"}],"source_content_type":"text/x-python","patch_set":3,"id":"d0d12c9e_32718bc3","line":3051,"updated":"2026-08-28 23:16:18.000000000","message":"I don\u0027t like this AT ALL - a LOT of our requests are Range requests - therefore a LOT of our 404s are Range requests - therefore MOST of the value we could get here is being cut off for a *request parameter*\n\nWhich doesn\u0027t even make sense given that this change logically should be about early stopping on a hopeless response pattern.\n\n1002829: sq? test Range EC 404 waits for a hung node | https://review.opendev.org/c/openstack/swift/+/1002829\n\nI think the fundemental issue actually is because the current code conflates the bad_bucket 416 stopping incorrectly instead of grouped by timestamp but rather than \"fix\" that we\u0027re choosing to exculde/confront the inhert complexity in the problem","commit_id":"58567198339842c43924bd35ffee5c513f1754d8"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"c0d4119acc2532697eb77bc21eb6e1d048615b9e","unresolved":true,"context_lines":[{"line_number":3147,"context_line":"                        if (rebalance_missing_suppression_count \u003e 0 and"},{"line_number":3148,"context_line":"                                getter.status \u003d\u003d HTTP_NOT_FOUND and"},{"line_number":3149,"context_line":"                                not timestamp):"},{"line_number":3150,"context_line":"                            rebalance_missing_suppression_count -\u003d 1"},{"line_number":3151,"context_line":"                            continue"},{"line_number":3152,"context_line":"                        statuses.append(getter.status)"},{"line_number":3153,"context_line":"                        reasons.append(getter.reason)"}],"source_content_type":"text/x-python","patch_set":3,"id":"d0dc2c22_a730a447","line":3150,"updated":"2026-08-28 23:16:18.000000000","message":"I\u0027d bet if we go back and look at the change that introduced this, it was added as an after thought w/o much careful consideration to what was *decidely* a replicated policy deicision\n\nhttps://review.opendev.org/c/openstack/swift/+/744942/3#message-b458f3dd3282f0d95f626d2e036faea9bf5643da\n\nLike I\u0027m sure it was ~fine - but I seriously doubt it load bearing in prod.","commit_id":"58567198339842c43924bd35ffee5c513f1754d8"}],"test/unit/proxy/controllers/test_obj.py":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"c0d4119acc2532697eb77bc21eb6e1d048615b9e","unresolved":true,"context_lines":[{"line_number":3819,"context_line":"        self.assertEqual(resp.status_int, 503)"},{"line_number":3820,"context_line":"        self.assertEqual(node_count, len(log.requests))"},{"line_number":3821,"context_line":""},{"line_number":3822,"context_line":"    def test_GET_waits_for_412_that_outranks_404(self):"},{"line_number":3823,"context_line":"        req \u003d swift.common.swob.Request.blank("},{"line_number":3824,"context_line":"            \u0027/v1/a/c/o\u0027, headers\u003d{\u0027If-Match\u0027: \u0027\"nope\"\u0027})"},{"line_number":3825,"context_line":"        timeout \u003d self.app.recoverable_node_timeout \u003d 1.0"}],"source_content_type":"text/x-python","patch_set":3,"id":"0f87060c_5e10a397","line":3822,"updated":"2026-08-28 23:16:18.000000000","message":"I think this is a policy mis-classification\n\nwe have the \"special case\" where a long list of 404s (no timestamp) could see a single 200 and decide the whole request should be 503\n\n^ this policy *can* actually trigger false 503s from lone fragments - but lone fragments aren\u0027t *supposed* to happen so we sort of let it point us at the problem\n\n... the idea with the 503 policy is that if we have a (set of) 200s we can\u0027t rebuild that\u0027s a 503 - even with a quorum of 404s (unless the 404s are tombstone/timestamp)\n\nTHIS is test is saying that policy is the same for a 412 - and that\u0027s bollocks\n\n1) we should 503 in this case, it\u0027s not like you can turn around and a request *w/o* If-Match and suddenly the one lone 416 will make your request 200\n2) we don\u0027t have to wait for it - if we have enough confidence in our 404 to say this will never 200, then we have enough confidence in our 404 to say it will \"never\" 416 (even if we DON\u0027T *actually* FIX the policy that says: just like nonconditional requests, a single \"exists\" response will cause a quorum of 404 to 503)\n\nso this test is exposing a bad policy for status codes - and then enshrining a latency to make that incorrect decision.\n\n1002831: WIP: explore EC response-confidence abandonment | https://review.opendev.org/c/openstack/swift/+/1002831","commit_id":"58567198339842c43924bd35ffee5c513f1754d8"}]}