)]}'
{"/PATCHSET_LEVEL":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8bcd07b797ee36a0e2c17065dadb5e618dda4a90","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"a9f00f16_c9b0e0b0","updated":"2024-11-15 22:12:50.000000000","message":"i don\u0027t think i even understand how this change works - it seems like it\u0027s borrowing machinery that\u0027s the existing redirect code that tries to handle when a async_pending is targeting the root but should go to a shard and somehow reversing that so that it goes back from the shard to the root (on the next pass?)\n\nWe\u0027ve added a new key db_state for when we have a async pointed at a root that should go to a shard; I think the current effect is just to skip the inline update - and go directly to async\n\nI think with this change; we\u0027ll hit the root async - but if it\u0027s still overloaded... we\u0027ll we just try again.  It\u0027s only when the shard 404s (it shrunk?  it got split into new shards?) that we try to go back to the root?\n\nI think Matt had a patch that was trying to get the updater to look in *memcache* for most recent shard update table:\n\n874721: updater: add memcache shard update lookup support | https://review.opendev.org/c/openstack/swift/+/874721\n\nmaybe we should look at that more closely before we decide what to do here.","commit_id":"2ac7d9dfcdf440f3b6120341002c65beb895cb8c"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"e343d21fc02be79ecd01bc1d7a28805f29383c16","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"33dd57d2_b44729fb","updated":"2023-07-06 19:17:55.000000000","message":"thanks for working this up!  it\u0027s a smaller diff that I would have expected.\n\nHowever, in our case having our \"rather old\" asyncs processed into the correct shards would result in dark-data/ghost-listings, and we should probably assume other clusters might be in a similar state.  I don\u0027t want to merge this until I fix my cluster, and maybe not in general until we deal with old async pendings https://bugs.launchpad.net/swift/+bug/1619408\n\nAlso, we\u0027ve had problems with the root getting overwhelemed by the updater - matt is working on optimizing that https://review.opendev.org/c/openstack/swift/+/874721/5 but throwing more load at the root doesn\u0027t seem obviously the correct solution to me right now.","commit_id":"2ac7d9dfcdf440f3b6120341002c65beb895cb8c"}],"swift/obj/updater.py":[{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"bd58195f29d25474d9468776fbaa3afae50cb4dc","unresolved":true,"context_lines":[{"line_number":661,"context_line":"            elif redirects:"},{"line_number":662,"context_line":"                # erase any previous successes"},{"line_number":663,"context_line":"                update.pop(\u0027successes\u0027, None)"},{"line_number":664,"context_line":"                redirect \u003d max(redirects, key\u003dlambda x: x[-1])[0]"},{"line_number":665,"context_line":"                redirect_history \u003d update.setdefault(\u0027redirect_history\u0027, [])"},{"line_number":666,"context_line":"                if redirect in redirect_history:"},{"line_number":667,"context_line":"                    # force next update to be sent to root, reset history"}],"source_content_type":"text/x-python","patch_set":1,"id":"91ab3abc_77c1fe36","line":664,"updated":"2023-07-05 20:12:46.000000000","message":"Should maybe check for `None` here and only use it if `len(redirects)` is at least a quorum of `len(events)` -- getting back `500, 500, 404` is likely to be an overloaded shard during a rebalance.","commit_id":"2ac7d9dfcdf440f3b6120341002c65beb895cb8c"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"bd58195f29d25474d9468776fbaa3afae50cb4dc","unresolved":true,"context_lines":[{"line_number":739,"context_line":"                        \u0027redirect location: %s\u0027 % (obj, err))"},{"line_number":740,"context_line":""},{"line_number":741,"context_line":"            success \u003d is_success(status)"},{"line_number":742,"context_line":"            if is_shard_update and is_client_error(status):"},{"line_number":743,"context_line":"                # This is probably a 404; regardless, we don\u0027t really expect"},{"line_number":744,"context_line":"                # later retries to succeed. We have a root-db we could check,"},{"line_number":745,"context_line":"                # though; hopefully we\u0027ll have better luck there?"}],"source_content_type":"text/x-python","patch_set":1,"id":"a343a82a_2a634073","line":742,"range":{"start_line":742,"start_character":35,"end_line":742,"end_character":58},"updated":"2023-07-05 20:12:46.000000000","message":"Might want to scope this down to `status \u003d\u003d 404` -- though presumably if we get back 400s at the shard, we\u0027ll get them back at the root, too.","commit_id":"2ac7d9dfcdf440f3b6120341002c65beb895cb8c"}]}