)]}'
{"/COMMIT_MSG":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"952c19be306e3afe53a8858011e9382007b19557","unresolved":true,"context_lines":[{"line_number":6,"context_line":""},{"line_number":7,"context_line":"sharding: shard deleted containers that have shards with objects"},{"line_number":8,"context_line":""},{"line_number":9,"context_line":"There is an edgecase where it\u0027s possible to get unsharded deleted roots that"},{"line_number":10,"context_line":"have shards with objects in them. If this happens they stick around and"},{"line_number":11,"context_line":"never get moved to the sharding state."},{"line_number":12,"context_line":""}],"source_content_type":"text/x-gerrit-commit-message","patch_set":4,"id":"e393a64a_1b92edbd","line":9,"updated":"2022-07-05 13:48:27.000000000","message":"Please wrap commit message lines at 72 chars \n\nhttps://wiki.openstack.org/wiki/GitCommitMessages#Summary_of_Git_commit_message_structure","commit_id":"95b0e6418ec358983d4619174f483731aa741f83"}],"/PATCHSET_LEVEL":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"5ab35315914eb131e601bdd112b11a01f6c2cac8","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"80b8c658_9605b922","updated":"2022-06-03 14:31:40.000000000","message":"nice work!","commit_id":"21ed20578a017a122dc7c0638e9d43883c641e6f"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"7e659b5fc31e7782fede584675ba7e030d1edb61","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":2,"id":"71e88bf0_2bd5b2a5","updated":"2022-06-13 16:31:16.000000000","message":"I think back when we were SUPER skeptical about sharding first rolling out we really wanted a checkpoint where we have injected shard ranges into one of the roots, and they\u0027ve fully replicated to all the nodes - but no one is trying to CLEAVE yet (basically a \"do no harm\" approach)\n\n... then we manually ran the sharder or added the enabled metadata or something\n\nbasically we made what could be a single state transistion in the distributed model into a multi-phase commit; but we never really implemented the rollback because we now mostly just love sharding - GO GO GO!  I think we can collapse unsharded -\u003e sharding_initiated -\u003e sharding into a single pass of the replicator, and should mostly head that direction after we find a quick/safe fix for this failing probetest.","commit_id":"b41f073af82cb3ef9e92f4bb22cd7730c8beac88"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"54bf9df26a44b388975b3ed59ffa197571479abe","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":2,"id":"0c6ecc8d_e7c51202","updated":"2022-06-08 07:17:42.000000000","message":"This is rough, but we could do something like this.. The probe test now fails (haven\u0027t fixed it). I seem to get a 409 conflict, which might be the delete failing because it now isn\u0027t considered deleted.  But ran out of time so havne\u0027t looked into it.","commit_id":"b41f073af82cb3ef9e92f4bb22cd7730c8beac88"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"952c19be306e3afe53a8858011e9382007b19557","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":4,"id":"1348bfc9_aa634fbb","updated":"2022-07-05 13:48:27.000000000","message":"I forgot to drop these comments last week.  I think the strategy seems sane, but the connection plumbing changes are intrusive and inconsistent - they seem like an orthogonal optimization.\n\nI\u0027d like to see a re-spin with passing tests.","commit_id":"95b0e6418ec358983d4619174f483731aa741f83"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"c49674e462808da0793631e52fd277c4551d460c","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":4,"id":"c31cfca0_8154292f","updated":"2022-07-05 18:56:57.000000000","message":"I know that we want to get is_deleted to respond false in the sharder when a root with a delete timestamp still has shard ranges with objects in them\n\nI don\u0027t know how to reconicle the difference between db_state and the osr.state wrt to alternate object counts - and the tests are trying to tell us we don\u0027t have a clear model for how that works - we need to make it LESS complex\n\n... but we definately need to at least have PASSING tests https://review.opendev.org/c/openstack/swift/+/848776","commit_id":"95b0e6418ec358983d4619174f483731aa741f83"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"d2503bb89410b810c190cd02e55590db97b7fe29","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":4,"id":"27d033c5_f73f8a9f","updated":"2022-07-07 15:24:10.000000000","message":"I took an alternative approach here https://review.opendev.org/c/openstack/swift/+/848962","commit_id":"95b0e6418ec358983d4619174f483731aa741f83"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"7670ee02ca312c54e3a8d3557790dfd7078bbddc","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":4,"id":"122bd4cf_1a486296","updated":"2022-07-05 20:11:51.000000000","message":"i\u0027m realizing some of the connection plumbing may have been in response to the weirdness with in-memory databases that popped up for me working on https://review.opendev.org/c/openstack/swift/+/848780","commit_id":"95b0e6418ec358983d4619174f483731aa741f83"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"5c4c8d3eebb50a81ffb1da89cad02d94956ba95b","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":4,"id":"d46ba375_a0afb91c","in_reply_to":"122bd4cf_1a486296","updated":"2022-07-05 23:06:49.000000000","message":"yeah, I found it out the hard way when testing in-memory brokers. Fun times. I\u0027ll take a look at yours.","commit_id":"95b0e6418ec358983d4619174f483731aa741f83"}],"swift/container/backend.py":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"7e659b5fc31e7782fede584675ba7e030d1edb61","unresolved":true,"context_lines":[{"line_number":416,"context_line":"        Returns True if a broker has shard range state that would be necessary"},{"line_number":417,"context_line":"        for sharding to have been initiated, False otherwise."},{"line_number":418,"context_line":"        \"\"\""},{"line_number":419,"context_line":"        own_shard_range \u003d self._own_shard_range()"},{"line_number":420,"context_line":"        if own_shard_range.state in (ShardRange.SHARDING,"},{"line_number":421,"context_line":"                                     ShardRange.SHRINKING,"},{"line_number":422,"context_line":"                                     ShardRange.SHARDED,"}],"source_content_type":"text/x-python","patch_set":2,"id":"40cbef94_76bfcb6a","line":419,"updated":"2022-06-13 16:31:16.000000000","message":"this seems like a subtle difference, the public method does some extra queries - so i\u0027m guessing this is just an optimization since we only need the osr.state","commit_id":"b41f073af82cb3ef9e92f4bb22cd7730c8beac88"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"294138416f7052f61d57c1e29ae70bae85354543","unresolved":true,"context_lines":[{"line_number":416,"context_line":"        Returns True if a broker has shard range state that would be necessary"},{"line_number":417,"context_line":"        for sharding to have been initiated, False otherwise."},{"line_number":418,"context_line":"        \"\"\""},{"line_number":419,"context_line":"        own_shard_range \u003d self._own_shard_range()"},{"line_number":420,"context_line":"        if own_shard_range.state in (ShardRange.SHARDING,"},{"line_number":421,"context_line":"                                     ShardRange.SHRINKING,"},{"line_number":422,"context_line":"                                     ShardRange.SHARDED,"}],"source_content_type":"text/x-python","patch_set":2,"id":"66955469_82351a24","line":419,"in_reply_to":"40cbef94_76bfcb6a","updated":"2022-06-14 07:22:35.000000000","message":"Yeah, we only need toe curent state. Also the public method call will call get_info to get object states which calls get_alternate_obj_states which calls sharding_initiated (as of this patch), so gives us recursion loop. BUT we don\u0027t care abount the current calculated stats we just want the state (and ranges).","commit_id":"b41f073af82cb3ef9e92f4bb22cd7730c8beac88"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"7e659b5fc31e7782fede584675ba7e030d1edb61","unresolved":true,"context_lines":[{"line_number":909,"context_line":"    def _get_alternate_object_stats(self):"},{"line_number":910,"context_line":"        state \u003d self.get_db_state()"},{"line_number":911,"context_line":"        if state \u003d\u003d UNSHARDED and self._get_info()[\u0027object_count\u0027] \u003d\u003d 0 and \\"},{"line_number":912,"context_line":"                self.sharding_initiated():"},{"line_number":913,"context_line":"            stats \u003d self.get_shard_usage()"},{"line_number":914,"context_line":"        elif state \u003d\u003d SHARDING:"},{"line_number":915,"context_line":"            other_info \u003d self.get_brokers()[0]._get_info()"}],"source_content_type":"text/x-python","patch_set":2,"id":"6ac4aa33_354a302d","line":912,"updated":"2022-06-13 16:31:16.000000000","message":"\"sharding_initiated\" seems like reason enough to return the summed up shard usage - the is_deleted method doesn\u0027t even look at the value of the returned state\n\nheck; does it *really* need to be conditional at all?  If we have active shard ranges telling us they have objects in them - are we really deleted?  I don\u0027t think so... unless we have stale shard_range state?  but that\u0027d be a different bug.","commit_id":"b41f073af82cb3ef9e92f4bb22cd7730c8beac88"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"294138416f7052f61d57c1e29ae70bae85354543","unresolved":true,"context_lines":[{"line_number":909,"context_line":"    def _get_alternate_object_stats(self):"},{"line_number":910,"context_line":"        state \u003d self.get_db_state()"},{"line_number":911,"context_line":"        if state \u003d\u003d UNSHARDED and self._get_info()[\u0027object_count\u0027] \u003d\u003d 0 and \\"},{"line_number":912,"context_line":"                self.sharding_initiated():"},{"line_number":913,"context_line":"            stats \u003d self.get_shard_usage()"},{"line_number":914,"context_line":"        elif state \u003d\u003d SHARDING:"},{"line_number":915,"context_line":"            other_info \u003d self.get_brokers()[0]._get_info()"}],"source_content_type":"text/x-python","patch_set":2,"id":"979fab96_705765b8","line":912,"in_reply_to":"6ac4aa33_354a302d","updated":"2022-06-14 07:22:35.000000000","message":"Interesting thought. I know Al was under the impression, if we\u0027re in the UNSHARDED state we should not look elsewhere.\n\nThe sharding_initiated is looking at the OSR state, and if so checks to see if there are shardranges, but it\u0027s an any non-deleted shardrange not just ACTIVE ranges.\n\nSo we have 2 state machines. This is currently looking at the DB state machine. sharding_initiated is looking at the OSR one. When the DB state is SHARDING we have 2 brokers on disk, and we want to look at the metadata in the old broker. \n\nIf we are in the UNSHARDED DB state, but OSR is in SHARDING then can we trust the shards we have to give us the \"correct enough\" details? In theory the FOUND will have the object_counts from the initial find, but currently those in the CREATED state can be 0\u0027ed if the sharder gets to them, that is until (https://review.opendev.org/c/openstack/swift/+/837811) lands.\nBut having said that, I\u0027m not sure we\u0027re tracking the byte_count, so the states aren\u0027t necessarily correct when ranges state \u003c ACTIVE.\n\nArguably, in the case of what we are trying to fix, _ANY_ object in a shardrange means it can\u0027t be deleted, so loosening the stats belt make sense in that situation. But that about at other times when we want to most up to date stats.\n\nMaybe it needs to be:\n\n - SHARDING: Look at old broker\n - SHARDED: Look at shards\n - UNSHARDED: Look at self (object table), unless OSR is in SHARDED state, at which case the shards should be \"active\" enough to be the source of knowledge?","commit_id":"b41f073af82cb3ef9e92f4bb22cd7730c8beac88"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"952c19be306e3afe53a8858011e9382007b19557","unresolved":true,"context_lines":[{"line_number":421,"context_line":"                                     ShardRange.SHRINKING,"},{"line_number":422,"context_line":"                                     ShardRange.SHARDED,"},{"line_number":423,"context_line":"                                     ShardRange.SHRUNK):"},{"line_number":424,"context_line":"            return bool(self.get_shard_ranges())"},{"line_number":425,"context_line":"        return False"},{"line_number":426,"context_line":""},{"line_number":427,"context_line":"    def sharding_required(self):"}],"source_content_type":"text/x-python","patch_set":4,"id":"3ddb39de_8afa9382","line":424,"updated":"2022-07-05 13:48:27.000000000","message":"FWIW, it\u0027s still making two connections","commit_id":"95b0e6418ec358983d4619174f483731aa741f83"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"952c19be306e3afe53a8858011e9382007b19557","unresolved":true,"context_lines":[{"line_number":820,"context_line":"            SELECT put_timestamp, delete_timestamp, object_count"},{"line_number":821,"context_line":"            FROM container_stat\u0027\u0027\u0027).fetchone()"},{"line_number":822,"context_line":"        info \u003d dict(info)"},{"line_number":823,"context_line":"        info.update(self._get_alternate_object_stats(connection\u003dconn)[1])"},{"line_number":824,"context_line":"        return self._is_deleted_info(**info)"},{"line_number":825,"context_line":""},{"line_number":826,"context_line":"    def is_old_enough_to_reclaim(self, now, reclaim_age):"}],"source_content_type":"text/x-python","patch_set":4,"id":"9baab087_6559a13a","line":823,"updated":"2022-07-05 13:48:27.000000000","message":"i think this is the begining of the connection plumbing","commit_id":"95b0e6418ec358983d4619174f483731aa741f83"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"952c19be306e3afe53a8858011e9382007b19557","unresolved":true,"context_lines":[{"line_number":913,"context_line":"        #         (without an epoch in the name)."},{"line_number":914,"context_line":"        #       - In this case we should just look at the local object table"},{"line_number":915,"context_line":"        #         for stats."},{"line_number":916,"context_line":"        #  2. SHARDED: There are 2 databases, the old expiring database and"},{"line_number":917,"context_line":"        #         a fresh new one."},{"line_number":918,"context_line":"        #       - Defer to the old retiring database for answers."},{"line_number":919,"context_line":"        #  3. SHARDED: There is only a sharded container, one with an epoch."}],"source_content_type":"text/x-python","patch_set":4,"id":"b9e386ae_9fdd7c18","line":916,"updated":"2022-07-05 13:48:27.000000000","message":"we might call this state \u0027SHARDED (initiated)\u0027 or something","commit_id":"95b0e6418ec358983d4619174f483731aa741f83"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"c49674e462808da0793631e52fd277c4551d460c","unresolved":true,"context_lines":[{"line_number":930,"context_line":"        # have shard metadata so defer to it."},{"line_number":931,"context_line":"        state \u003d self.get_db_state()"},{"line_number":932,"context_line":"        osr \u003d self._own_shard_range(no_default\u003dTrue, connection\u003dconnection)"},{"line_number":933,"context_line":"        if state \u003d\u003d UNSHARDED and osr and osr.state \u003d\u003d ShardRange.SHARDED:"},{"line_number":934,"context_line":"            stats \u003d self.get_shard_usage()"},{"line_number":935,"context_line":"        elif state \u003d\u003d SHARDING:"},{"line_number":936,"context_line":"            other_info \u003d self.get_brokers()[0]._get_info()"}],"source_content_type":"text/x-python","patch_set":4,"id":"e8cbc2a0_788b78e9","line":933,"updated":"2022-07-05 18:56:57.000000000","message":"UNSHARDED from get_db_state is something like:\n\nonly one db, and it doesn\u0027t have an epoch\n\n... but if the osr is sharded by another node, and has already shipped us shard ranges... this is the state we\u0027re interested in, sharding-initiated-but-unstarted - THAT state, OR \"SHARDED\" state should return the extra shard_usage for the purpose of answering \"is_deleted\" at a minimum","commit_id":"95b0e6418ec358983d4619174f483731aa741f83"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"952c19be306e3afe53a8858011e9382007b19557","unresolved":true,"context_lines":[{"line_number":931,"context_line":"        state \u003d self.get_db_state()"},{"line_number":932,"context_line":"        osr \u003d self._own_shard_range(no_default\u003dTrue, connection\u003dconnection)"},{"line_number":933,"context_line":"        if state \u003d\u003d UNSHARDED and osr and osr.state \u003d\u003d ShardRange.SHARDED:"},{"line_number":934,"context_line":"            stats \u003d self.get_shard_usage()"},{"line_number":935,"context_line":"        elif state \u003d\u003d SHARDING:"},{"line_number":936,"context_line":"            other_info \u003d self.get_brokers()[0]._get_info()"},{"line_number":937,"context_line":"            stats \u003d {\u0027object_count\u0027: other_info[\u0027object_count\u0027],"}],"source_content_type":"text/x-python","patch_set":4,"id":"8a67be65_7f5aa89f","line":934,"updated":"2022-07-05 13:48:27.000000000","message":"why not go ahead and plumb connection re-use through here too?  next call is get_shard_ranges\n\nwhy don\u0027t we need to check is_root_container?","commit_id":"95b0e6418ec358983d4619174f483731aa741f83"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"952c19be306e3afe53a8858011e9382007b19557","unresolved":true,"context_lines":[{"line_number":1866,"context_line":"            for row in self._get_shard_range_rows("},{"line_number":1867,"context_line":"                includes\u003dincludes, include_deleted\u003dinclude_deleted,"},{"line_number":1868,"context_line":"                states\u003dstates, include_own\u003dinclude_own,"},{"line_number":1869,"context_line":"                exclude_others\u003dexclude_others, connection\u003dconnection)]"},{"line_number":1870,"context_line":""},{"line_number":1871,"context_line":"        shard_ranges.sort(key\u003dShardRange.sort_key)"},{"line_number":1872,"context_line":"        if includes:"}],"source_content_type":"text/x-python","patch_set":4,"id":"9481f73f_be74bed3","line":1869,"updated":"2022-07-05 13:48:27.000000000","message":"I think this is the bottom of the connection re-use plumbing\n\n_get_shard_range_rows always accepted a connection kwarg and put it through swift.common.db.DatabaseBroker.maybe_get","commit_id":"95b0e6418ec358983d4619174f483731aa741f83"}],"swift/container/sharder.py":[{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"69f69130babf2a57a55f79a4c61c956ca0c96679","unresolved":true,"context_lines":[{"line_number":2041,"context_line":"        self._move_misplaced_objects(broker)"},{"line_number":2042,"context_line":""},{"line_number":2043,"context_line":"        has_shard_objects \u003d broker.get_shard_usage()[\u0027object_count\u0027] \u003e 0"},{"line_number":2044,"context_line":"        if broker.is_deleted() and not has_shard_objects:"},{"line_number":2045,"context_line":"            # This container is deleted so we can skip it. We still want"},{"line_number":2046,"context_line":"            # deleted containers to go via misplaced items because they may"},{"line_number":2047,"context_line":"            # have new objects sitting in them that may need to move."}],"source_content_type":"text/x-python","patch_set":1,"id":"33c1621a_170e40fc","line":2044,"updated":"2022-06-03 07:38:21.000000000","message":"I do wonder in retrospect, if we need to take into account some kind of:\n\n  state \u003d\u003d UNSHARDED and broker.get_shard_ranges()\n\ntoo? Or will a unsharded root with shardranges, but not have any objects be ok.. maybe they will.. just a thought while I was cooking. Haven\u0027t thought it out but wanted to put it somewhere.","commit_id":"21ed20578a017a122dc7c0638e9d43883c641e6f"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"5ab35315914eb131e601bdd112b11a01f6c2cac8","unresolved":true,"context_lines":[{"line_number":2041,"context_line":"        self._move_misplaced_objects(broker)"},{"line_number":2042,"context_line":""},{"line_number":2043,"context_line":"        has_shard_objects \u003d broker.get_shard_usage()[\u0027object_count\u0027] \u003e 0"},{"line_number":2044,"context_line":"        if broker.is_deleted() and not has_shard_objects:"},{"line_number":2045,"context_line":"            # This container is deleted so we can skip it. We still want"},{"line_number":2046,"context_line":"            # deleted containers to go via misplaced items because they may"},{"line_number":2047,"context_line":"            # have new objects sitting in them that may need to move."}],"source_content_type":"text/x-python","patch_set":1,"id":"e5d6dd4f_7efaa10c","line":2044,"in_reply_to":"33c1621a_170e40fc","updated":"2022-06-03 14:31:40.000000000","message":"I find it really questionable that a broker with shard ranges and objects can return True from is_deleted - that method should know better.","commit_id":"21ed20578a017a122dc7c0638e9d43883c641e6f"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"716b8e0ba71bca1aa43f9212d23e7ecd4d8e9329","unresolved":true,"context_lines":[{"line_number":2041,"context_line":"        self._move_misplaced_objects(broker)"},{"line_number":2042,"context_line":""},{"line_number":2043,"context_line":"        has_shard_objects \u003d broker.get_shard_usage()[\u0027object_count\u0027] \u003e 0"},{"line_number":2044,"context_line":"        if broker.is_deleted() and not has_shard_objects:"},{"line_number":2045,"context_line":"            # This container is deleted so we can skip it. We still want"},{"line_number":2046,"context_line":"            # deleted containers to go via misplaced items because they may"},{"line_number":2047,"context_line":"            # have new objects sitting in them that may need to move."}],"source_content_type":"text/x-python","patch_set":1,"id":"b906f903_110b7915","line":2044,"in_reply_to":"e5d6dd4f_7efaa10c","updated":"2022-06-06 08:12:38.000000000","message":"I\u0027m with Clay - unless I have forgotten a gotcha, it strikes me that if a broker with *sharding enabled* and shard ranges with non-zero object count should return is_deleted() \u003d\u003d False just like it would if it had object rows.\n\nIn fact, maybe there should be a unit test somewhere that asserts that brokers in different db_state, but otherwise consistent state, are consistent in their is_deleted() return values.\n\nThat said, we might want to be careful about changing when brokers start using shard ranges object counts for the broker info (_get_alternate_object_stats()) - while the db is UNSHARDED we might still want to use the db object count (as master does now).\n\nSo, IDK, maybe ContainerBroker._get_alternate_object_stats() should have a clause like\n\n  if state \u003d\u003d UNSHARDED and stats[\u0027object_count\u0027] \u003d\u003d 0:\n      stats \u003d self.get_shard_usage()","commit_id":"21ed20578a017a122dc7c0638e9d43883c641e6f"}],"test/probe/test_sharder.py":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"5ab35315914eb131e601bdd112b11a01f6c2cac8","unresolved":true,"context_lines":[{"line_number":3581,"context_line":"        # in order to simulate some rebalances and the unsharded winning the"},{"line_number":3582,"context_line":"        # replication rsync race. We should end up with 3x unsharded + deleted"},{"line_number":3583,"context_line":"        for broker in sharded_brokers:"},{"line_number":3584,"context_line":"            os.remove(broker.db_file)"},{"line_number":3585,"context_line":""},{"line_number":3586,"context_line":"        # now run the replicator to push out the unsharded node"},{"line_number":3587,"context_line":"        self.replicators.once()"}],"source_content_type":"text/x-python","patch_set":1,"id":"a4c17ff1_95c85a5e","line":3584,"updated":"2022-06-03 14:31:40.000000000","message":"this is *incredible* - so not only is the \"fix\" as simple as just letting the sharder process the \"deleted\" broker.  That \"unsharded\" broker has enough state replicated from the other nodes to completly heal the system to a consistent state if we let it.\n\nI think this proves it\u0027s fundementally wrong for that \"deleted\" database to say its \"unsharded\" once it\u0027s recieved the sharding state from it\u0027s peers.","commit_id":"21ed20578a017a122dc7c0638e9d43883c641e6f"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"7e659b5fc31e7782fede584675ba7e030d1edb61","unresolved":true,"context_lines":[{"line_number":3581,"context_line":"        # in order to simulate some rebalances and the unsharded winning the"},{"line_number":3582,"context_line":"        # replication rsync race. We should end up with 3x unsharded + deleted"},{"line_number":3583,"context_line":"        for broker in sharded_brokers:"},{"line_number":3584,"context_line":"            os.remove(broker.db_file)"},{"line_number":3585,"context_line":""},{"line_number":3586,"context_line":"        # now run the replicator to push out the unsharded node"},{"line_number":3587,"context_line":"        self.replicators.once()"}],"source_content_type":"text/x-python","patch_set":1,"id":"f59a0262_1ab8386e","line":3584,"in_reply_to":"a4c17ff1_95c85a5e","updated":"2022-06-13 16:31:16.000000000","message":"it seems maybe it IS at least \"sharding_initiated\" which should be a subset of \"sharding\" which is definately NOT \"unsharded\" (so not-un means *is* ... SHARDED).  QED","commit_id":"21ed20578a017a122dc7c0638e9d43883c641e6f"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"294138416f7052f61d57c1e29ae70bae85354543","unresolved":true,"context_lines":[{"line_number":3581,"context_line":"        # in order to simulate some rebalances and the unsharded winning the"},{"line_number":3582,"context_line":"        # replication rsync race. We should end up with 3x unsharded + deleted"},{"line_number":3583,"context_line":"        for broker in sharded_brokers:"},{"line_number":3584,"context_line":"            os.remove(broker.db_file)"},{"line_number":3585,"context_line":""},{"line_number":3586,"context_line":"        # now run the replicator to push out the unsharded node"},{"line_number":3587,"context_line":"        self.replicators.once()"}],"source_content_type":"text/x-python","patch_set":1,"id":"55212029_2677e73a","line":3584,"in_reply_to":"f59a0262_1ab8386e","updated":"2022-06-14 07:22:35.000000000","message":"Not sure this probetest is working with the new code yet. If the new code works the replica with shards wont be \"deleted\" anymore if we have shards with objects. So will need to adjust.","commit_id":"21ed20578a017a122dc7c0638e9d43883c641e6f"}]}