)]}'
{"/COMMIT_MSG":[{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"5aec13a5357b597fa83765b72c5fd3098240f3ef","unresolved":false,"context_lines":[{"line_number":6,"context_line":""},{"line_number":7,"context_line":"Shard range updates should not revive deleted container"},{"line_number":8,"context_line":""},{"line_number":9,"context_line":"Once a root container is deleted an update from a shard container"},{"line_number":10,"context_line":"shoudd not revive it.  Also, shard range updates should not bump the"},{"line_number":11,"context_line":"container put timestamp that is reported to a client.  However, shard"},{"line_number":12,"context_line":"range PUTs to a shard container should autocreate it."},{"line_number":13,"context_line":""}],"source_content_type":"text/x-gerrit-commit-message","patch_set":2,"id":"bf659307_eff4d7d1","line":10,"range":{"start_line":9,"start_character":0,"end_line":10,"end_character":21},"updated":"2018-04-06 16:51:56.000000000","message":"Well....\n\nMaybe. Or maybe it should -- what if the shard range actually has live object rows? The trouble I was seeing was limited to shards that were reporting back to the root, \"Hey, I\u0027m empty!\"\n\nIf we *never* let shard updates revive a container, how will we do something like https://github.com/openstack/swift/blob/2.17.0/test/probe/test_object_async_update.py#L110-L121 when sharding? An object update is sitting in async-pending somewhere, directed at a shard that\u0027s currently empty... shard\u0027s reported the fact that it\u0027s empty up to the root, who looks over *all* shards and thinks it\u0027s empty... DELETE comes in, we 204, but what happens when that object-updater finally runs?\n\nI actually wonder if we should have DELETE *require* that the only (live) shard range in the root is its own -- otherwise, 409.\n\nThis would mean that anyone doing manual sharding will need to set DBs to shrinking though (which we don\u0027t currently have tooling for) -- or just live with the fact that once you start sharding a container, it\u0027s never going away until we get automatic shrinking.","commit_id":"cb64f889bcfea05fbae9d09bc36066be4a726473"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"b23e7ccf15e75bdd269ce190a927b4451a8ac60c","unresolved":false,"context_lines":[{"line_number":6,"context_line":""},{"line_number":7,"context_line":"Shard range updates should not revive deleted container"},{"line_number":8,"context_line":""},{"line_number":9,"context_line":"Once a root container is deleted an update from a shard container"},{"line_number":10,"context_line":"shoudd not revive it.  Also, shard range updates should not bump the"},{"line_number":11,"context_line":"container put timestamp that is reported to a client.  However, shard"},{"line_number":12,"context_line":"range PUTs to a shard container should autocreate it."},{"line_number":13,"context_line":""}],"source_content_type":"text/x-gerrit-commit-message","patch_set":2,"id":"bf659307_8f2d1baa","line":10,"range":{"start_line":9,"start_character":0,"end_line":10,"end_character":21},"in_reply_to":"bf659307_eff4d7d1","updated":"2018-04-06 17:15:36.000000000","message":"bad commit message - shard range updates are still accepted by a deleted container and if they have non-zero object count then the container is revived. Just like object updates.","commit_id":"cb64f889bcfea05fbae9d09bc36066be4a726473"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"0fd6124ea8fdd87e600efd1fd48e40c997af6c45","unresolved":false,"context_lines":[{"line_number":11,"context_line":"Before, *any* shard range update received by a deleted container would"},{"line_number":12,"context_line":"bump its put timestamp and revive it. Now, shard range updates do not"},{"line_number":13,"context_line":"bump the put timestamp, but are still accepted by a deleted"},{"line_number":14,"context_line":"container. If the shard range update has non-zero object count then"},{"line_number":15,"context_line":"the container will be revived, just as if an object update had been"},{"line_number":16,"context_line":"received after the delete timestamp. Otherwise the container will"},{"line_number":17,"context_line":"remain deleted."},{"line_number":18,"context_line":""}],"source_content_type":"text/x-gerrit-commit-message","patch_set":3,"id":"bf659307_a9a45409","line":15,"range":{"start_line":14,"start_character":11,"end_line":15,"end_character":29},"updated":"2018-04-10 19:32:37.000000000","message":"Does it? I didn\u0027t notice any looking at object_count in _maybe_autocreate...\n\nOh, I guess it must happen down in merge_shard_ranges.","commit_id":"d6928552a061416fbfccd1e593f9e41458212104"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"f136ed3b920f0c21e871509b23156b36889b33e0","unresolved":false,"context_lines":[{"line_number":11,"context_line":"Before, *any* shard range update received by a deleted container would"},{"line_number":12,"context_line":"bump its put timestamp and revive it. Now, shard range updates do not"},{"line_number":13,"context_line":"bump the put timestamp, but are still accepted by a deleted"},{"line_number":14,"context_line":"container. If the shard range update has non-zero object count then"},{"line_number":15,"context_line":"the container will be revived, just as if an object update had been"},{"line_number":16,"context_line":"received after the delete timestamp. Otherwise the container will"},{"line_number":17,"context_line":"remain deleted."},{"line_number":18,"context_line":""}],"source_content_type":"text/x-gerrit-commit-message","patch_set":3,"id":"bf659307_d40a691e","line":15,"range":{"start_line":14,"start_character":11,"end_line":15,"end_character":29},"in_reply_to":"bf659307_a9a45409","updated":"2018-04-11 10:03:24.000000000","message":"The container is revived as a result of having non-zero object count or shard object count\n\nsee https://review.openstack.org/#/c/559324/3/test/unit/container/test_server.py@2170","commit_id":"d6928552a061416fbfccd1e593f9e41458212104"}],"swift/container/server.py":[{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"0fd6124ea8fdd87e600efd1fd48e40c997af6c45","unresolved":false,"context_lines":[{"line_number":460,"context_line":"            except (ValueError, KeyError) as err:"},{"line_number":461,"context_line":"                return HTTPBadRequest(\u0027Invalid body: %r\u0027 % err)"},{"line_number":462,"context_line":"            created \u003d self._maybe_autocreate(broker, req_timestamp, account,"},{"line_number":463,"context_line":"                                             requested_policy_index)"},{"line_number":464,"context_line":"            self._update_metadata(req, broker, req_timestamp, \u0027PUT\u0027)"},{"line_number":465,"context_line":"            if shard_ranges:"},{"line_number":466,"context_line":"                # TODO: consider writing the shard ranges into the pending"}],"source_content_type":"text/x-python","patch_set":3,"id":"bf659307_092b0082","line":463,"range":{"start_line":463,"start_character":45,"end_line":463,"end_character":67},"updated":"2018-04-10 19:32:37.000000000","message":"Huh... yeah, I suppose you *could* include X-Storage-Policy when PUTting shards... hmm....","commit_id":"d6928552a061416fbfccd1e593f9e41458212104"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"0fd6124ea8fdd87e600efd1fd48e40c997af6c45","unresolved":false,"context_lines":[{"line_number":462,"context_line":"            created \u003d self._maybe_autocreate(broker, req_timestamp, account,"},{"line_number":463,"context_line":"                                             requested_policy_index)"},{"line_number":464,"context_line":"            self._update_metadata(req, broker, req_timestamp, \u0027PUT\u0027)"},{"line_number":465,"context_line":"            if shard_ranges:"},{"line_number":466,"context_line":"                # TODO: consider writing the shard ranges into the pending"},{"line_number":467,"context_line":"                # file, but if so ensure an all-or-none semantic for the write"},{"line_number":468,"context_line":"                broker.merge_shard_ranges(shard_ranges)"}],"source_content_type":"text/x-python","patch_set":3,"id":"bf659307_49457842","line":465,"range":{"start_line":465,"start_character":12,"end_line":465,"end_character":27},"updated":"2018-04-10 19:32:37.000000000","message":"I kinda feel like including x-backend-record-type: shard and not providing any shards ought to be a 400, before we even get to updating metadata...","commit_id":"d6928552a061416fbfccd1e593f9e41458212104"}],"test/probe/test_sharder.py":[{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"0fd6124ea8fdd87e600efd1fd48e40c997af6c45","unresolved":false,"context_lines":[{"line_number":1140,"context_line":"            # object, targeted at the now retired first shard..."},{"line_number":1141,"context_line":"            self.updaters.once()"},{"line_number":1142,"context_line":"            # and the container is revived!"},{"line_number":1143,"context_line":"            self.assert_container_listing([beta])"},{"line_number":1144,"context_line":""},{"line_number":1145,"context_line":"            # finally, clear out the container"},{"line_number":1146,"context_line":"            client.delete_object("}],"source_content_type":"text/x-python","patch_set":3,"id":"bf659307_29c92477","line":1143,"updated":"2018-04-10 19:32:37.000000000","message":"So the updater gets an object row into the deleted shard... but has the sharder sent that stat back to the root? When?","commit_id":"d6928552a061416fbfccd1e593f9e41458212104"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"f136ed3b920f0c21e871509b23156b36889b33e0","unresolved":false,"context_lines":[{"line_number":1140,"context_line":"            # object, targeted at the now retired first shard..."},{"line_number":1141,"context_line":"            self.updaters.once()"},{"line_number":1142,"context_line":"            # and the container is revived!"},{"line_number":1143,"context_line":"            self.assert_container_listing([beta])"},{"line_number":1144,"context_line":""},{"line_number":1145,"context_line":"            # finally, clear out the container"},{"line_number":1146,"context_line":"            client.delete_object("}],"source_content_type":"text/x-python","patch_set":3,"id":"bf659307_74f91dc6","line":1143,"in_reply_to":"bf659307_29c92477","updated":"2018-04-11 10:03:24.000000000","message":"the comment at line 1131 is misleading. See line 1005 - the beta object in async pending is targeted at the root. I\u0027d love it to be targeted at the shard but we don\u0027t have test infrastructure to facilitate that. See https://trello.com/c/QVz5OTp7","commit_id":"d6928552a061416fbfccd1e593f9e41458212104"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"6298bfadebeb7887491930ee2c6af10593f3c2bc","unresolved":false,"context_lines":[{"line_number":1154,"context_line":"            self.direct_head_container(expect_failure\u003dTrue)"},{"line_number":1155,"context_line":""},{"line_number":1156,"context_line":"            # now run updaters to deal with the async pending for the beta"},{"line_number":1157,"context_line":"            # object, targeted at the now retired first shard..."},{"line_number":1158,"context_line":"            self.updaters.once()"},{"line_number":1159,"context_line":"            # and the container is revived!"},{"line_number":1160,"context_line":"            self.assert_container_listing([beta])"}],"source_content_type":"text/x-python","patch_set":5,"id":"bf659307_0c950e40","side":"PARENT","line":1157,"updated":"2018-04-12 17:48:36.000000000","message":"Thanks","commit_id":"45f81796e11165e62349c0fe71ece9ad0148e4f7"}],"test/unit/container/test_server.py":[{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"f136ed3b920f0c21e871509b23156b36889b33e0","unresolved":false,"context_lines":[{"line_number":2167,"context_line":"        resp \u003d req.get_response(self.controller)"},{"line_number":2168,"context_line":"        self.assertEqual(404, resp.status_int)"},{"line_number":2169,"context_line":""},{"line_number":2170,"context_line":"        # unless shard range has non-zero stats, then container is revived"},{"line_number":2171,"context_line":"        shard_range.update_meta(99, 1234, meta_timestamp\u003dnext(ts_iter))"},{"line_number":2172,"context_line":"        req \u003d Request.blank(\u0027/sda1/p/a/c\u0027, method\u003d\u0027PUT\u0027, headers\u003dheaders,"},{"line_number":2173,"context_line":"                            body\u003djson.dumps([dict(shard_range)]))"},{"line_number":2174,"context_line":"        resp \u003d req.get_response(self.controller)"},{"line_number":2175,"context_line":"        self.assertEqual(202, resp.status_int)"},{"line_number":2176,"context_line":"        self.assertFalse(broker.get_info_is_deleted()[1])"},{"line_number":2177,"context_line":"        self.assertEqual(delete_time, broker.get_info()[\u0027delete_timestamp\u0027])"},{"line_number":2178,"context_line":"        self.assertEqual(put_time, broker.get_info()[\u0027put_timestamp\u0027])"},{"line_number":2179,"context_line":"        req \u003d Request.blank(\u0027/sda1/p/a/c\u0027, method\u003d\u0027GET\u0027,"},{"line_number":2180,"context_line":"                            headers\u003d{\u0027X-Timestamp\u0027: put_time})"},{"line_number":2181,"context_line":"        resp \u003d req.get_response(self.controller)"},{"line_number":2182,"context_line":"        self.assertEqual(204, resp.status_int)"},{"line_number":2183,"context_line":"        self.assertEqual(\u002799\u0027, resp.headers[\u0027X-Container-Object-Count\u0027])"},{"line_number":2184,"context_line":""},{"line_number":2185,"context_line":"    def test_PUT_shard_range_json_in_body(self):"},{"line_number":2186,"context_line":"        ts_iter \u003d make_timestamp_iter()"}],"source_content_type":"text/x-python","patch_set":3,"id":"bf659307_d93ce4e2","line":2183,"range":{"start_line":2170,"start_character":8,"end_line":2183,"end_character":72},"updated":"2018-04-11 10:03:24.000000000","message":"the container is \u0027revived\u0027 because it now has non-zero shard usage","commit_id":"d6928552a061416fbfccd1e593f9e41458212104"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"6298bfadebeb7887491930ee2c6af10593f3c2bc","unresolved":false,"context_lines":[{"line_number":2167,"context_line":"        resp \u003d req.get_response(self.controller)"},{"line_number":2168,"context_line":"        self.assertEqual(404, resp.status_int)"},{"line_number":2169,"context_line":""},{"line_number":2170,"context_line":"        # unless shard range has non-zero stats, then container is revived"},{"line_number":2171,"context_line":"        shard_range.update_meta(99, 1234, meta_timestamp\u003dnext(ts_iter))"},{"line_number":2172,"context_line":"        req \u003d Request.blank(\u0027/sda1/p/a/c\u0027, method\u003d\u0027PUT\u0027, headers\u003dheaders,"},{"line_number":2173,"context_line":"                            body\u003djson.dumps([dict(shard_range)]))"},{"line_number":2174,"context_line":"        resp \u003d req.get_response(self.controller)"},{"line_number":2175,"context_line":"        self.assertEqual(202, resp.status_int)"},{"line_number":2176,"context_line":"        self.assertFalse(broker.get_info_is_deleted()[1])"},{"line_number":2177,"context_line":"        self.assertEqual(delete_time, broker.get_info()[\u0027delete_timestamp\u0027])"},{"line_number":2178,"context_line":"        self.assertEqual(put_time, broker.get_info()[\u0027put_timestamp\u0027])"},{"line_number":2179,"context_line":"        req \u003d Request.blank(\u0027/sda1/p/a/c\u0027, method\u003d\u0027GET\u0027,"},{"line_number":2180,"context_line":"                            headers\u003d{\u0027X-Timestamp\u0027: put_time})"},{"line_number":2181,"context_line":"        resp \u003d req.get_response(self.controller)"},{"line_number":2182,"context_line":"        self.assertEqual(204, resp.status_int)"},{"line_number":2183,"context_line":"        self.assertEqual(\u002799\u0027, resp.headers[\u0027X-Container-Object-Count\u0027])"},{"line_number":2184,"context_line":""},{"line_number":2185,"context_line":"    def test_PUT_shard_range_json_in_body(self):"},{"line_number":2186,"context_line":"        ts_iter \u003d make_timestamp_iter()"}],"source_content_type":"text/x-python","patch_set":3,"id":"bf659307_ecec5abc","line":2183,"range":{"start_line":2170,"start_character":8,"end_line":2183,"end_character":72},"in_reply_to":"bf659307_d93ce4e2","updated":"2018-04-12 17:48:36.000000000","message":"OK, and that \"liveness\" is only determined in the GET path -- I was kinda expecting some switch to get flipped during the PUT but it makes sense that we still have the same delete timestamp and everything (in case that shard really was supposed to go to zero, and just had some async DELETEs that hadn\u0027t registered).","commit_id":"d6928552a061416fbfccd1e593f9e41458212104"}]}