)]}'
{"swift/container/sharder.py":[{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"5935f8c46a7abec418c3653c2184939188f31239","unresolved":false,"context_lines":[{"line_number":1144,"context_line":"            objects \u003d []"},{"line_number":1145,"context_line":"            for record in new_items:"},{"line_number":1146,"context_line":"                item \u003d broker._record_to_dict(record)"},{"line_number":1147,"context_line":"                item[\u0027storage_policy_index\u0027] \u003d broker.storage_policy_index"},{"line_number":1148,"context_line":"                objects.append(item)"},{"line_number":1149,"context_line":"            broker_to_update.merge_items(objects)"},{"line_number":1150,"context_line":""}],"source_content_type":"text/x-python","patch_set":1,"id":"7f515b1d_7ed635c6","line":1147,"updated":"2017-10-04 17:01:45.000000000","message":"_record_to_dict already looks at storage_policy_index, doesn\u0027t it? Does this actually do anything? If it does, should it? This seems like a way to change recorded storage policies without actually moving data :-/ but maybe we only ever call this with a qry specifying the broker\u0027s storage policy?","commit_id":"bf4f4effa9d45d346a8af5ca2b5a4aee2e650284"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"0bd389d19654056434643ae468a2a07801818388","unresolved":false,"context_lines":[{"line_number":1144,"context_line":"            objects \u003d []"},{"line_number":1145,"context_line":"            for record in new_items:"},{"line_number":1146,"context_line":"                item \u003d broker._record_to_dict(record)"},{"line_number":1147,"context_line":"                item[\u0027storage_policy_index\u0027] \u003d broker.storage_policy_index"},{"line_number":1148,"context_line":"                objects.append(item)"},{"line_number":1149,"context_line":"            broker_to_update.merge_items(objects)"},{"line_number":1150,"context_line":""}],"source_content_type":"text/x-python","patch_set":1,"id":"7f515b1d_1b9aa86d","line":1147,"in_reply_to":"7f515b1d_7ed635c6","updated":"2017-10-04 21:58:58.000000000","message":"IIRC the query in list_objects_iter does not select storage policy index, so depsite record_to_dict attempting to zip that key the dict we get back doesn\u0027t have the policy index.\n\nThat said, the next patch in this chain https://review.openstack.org/#/c/445160 introduces get_objects which we should use here too, I missed it but will fix that.\n\nAs for whether we should be forcing the policy index - good question, I will revisit that","commit_id":"bf4f4effa9d45d346a8af5ca2b5a4aee2e650284"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"5935f8c46a7abec418c3653c2184939188f31239","unresolved":false,"context_lines":[{"line_number":1294,"context_line":"            self._sharding_complete(root_account, root_container, broker)"},{"line_number":1295,"context_line":"            self.logger.increment(\u0027sharding_complete\u0027)"},{"line_number":1296,"context_line":""},{"line_number":1297,"context_line":"    # TODO: unused method - ok to delete?"},{"line_number":1298,"context_line":"    # def _push_shard_ranges_to_container(self, root_account,"},{"line_number":1299,"context_line":"    #                                     root_container, shard_range,"},{"line_number":1300,"context_line":"    #                                     storage_policy_index):"}],"source_content_type":"text/x-python","patch_set":1,"id":"7f515b1d_7e72b5e5","line":1297,"updated":"2017-10-04 17:01:45.000000000","message":"Coward :P","commit_id":"bf4f4effa9d45d346a8af5ca2b5a4aee2e650284"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"0bd389d19654056434643ae468a2a07801818388","unresolved":false,"context_lines":[{"line_number":1294,"context_line":"            self._sharding_complete(root_account, root_container, broker)"},{"line_number":1295,"context_line":"            self.logger.increment(\u0027sharding_complete\u0027)"},{"line_number":1296,"context_line":""},{"line_number":1297,"context_line":"    # TODO: unused method - ok to delete?"},{"line_number":1298,"context_line":"    # def _push_shard_ranges_to_container(self, root_account,"},{"line_number":1299,"context_line":"    #                                     root_container, shard_range,"},{"line_number":1300,"context_line":"    #                                     storage_policy_index):"}],"source_content_type":"text/x-python","patch_set":1,"id":"7f515b1d_9b87b810","line":1297,"in_reply_to":"7f515b1d_7e72b5e5","updated":"2017-10-04 21:58:58.000000000","message":"yeah I even needed to reformat it :O","commit_id":"bf4f4effa9d45d346a8af5ca2b5a4aee2e650284"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"cc2b60e0111a4715258bc02e9bbbcaa295caa13a","unresolved":false,"context_lines":[{"line_number":1294,"context_line":"            self._sharding_complete(root_account, root_container, broker)"},{"line_number":1295,"context_line":"            self.logger.increment(\u0027sharding_complete\u0027)"},{"line_number":1296,"context_line":""},{"line_number":1297,"context_line":"    # TODO: unused method - ok to delete?"},{"line_number":1298,"context_line":"    # def _push_shard_ranges_to_container(self, root_account,"},{"line_number":1299,"context_line":"    #                                     root_container, shard_range,"},{"line_number":1300,"context_line":"    #                                     storage_policy_index):"}],"source_content_type":"text/x-python","patch_set":1,"id":"7f515b1d_0ce9497f","line":1297,"in_reply_to":"7f515b1d_9b87b810","updated":"2017-10-05 02:55:03.000000000","message":"Ahh the push_shard_ranges_to_container.. let me tell you story.\n\nIn the current version of the code, we pass shard ranges via the container PUT api. We should be doing this in 2 or 3 places:\n\n  - when we\u0027re updating the root container after cleaving\n  - When we\u0027re deleting a range after shrinking/melding and\n  - after the sharder visits a shard and wants to send that latest obj count and bytes used to the root container.\n\nBecause we are using container PUT, it means these updates end up in .pending files until they get merged into the database.. which I think is fine, so long as we can deal with a potential lag.\n\nThis function is older then the container PUT api. It would use the same cleaving mechanism (create a broker in a handoff location and replicate it out). The trick is, it would create a root container on a handoff node and only add the changed shard_ranges (to update counts, delete a range or new ranges) and then use replication.\n\nthis would bypass the pending file and get them straight into the db, so they got there faster. The trick would be (now that we have the sharding state) to only ever send this via usync which will always push all shard_ranges. If you used rsync_then_merge, I think the rsynced container to the /tmp location (in this case the small one) would become the new container on that node and _all_ the records of the old root (one with more data) would push it\u0027s rows into it... this worries me.\n\nSo it\u0027s a dinosaur from pre container PUT shard ranges era. I only kept the old method around in case the pending lag became an issue.","commit_id":"bf4f4effa9d45d346a8af5ca2b5a4aee2e650284"}]}