)]}'
{"/COMMIT_MSG":[{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"c802c485abf0a70ccedcf3fe2f8e77ab3517fdc2","unresolved":true,"context_lines":[{"line_number":12,"context_line":"and name) of the shard ranges into Memcache and reconstruct the list"},{"line_number":13,"context_line":"of Namespaces based on those attributes. Thus, a full namespaces GET"},{"line_number":14,"context_line":"interface can be added into the backend container-server to only"},{"line_number":15,"context_line":"return a list of those Namespace attributes."},{"line_number":16,"context_line":""},{"line_number":17,"context_line":"On a container server setup which serves a container with ~12000"},{"line_number":18,"context_line":"shard ranges, benchmarking results show that the request rate of"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":7,"id":"ab8841b4_5413263f","line":15,"updated":"2023-08-18 14:45:34.000000000","message":"I think we should stick to having only two types of data structures: Namespace and ShardRange. For each, there is a dict representation that can be obtained by e.g. dict(Namespace(\u0027name\u0027, \u0027l\u0027, \u0027u\u0027).\n\nIIUC this patch is introducing a third data struct (which is referred to as namespace) that only has \u0027name\u0027 and \u0027lower\u0027 attributes. While that does reduce the GET response body, it makes the API confusing: GET for record-type namespace does not return sufficient data to construct a list of Namespaces. \n\nYes, the proxy throws away the upper before caching namespaces, but that is a proxy concern. I\u0027d prefer the container server API to return a serialisation of one of our existing data types.\n\nAs it is, the new API is only ever going to be useful for getting the complete list of namespaces. If we ever want to add marker, end marker capability then we\u0027ll need to return (name, lower, upper) and this API won\u0027t support that.","commit_id":"1ff63be3bd812db97b3a319444346f5702457f59"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"d7efa6e02d77ec38b860851868e2da2de3d45bb8","unresolved":true,"context_lines":[{"line_number":12,"context_line":"and name) of the shard ranges into Memcache and reconstruct the list"},{"line_number":13,"context_line":"of Namespaces based on those attributes. Thus, a full namespaces GET"},{"line_number":14,"context_line":"interface can be added into the backend container-server to only"},{"line_number":15,"context_line":"return a list of those Namespace attributes."},{"line_number":16,"context_line":""},{"line_number":17,"context_line":"On a container server setup which serves a container with ~12000"},{"line_number":18,"context_line":"shard ranges, benchmarking results show that the request rate of"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":7,"id":"e239f735_853d0e82","line":15,"in_reply_to":"ab8841b4_5413263f","updated":"2023-08-24 12:22:29.000000000","message":"On reflection (and reminded by Jianjian) - we do have a NamespaceBoundsList data structure that does serialize as a list of dicts with just \u0027name\u0027 and \u0027lower\u0027. And this is the data structure that a proxy server wants to cache, so I\u0027m a little more comfortable with the principle of a more compact response body, but would prefer to make it obvious that it is a list of all namespaces by using, for example, x-backend-record-type\u003dnamespace_list.\n\nHowever, for the API to make sense for *any* container (not just root containers), we do need to represent the outer bounds of the list.\n\nOr, we make it a precondition of the namespace-list record type that the container must be a root??? That may seem an odd decision when we look back, because it\u0027s driven purely by the desire to save bytes on the wire than the capability of the container resource.","commit_id":"1ff63be3bd812db97b3a319444346f5702457f59"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"dba5cdc4d8314b4dc7c21ec88d42d98beac2cfbf","unresolved":false,"context_lines":[{"line_number":12,"context_line":"and name) of the shard ranges into Memcache and reconstruct the list"},{"line_number":13,"context_line":"of Namespaces based on those attributes. Thus, a full namespaces GET"},{"line_number":14,"context_line":"interface can be added into the backend container-server to only"},{"line_number":15,"context_line":"return a list of those Namespace attributes."},{"line_number":16,"context_line":""},{"line_number":17,"context_line":"On a container server setup which serves a container with ~12000"},{"line_number":18,"context_line":"shard ranges, benchmarking results show that the request rate of"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":7,"id":"33f6a7fa_1f63267b","line":15,"in_reply_to":"e239f735_853d0e82","updated":"2023-08-30 02:56:43.000000000","message":"For latest implementation, there is no performance difference anymore for returning NamespaceBoundsList or returning the complete Namespace. So latest patch switched to return Namespace per offline discussions.","commit_id":"1ff63be3bd812db97b3a319444346f5702457f59"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"74eaf120af694826c670d061515def674745fb63","unresolved":true,"context_lines":[{"line_number":9,"context_line":"The proxy-server makes GET requests to the container server to fetch"},{"line_number":10,"context_line":"full lists of shard ranges when handling object PUT/POST/DELETE and"},{"line_number":11,"context_line":"container GETs, then it only stores the Namespace attributes (lower"},{"line_number":12,"context_line":"and name) of the shard ranges into Memcache and reconstruct the list"},{"line_number":13,"context_line":"of Namespaces based on those attributes. Thus, a namespaces GET"},{"line_number":14,"context_line":"interface can be added into the backend container-server to only"},{"line_number":15,"context_line":"return a list of those Namespace attributes."}],"source_content_type":"text/x-gerrit-commit-message","patch_set":52,"id":"4d8fa296_89b226e8","line":12,"range":{"start_line":12,"start_character":48,"end_line":12,"end_character":59},"updated":"2024-01-05 15:04:39.000000000","message":"s/reconstruct/reconstructs/","commit_id":"b272276accd44edfa6aa557c0bd19e4e2e999dd4"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"b15761c0e0e2d7e69f93b97e5e7b7847078fdb4a","unresolved":false,"context_lines":[{"line_number":9,"context_line":"The proxy-server makes GET requests to the container server to fetch"},{"line_number":10,"context_line":"full lists of shard ranges when handling object PUT/POST/DELETE and"},{"line_number":11,"context_line":"container GETs, then it only stores the Namespace attributes (lower"},{"line_number":12,"context_line":"and name) of the shard ranges into Memcache and reconstruct the list"},{"line_number":13,"context_line":"of Namespaces based on those attributes. Thus, a namespaces GET"},{"line_number":14,"context_line":"interface can be added into the backend container-server to only"},{"line_number":15,"context_line":"return a list of those Namespace attributes."}],"source_content_type":"text/x-gerrit-commit-message","patch_set":52,"id":"30af390b_35a4fbf9","line":12,"range":{"start_line":12,"start_character":48,"end_line":12,"end_character":59},"in_reply_to":"4d8fa296_89b226e8","updated":"2024-01-09 15:23:00.000000000","message":"Done","commit_id":"b272276accd44edfa6aa557c0bd19e4e2e999dd4"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"dced6e75bce319a699b4e3a3353f1af5d40459bf","unresolved":false,"context_lines":[{"line_number":9,"context_line":"The proxy-server makes GET requests to the container server to fetch"},{"line_number":10,"context_line":"full lists of shard ranges when handling object PUT/POST/DELETE and"},{"line_number":11,"context_line":"container GETs, then it only stores the Namespace attributes (lower"},{"line_number":12,"context_line":"and name) of the shard ranges into Memcache and reconstruct the list"},{"line_number":13,"context_line":"of Namespaces based on those attributes. Thus, a namespaces GET"},{"line_number":14,"context_line":"interface can be added into the backend container-server to only"},{"line_number":15,"context_line":"return a list of those Namespace attributes."}],"source_content_type":"text/x-gerrit-commit-message","patch_set":52,"id":"fcce95de_dbc4c2ce","line":12,"range":{"start_line":12,"start_character":48,"end_line":12,"end_character":59},"in_reply_to":"4d8fa296_89b226e8","updated":"2024-01-06 01:13:39.000000000","message":"Done","commit_id":"b272276accd44edfa6aa557c0bd19e4e2e999dd4"}],"/PATCHSET_LEVEL":[{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"c03b18b305666b4f043dd6c764836f425e39f7fa","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":6,"id":"7e28126a_f5bb175d","updated":"2023-08-18 04:03:19.000000000","message":"thanks for the review and help!","commit_id":"a7aed733f5db82d9070a2c245a4bc7936fdbe218"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"f35d2431a428cfec4da64577f186cb5c684ba018","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":8,"id":"61b54892_628951aa","updated":"2023-08-24 07:12:15.000000000","message":"Looking really interesting JianJian, any chance you can share some of your inital benchmark numbers when comparing with the sort in python vs moving it to sqlite?","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"d7efa6e02d77ec38b860851868e2da2de3d45bb8","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":8,"id":"3fdb0ad3_64e47327","updated":"2023-08-24 12:22:29.000000000","message":"There\u0027s a couple of tactical issues to fix, but I\u0027m still not sure about the strategic choice of response data structure.\n\nI put some ideas in a follow up patch 892671: sq: suggestions for ContainerBroker.get_all_namespaces | https://review.opendev.org/c/openstack/swift/+/892671\n\n^^ that may or may not *all* be helpful, select as you see fit 😊","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"d9f2b72057b8621e8971a513b7a78e1d82a5689a","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":8,"id":"d03c91b2_e4610875","updated":"2023-08-25 04:21:08.000000000","message":"thanks for the reviews!","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"4e664f0e0766cb7f496d3aadc2f927ce29554976","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":8,"id":"d4650dd0_def5f895","updated":"2023-08-23 05:23:12.000000000","message":"thanks for the reviews. Patch still needs more test cases, but will be great to get consensus first on the new interface.","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"496427a785a2873bb0ace7d2b8fa6b9a2dbcb91a","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":8,"id":"9adb16e2_3e122db3","in_reply_to":"3fdb0ad3_64e47327","updated":"2023-08-25 05:42:55.000000000","message":"thanks for the fixes, the sq patch looks good to me. I will include it after we figure out the strategic choice of response data structure.","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"dba5cdc4d8314b4dc7c21ec88d42d98beac2cfbf","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":11,"id":"a6564115_5cf1c101","updated":"2023-08-30 02:56:43.000000000","message":"thanks a lot for the reviews!","commit_id":"884196289a3442ce0552c88e4f35ab41cffb4d1d"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"bf33de1f5f106065568e0985e20cc76fff8f97e0","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":12,"id":"b396a82a_d4e07dc2","updated":"2023-08-31 06:46:04.000000000","message":"This is awesome Jianjian. Love it. Just some questions/comments inline, and maybe we can add a few more test cases. Otherwise looks awesome","commit_id":"4638670d9e4dc6850b5447e2d5756be79b4c1aff"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"55553eee59e4c111d19a828d5b6ac7d371473253","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":12,"id":"525d9948_e6876b84","updated":"2023-08-30 21:48:32.000000000","message":"recheck","commit_id":"4638670d9e4dc6850b5447e2d5756be79b4c1aff"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"8b6d870b800f034be2c07a7e937e8ccda4da12db","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":12,"id":"6dcae8e2_22dfa6a5","updated":"2023-08-30 16:18:20.000000000","message":"recheck\ncentos7-probetest had one failure caused by random network issue \"ClientException: Object GET failed\"","commit_id":"4638670d9e4dc6850b5447e2d5756be79b4c1aff"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"fd158f484a16e6ddbe812130fa865b4ebe5e7a8b","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":14,"id":"b2b7ff5c_504eac36","updated":"2023-08-31 21:06:25.000000000","message":"thanks for the reviews!","commit_id":"1519f70a6adb43c9cc5dd2c80babb86ce6126ed3"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"c15c91b11bc946d098b3669e8b7532fa4a96d0e1","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":17,"id":"bd2fae67_2be6eed3","updated":"2023-09-05 04:59:38.000000000","message":"This is looking really good. I do wonder if an includes param would make sense? I know saying this so far along the process is a pain I know, and maybe we could do it as a follow up. But we use the includes option for obj proxy to get the updating shard_range. Which we probably use for warming the updating cache. But when it misses, we probaby don\u0027t want to get the whole list each time, and we don\u0027t want to keep them as shard_ranges.","commit_id":"6b34ca53368256d5f7109d8f184fe4a5fc6abf2b"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"3ae9f087bde3d7f7c2bb592b87773a81dbd5fe1c","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":17,"id":"9811a793_c2515a38","updated":"2023-09-05 14:53:40.000000000","message":"We need to get the upgrade strategy in place.\n\nThen we could restrict this change to initially only updating states, which still means adding the includes param.\n\nOr we need to ensure that all the proxy use cases are correctly handled i.e. add marker/end_marker for the listing states use case.","commit_id":"6b34ca53368256d5f7109d8f184fe4a5fc6abf2b"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"3919b5f331df82d39b944f6ec52506ac2ae65a31","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":17,"id":"5db2fdaa_ab00bf03","updated":"2023-09-06 00:57:44.000000000","message":"thanks for all the feedbacks!","commit_id":"6b34ca53368256d5f7109d8f184fe4a5fc6abf2b"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"b1653e9d53d095ff42423ad8cc33df36169edf94","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":19,"id":"3a248fa8_e75c6623","updated":"2023-09-07 22:15:39.000000000","message":"thanks for all the inputs!","commit_id":"0ed109b9dc319f5ec5b032c0bcb0b272b179b70c"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"e735d8c2861a8cd59022823cd8c9a3662f71e67d","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":19,"id":"c9bb41d7_3b8e50cf","updated":"2023-09-08 00:26:09.000000000","message":"this patch needs a respin, depending on who\u0027s writing the code I\u0027m sure there\u0027s a half-dozen different ways they could write it to meet requirements\n\nI like the idea of new backend header to specify the \"record format\" of shard records, I think the conflict of the term with \"encoding format\" in the is probably not great.  maybe we can find a synonym, \"schema\"?\n\nMaybe there\u0027s a future where we don\u0027t return json (currently container-servers really like json), and the current handling of format-qs/accept-header WRT shard ranges is suspect, but I suppose future me could justify \u0027x-backend-shard-record-format: namespaces2\u0027 having a different non-json encoding format (even if we had to continue sending format\u003djson for backwards compat)\n\nmaybe short term we could try to do some work *just* to make sure that new servers that respond to x-backend-shard-record-format: namespaces *always and only* return json (ignores format-qs and accept-header) then at least we have some precedent to excpect that a future `x-backend-shard-record-format: namespace_gzip` or whatever does what we expect and not something we don\u0027t need or want.","commit_id":"0ed109b9dc319f5ec5b032c0bcb0b272b179b70c"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"4621264a31586fe757b1846045e261e71f449a32","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":20,"id":"5cbffb38_8bc52dc5","updated":"2023-09-11 03:07:01.000000000","message":"thanks for the reviews!","commit_id":"5cd3b0f54d1debee30cc977fe4eb82441278400d"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"dd2266c879ea38eae3d5fd5fe501e62a1982679c","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":22,"id":"07022ae2_a52204e2","updated":"2023-09-13 19:57:42.000000000","message":"Looks to be fully backwards compatible and opt-in, with nothing opting in yet. I\u0027m willing to carry, but I\u0027m not clear on what we\u0027ll learn from carrying it without Matt\u0027s (currently-in-conflict) follow-up.","commit_id":"b105e043017904fcf520fd7cdad5c07aba47e76f"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"693fbccfb57938a9ccae311ba3d53701e72a9a9d","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":23,"id":"6119919d_2ced7c92","updated":"2023-09-21 04:37:28.000000000","message":"thanks for the reviews!","commit_id":"233b799167f971b45d57027c3c50d3caba6ba9e2"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"aef4a35e9f00ec228a3f60bfba95722c8d3e2bd6","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":25,"id":"f4477612_a175bd3d","updated":"2023-09-29 16:43:17.000000000","message":"I wanted to validate my claim that there was duplicate code so hacked together\nhttps://review.opendev.org/c/openstack/swift/+/896954 - take it or leave it 😊","commit_id":"11803c122b27001d9365fe3dc6686ce271bfa372"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"1e51ac4adf5517da06b2312f597ecdedcae202b3","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":27,"id":"595648a8_87cb33b0","updated":"2023-10-02 18:14:58.000000000","message":"The container server diff is now quite limited which is good - easy to track through where the upgraded container-server will deviate from legacy.","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":27,"id":"a255c0ca_82254c55","updated":"2023-10-02 22:23:29.000000000","message":"i think this is ok, maybe some tactical things that annoy me and some cruft we\u0027re carrying forward; I don\u0027t understand fill_gaps\u003dFalse support in get_namespaces","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"bcb52432e997cd207866f8b22a7c55a1f0a34204","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":28,"id":"4bf40365_45565e6a","updated":"2023-10-31 16:11:06.000000000","message":"Thanks a lot for the reviews!","commit_id":"e8052b259584a1587d53670bb9c800fcd4b912f3"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8a24e200d9cb30696c694717a3c20781843d2652","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":29,"id":"360333a5_ae0f17ed","updated":"2023-10-31 19:21:20.000000000","message":"Please consider not propogating the states\u003d\u003cint\u003e interface that\u0027s not actually used in the container server.  \n\nPlease consider breaking up the tests:\nhttps://review.opendev.org/c/openstack/swift/+/899744\n\nPlease consider if it makes sense to break-up the object/shard path from GET but then route them back together through \"create_listing\" - perhaps that method would make more sense as \"create_response\" and the GET_object and GET_shard methods should pass in their own update_[object|shard]_record listing with far fewer kwargs?\n\nFWIW I sympathize that all this shuffling around of method-dispatch/param-parsing could be a standalone pre-factor and it arguably orthogonal to actually adding the namespace format.  But I\u0027d counter that\u0027s just what happens when you want to add \"just one more behavior\" into old code that was already getting messy.","commit_id":"95dcd77bf6b77f83ab3471d40b0f658ee5c5014f"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"550053f673403ab48c7e04256a04ddb0cb8b157d","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":29,"id":"c1a4c999_b4675951","in_reply_to":"360333a5_ae0f17ed","updated":"2023-11-01 05:40:38.000000000","message":"Thanks for the refactoring patch, I got it squashed in. That’s all for today because it’s Halloween. I will work on the other comments tomorrow.","commit_id":"95dcd77bf6b77f83ab3471d40b0f658ee5c5014f"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"b263eb52a246427f54c4caf4fda57755aa96fa73","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":31,"id":"eab27244_955abd9c","updated":"2023-11-02 00:49:07.000000000","message":"Thanks everyone for the reviews!","commit_id":"9a03bb9447212dc1321393cbddca702272200bce"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"c52649a9d1d9fcf00c4b0e959afdc27c2366af8f","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":31,"id":"5474b431_ae3429fd","updated":"2023-11-01 18:52:06.000000000","message":"this function is going to be the gift that keeps on giving!","commit_id":"9a03bb9447212dc1321393cbddca702272200bce"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"a57812bb8061973f9c43b94ddf758ccac9d7d357","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":32,"id":"7607c63a_39033d10","updated":"2023-11-02 20:54:39.000000000","message":"ok, my review completly fell apart.  Primarily; I don\u0027t see any tests that describe the behavior of the new x-backend-record-shard-format header WRT query params beyond states\u003dlisting \u0026 states\u003dupdating; the one I think is most significant being:\n\n\tvagrant@saio:~$ curl -s -XGET -I -H \u0027x-backend-record-type: auto\u0027 -H \u0027x-backend-record-shard-format: namespace\u0027 \"http://127.0.0.1:6011/sdb1/54/AUTH_test/lots-of-files?format\u003djson\" | grep -i format\n\tX-Backend-Record-Shard-Format: namespace\n\tvagrant@saio:~$ curl -s -XGET -I -H \u0027x-backend-record-type: auto\u0027 -H \u0027x-backend-record-shard-format: namespace\u0027 \"http://127.0.0.1:6011/sdb1/54/AUTH_test/lots-of-files?format\u003djson\u0026reverse\u003dtrue\" | grep -i format\n\tX-Backend-Record-Shard-Format: full\n\n... we should endeavor to document/enumerate ALL the query params/headers that cause the record-shard-format header to be ignored and return full shard range format.\n\nBut I think where I fell apart was trying to understand x-backend-override-shard-name-filter\n\nFirst of all with `x-backend-record-type: shard` marker operatees on the *object* namespace, same as includes, it\u0027s not like object-listings where the marker says which \"name\" field from the last returned list of dicts to continue.  That is to say I needed:\n\n\tvagrant@saio:~$ curl -s \"http://127.0.0.1:6011/sdb1/54/AUTH_test/lots-of-files?format\u003djson\u0026includes\u003dobj0998\" -H \u0027x-backend-record-type: shard\u0027 -H \u0027x-backend-record-shard-format: namespace\u0027 | python -m json.tool\n\t[\n\t    {\n\t\t\"name\": \".shards_AUTH_test/lots-of-files-353f815aaf251b9c38032f8fca4a6f9a-1698955312.34697-3\",\n\t\t\"timestamp\": \"1698955312.34697\",\n\t\t\"lower\": \"obj0974\",\n\t\t\"upper\": \"\",\n\t\t\"object_count\": 25,\n\t\t\"bytes_used\": 0,\n\t\t\"meta_timestamp\": \"1698955343.98350\",\n\t\t\"deleted\": 0,\n\t\t\"state\": 40,\n\t\t\"state_timestamp\": \"1698955312.34697\",\n\t\t\"epoch\": null,\n\t\t\"reported\": 0,\n\t\t\"tombstones\": 0,\n\t\t\"last_modified\": \"2023-11-02T20:01:52.346970\"\n\t    }\n\t]\n\nnot marker\u003d.shards_AUTH_test/lots-of-files-353f815aaf251b9c38032f8fca4a6f9a-1698955312.34697-2\n\nAnyway, it\u0027s not clear to me where or who sends these marker/end_marker/includes/reverse params and if they DO send them, do they want them to be applied or do they want full shard range listings?  And if they want the full listing ... do they want namespaces?!\n\nhttps://github.com/NVIDIA/swift/blob/master/swift/proxy/controllers/container.py#L272\n\nSo... I just ran out of time; hopefully I can get some more tests - maybe the follow-up proxy (or sharder internal client?) change(s) will make it more clear.","commit_id":"5380c3ba8763ed30da5f0837c5b2efc45e6a8003"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"371411a2ee5c6ef0a2c5de29663f90795ec8b125","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":32,"id":"8e9f1941_aa048c48","updated":"2023-11-02 03:48:43.000000000","message":"recheck\nAn unrelated failure \"Found unexpected object\" in test_reconciler_move_object_twice.","commit_id":"5380c3ba8763ed30da5f0837c5b2efc45e6a8003"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"abdaf1ad1905cacd3520e5dd5c60706bbac6553e","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":35,"id":"d006d762_d44ee08a","updated":"2023-11-06 05:04:07.000000000","message":"thanks all for the reviews!","commit_id":"aaedb89d4ad62d77a842b6cd41e416b422d6a727"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"18e816986da0f7d7dbb1575c0927744a62acb048","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":38,"id":"ea3be29e_1cd715dd","updated":"2023-11-10 19:48:08.000000000","message":"I think this is gotten to the point I\u0027m pretty much entirely happy with it?  I think I was just looking for the tests - and those are pretty awesome now!\n\nBut, I know we\u0027ve talked about merging a new backend interface without a consumer on master has a HIGH risk of getting it wrong.  I think I\u0027m +2 - but, I might have forgetten something I actually DO care about.  So I\u0027m pretty sure the plan is hold here for now.\n\nI started to look down the chain at the consumer(s) and in particular got distracted/excited about some refactoring Al is working on in the proxy:\n\nhttps://review.opendev.org/c/openstack/swift/+/900350\n\nwhich is ostensibly about renaming shard_ranges to namespaces AFTER this change - but seemed more significantly about just FIXING the shard record handling so that we can grok it well enough to ADD namespace!?  So I don\u0027t know where we\u0027ll end up.  This will probably sit right here until we have the plan for the proxy fleshed out.","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"e367c6989582ab6ef04e286014b46f4c1023eb77","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":38,"id":"e000c686_69d8998e","updated":"2023-11-13 23:38:06.000000000","message":"Now that I\u0027ve seen it, that `namespaces[-1].upper \u003d\u003d Namespace.MAX` bugs me. I think that\u0027s the only thing that\u0027s really blocking me from a +2, though I had a few other comments.","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"6641867462956b530bfcb3af9268e1a8f3f322eb","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":38,"id":"b9b67ef5_1c116fa6","updated":"2023-11-14 01:59:11.000000000","message":"Thanks!","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"3add8e7afddeff703c713a8d45f5e76623f5bef4","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":43,"id":"a981981c_a319d199","updated":"2023-11-22 23:08:43.000000000","message":"i was looking at the timing re-order; IIRC we we started carrying the proxy-request-namespaces before we could measure the object server GET response timing (or maybe trying to measure the improvement in container GET timing demonstrated the gap in telementry?)\n\nbut I think the re-order of the refactor is reasonbale since if the other is ready to go.","commit_id":"0799b998006292a8f2abc970cd335d462892d9df"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"84628c218a71364ffc56e4816acd47839e3c823c","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":43,"id":"ee55d814_f28810e5","in_reply_to":"a981981c_a319d199","updated":"2023-11-28 04:12:06.000000000","message":"Agreed, and thank you for merging the timing metrics patch! BTW, after we started to carry the container \u0026 proxy namespace patch, we saw the improvement before we added timing metrics, the overall container GET response timing on those busy containers reduced by 3X.","commit_id":"0799b998006292a8f2abc970cd335d462892d9df"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"842bbe0a38788868d7d93cd1a743428f8631947a","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":50,"id":"b3ccbe32_e3e45539","updated":"2023-12-11 19:00:06.000000000","message":"recheck","commit_id":"74660adbc4e1201bc1c6a9f091311f532db67713"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"b15761c0e0e2d7e69f93b97e5e7b7847078fdb4a","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":56,"id":"bbb80570_b0ef4e61","updated":"2024-01-09 15:23:00.000000000","message":"There\u0027s still some old comments that it would be good to resolve.\n\nI suggest some minor fixes for unit tests here, but they are not blockers.\n\nI wonder if there is still some performance gain to be squeezed out, but I\u0027d be ok with that being a follow up (I can imagine broker getting a get_namespaces_as_dicts() method if necessary).","commit_id":"e3676eff802dcde5eeabbda8d5707025f40412ad"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"74e9ae2e078e75459ac35188586733c97d906994","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":57,"id":"89ef0009_99118650","updated":"2024-01-10 14:27:08.000000000","message":"I repeated some benchmarking on this version of the patchset, using direct GETs to a container server with a DB having ~12k shard ranges:\n\n```\nAPI GET/s (~12000 shards in list)\n 3.65  master (full format shard ranges)\n 3.54  patch (full format shard ranges)\n13.23  patch (namespace format)\n```\n\nI\u0027d like to be confident that the companion patch https://review.opendev.org/c/openstack/swift/+/895602/45?usp\u003drelated-change is ready to merge before merging this.","commit_id":"7be1763455236ebf184d6be483798bad5e2acc9d"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"2a05535963b506bdea7d14b1837ad1324a75db19","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":57,"id":"44c01177_200fde0c","updated":"2024-01-11 05:56:18.000000000","message":"This is looking awesome, I only see one missed variable replacement in a test. Otherwise I think we\u0027re ready to land this one.\n\nIt\u0027s much faster, and we\u0027ve been running it in prod for a while, so we kow it\u0027s be thrown in the deep end of the pool, testing wise.","commit_id":"7be1763455236ebf184d6be483798bad5e2acc9d"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"22439211b73d9553dcf8bf959ece5bd110756bdc","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":58,"id":"588f9ead_fe27aef8","updated":"2024-01-12 03:31:54.000000000","message":"Great! I think this is ready to go! better we\u0027re already carrying it in prod and works great!\n\nI\u0027m going to land this sucker! Great work team!","commit_id":"c073933387347bec7a917f34ca7e2f15993b33a5"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"e77e3479d3023573255ad217d4a8ef9b220111a7","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":58,"id":"bb07212a_937622a4","updated":"2024-01-12 14:29:07.000000000","message":"never quite finished this review; but since it\u0027s merged I\u0027ll just post up where I had gotten too; no concerns from me!","commit_id":"c073933387347bec7a917f34ca7e2f15993b33a5"}],"swift/common/utils/__init__.py":[{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"bf33de1f5f106065568e0985e20cc76fff8f97e0","unresolved":true,"context_lines":[{"line_number":268,"context_line":"    :param value: An object to transform."},{"line_number":269,"context_line":"    :return: A set. If ``value`` is an instance of a ``list``, ``tuple`` or"},{"line_number":270,"context_line":"        ``set`` then a set containing the items in ``value`` is returned,"},{"line_number":271,"context_line":"        otherwise a set containing ``value`` is returned."},{"line_number":272,"context_line":"    \"\"\""},{"line_number":273,"context_line":"    if value is None:"},{"line_number":274,"context_line":"        return None"}],"source_content_type":"text/x-python","patch_set":12,"id":"6ca40bee_0b5904c0","line":271,"range":{"start_line":271,"start_character":34,"end_line":271,"end_character":57},"updated":"2023-08-31 06:46:04.000000000","message":"Although this is not strictly true. When value is None then None is returned not {None}.\n\nA set of None is valid, although it probably isn\u0027t in the code that calls this.","commit_id":"4638670d9e4dc6850b5447e2d5756be79b4c1aff"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"fd158f484a16e6ddbe812130fa865b4ebe5e7a8b","unresolved":false,"context_lines":[{"line_number":268,"context_line":"    :param value: An object to transform."},{"line_number":269,"context_line":"    :return: A set. If ``value`` is an instance of a ``list``, ``tuple`` or"},{"line_number":270,"context_line":"        ``set`` then a set containing the items in ``value`` is returned,"},{"line_number":271,"context_line":"        otherwise a set containing ``value`` is returned."},{"line_number":272,"context_line":"    \"\"\""},{"line_number":273,"context_line":"    if value is None:"},{"line_number":274,"context_line":"        return None"}],"source_content_type":"text/x-python","patch_set":12,"id":"7f6efff7_cf80d515","line":271,"range":{"start_line":271,"start_character":34,"end_line":271,"end_character":57},"in_reply_to":"6ca40bee_0b5904c0","updated":"2023-08-31 21:06:25.000000000","message":"Ack","commit_id":"4638670d9e4dc6850b5447e2d5756be79b4c1aff"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"dd2266c879ea38eae3d5fd5fe501e62a1982679c","unresolved":true,"context_lines":[{"line_number":294,"context_line":"    elif isinstance(value, (list, tuple, set)):"},{"line_number":295,"context_line":"        return set(value)"},{"line_number":296,"context_line":"    else:"},{"line_number":297,"context_line":"        return set((value,))"},{"line_number":298,"context_line":""},{"line_number":299,"context_line":""},{"line_number":300,"context_line":"def non_negative_float(value):"}],"source_content_type":"text/x-python","patch_set":22,"id":"8580a256_9691989d","line":297,"updated":"2023-09-13 19:57:42.000000000","message":"nit: We can just do `{value}` -- we don\u0027t have to worry about py26 or py30 at least! https://docs.python.org/3/whatsnew/2.7.html#python-3-1-features\n\nOh yeah, we might want to do something similar with `TRUE_VALUES` above, too...","commit_id":"b105e043017904fcf520fd7cdad5c07aba47e76f"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"693fbccfb57938a9ccae311ba3d53701e72a9a9d","unresolved":false,"context_lines":[{"line_number":294,"context_line":"    elif isinstance(value, (list, tuple, set)):"},{"line_number":295,"context_line":"        return set(value)"},{"line_number":296,"context_line":"    else:"},{"line_number":297,"context_line":"        return set((value,))"},{"line_number":298,"context_line":""},{"line_number":299,"context_line":""},{"line_number":300,"context_line":"def non_negative_float(value):"}],"source_content_type":"text/x-python","patch_set":22,"id":"0dd271f2_e0b8920a","line":297,"in_reply_to":"8580a256_9691989d","updated":"2023-09-21 04:37:28.000000000","message":"Done","commit_id":"b105e043017904fcf520fd7cdad5c07aba47e76f"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"dd2266c879ea38eae3d5fd5fe501e62a1982679c","unresolved":true,"context_lines":[{"line_number":5101,"context_line":"    def sort_key_order(name, lower, upper, state):"},{"line_number":5102,"context_line":"        # Use Namespace.MaxBound() for upper bound \u0027\u0027, this will allow this"},{"line_number":5103,"context_line":"        # record to be sorted correctly by upper."},{"line_number":5104,"context_line":"        upper \u003d upper if upper else Namespace.MaxBound()"},{"line_number":5105,"context_line":"        # defines the sort order for shard ranges"},{"line_number":5106,"context_line":"        # note if this ever changes to *not* sort by upper first then it breaks"},{"line_number":5107,"context_line":"        # a key assumption for bisect, which is used by utils.find_namespace"}],"source_content_type":"text/x-python","patch_set":22,"id":"8e60f6ce_9f888938","line":5104,"updated":"2023-09-13 19:57:42.000000000","message":":-/ I\u0027m realizing, I think this bit of trickery is what\u0027s keeping us from implementing the sort in sqlite...","commit_id":"b105e043017904fcf520fd7cdad5c07aba47e76f"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"693fbccfb57938a9ccae311ba3d53701e72a9a9d","unresolved":false,"context_lines":[{"line_number":5101,"context_line":"    def sort_key_order(name, lower, upper, state):"},{"line_number":5102,"context_line":"        # Use Namespace.MaxBound() for upper bound \u0027\u0027, this will allow this"},{"line_number":5103,"context_line":"        # record to be sorted correctly by upper."},{"line_number":5104,"context_line":"        upper \u003d upper if upper else Namespace.MaxBound()"},{"line_number":5105,"context_line":"        # defines the sort order for shard ranges"},{"line_number":5106,"context_line":"        # note if this ever changes to *not* sort by upper first then it breaks"},{"line_number":5107,"context_line":"        # a key assumption for bisect, which is used by utils.find_namespace"}],"source_content_type":"text/x-python","patch_set":22,"id":"df8c7e2f_833dfe74","line":5104,"in_reply_to":"8e60f6ce_9f888938","updated":"2023-09-21 04:37:28.000000000","message":"Ack","commit_id":"b105e043017904fcf520fd7cdad5c07aba47e76f"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":true,"context_lines":[{"line_number":5054,"context_line":"        # note if this ever changes to *not* sort by upper first then it breaks"},{"line_number":5055,"context_line":"        # a key assumption for bisect, which is used by utils.find_namespace"},{"line_number":5056,"context_line":"        # with shard ranges."},{"line_number":5057,"context_line":"        return sr.upper, sr.state, sr.lower, sr.name"},{"line_number":5058,"context_line":""},{"line_number":5059,"context_line":"    def is_child_of(self, parent):"},{"line_number":5060,"context_line":"        \"\"\""}],"source_content_type":"text/x-python","patch_set":27,"id":"b190ae76_907f002c","side":"PARENT","line":5057,"updated":"2023-10-02 22:23:29.000000000","message":"couldn\u0027t we just change this to:\n\n```\nupper \u003d upper if upper else Namespace.MaxBound()\nreturn upper, sr.state, sr.lower, sr.name\n```","commit_id":"5555980fb5fce79140efc0c8ab8a730bd552733c"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"bcb52432e997cd207866f8b22a7c55a1f0a34204","unresolved":false,"context_lines":[{"line_number":5054,"context_line":"        # note if this ever changes to *not* sort by upper first then it breaks"},{"line_number":5055,"context_line":"        # a key assumption for bisect, which is used by utils.find_namespace"},{"line_number":5056,"context_line":"        # with shard ranges."},{"line_number":5057,"context_line":"        return sr.upper, sr.state, sr.lower, sr.name"},{"line_number":5058,"context_line":""},{"line_number":5059,"context_line":"    def is_child_of(self, parent):"},{"line_number":5060,"context_line":"        \"\"\""}],"source_content_type":"text/x-python","patch_set":27,"id":"8882383d_cedc44dc","side":"PARENT","line":5057,"in_reply_to":"b190ae76_907f002c","updated":"2023-10-31 16:11:06.000000000","message":"Done","commit_id":"5555980fb5fce79140efc0c8ab8a730bd552733c"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":true,"context_lines":[{"line_number":275,"context_line":""},{"line_number":276,"context_line":""},{"line_number":277,"context_line":"# Used when reading config values"},{"line_number":278,"context_line":"TRUE_VALUES \u003d {\u0027true\u0027, \u00271\u0027, \u0027yes\u0027, \u0027on\u0027, \u0027t\u0027, \u0027y\u0027}"},{"line_number":279,"context_line":""},{"line_number":280,"context_line":""},{"line_number":281,"context_line":"def transform_to_set(value):"}],"source_content_type":"text/x-python","patch_set":27,"id":"b8d7e537_a22171f4","line":278,"updated":"2023-10-02 22:23:29.000000000","message":"nice!","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"ab133fd538b104017d417ae79e5e983aa083a419","unresolved":false,"context_lines":[{"line_number":275,"context_line":""},{"line_number":276,"context_line":""},{"line_number":277,"context_line":"# Used when reading config values"},{"line_number":278,"context_line":"TRUE_VALUES \u003d {\u0027true\u0027, \u00271\u0027, \u0027yes\u0027, \u0027on\u0027, \u0027t\u0027, \u0027y\u0027}"},{"line_number":279,"context_line":""},{"line_number":280,"context_line":""},{"line_number":281,"context_line":"def transform_to_set(value):"}],"source_content_type":"text/x-python","patch_set":27,"id":"18f78879_4c7967e7","line":278,"in_reply_to":"b8d7e537_a22171f4","updated":"2023-10-04 04:32:17.000000000","message":"Ack","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":true,"context_lines":[{"line_number":290,"context_line":"        the single ``value`` is returned."},{"line_number":291,"context_line":"    \"\"\""},{"line_number":292,"context_line":"    if value is None:"},{"line_number":293,"context_line":"        return None"},{"line_number":294,"context_line":"    elif isinstance(value, (list, tuple, set)):"},{"line_number":295,"context_line":"        return set(value)"},{"line_number":296,"context_line":"    else:"}],"source_content_type":"text/x-python","patch_set":27,"id":"b0e429f6_21cd7bd3","line":293,"updated":"2023-10-02 22:23:29.000000000","message":"kind of specific, empty set or {None} might seem more expected.","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"bcb52432e997cd207866f8b22a7c55a1f0a34204","unresolved":false,"context_lines":[{"line_number":290,"context_line":"        the single ``value`` is returned."},{"line_number":291,"context_line":"    \"\"\""},{"line_number":292,"context_line":"    if value is None:"},{"line_number":293,"context_line":"        return None"},{"line_number":294,"context_line":"    elif isinstance(value, (list, tuple, set)):"},{"line_number":295,"context_line":"        return set(value)"},{"line_number":296,"context_line":"    else:"}],"source_content_type":"text/x-python","patch_set":27,"id":"99863157_fa80f3b4","line":293,"in_reply_to":"613e06f7_aad7ebb5","updated":"2023-10-31 16:11:06.000000000","message":"Done","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"4fbcf228954a81390f569f70a531c5bff3f910ed","unresolved":true,"context_lines":[{"line_number":290,"context_line":"        the single ``value`` is returned."},{"line_number":291,"context_line":"    \"\"\""},{"line_number":292,"context_line":"    if value is None:"},{"line_number":293,"context_line":"        return None"},{"line_number":294,"context_line":"    elif isinstance(value, (list, tuple, set)):"},{"line_number":295,"context_line":"        return set(value)"},{"line_number":296,"context_line":"    else:"}],"source_content_type":"text/x-python","patch_set":27,"id":"613e06f7_aad7ebb5","line":293,"in_reply_to":"b0e429f6_21cd7bd3","updated":"2023-10-03 17:56:56.000000000","message":"Agreed; this feels a little odd, having `transform_to_set` not return a set...","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":true,"context_lines":[{"line_number":292,"context_line":"    if value is None:"},{"line_number":293,"context_line":"        return None"},{"line_number":294,"context_line":"    elif isinstance(value, (list, tuple, set)):"},{"line_number":295,"context_line":"        return set(value)"},{"line_number":296,"context_line":"    else:"},{"line_number":297,"context_line":"        return {value}"},{"line_number":298,"context_line":""}],"source_content_type":"text/x-python","patch_set":27,"id":"fb1c110d_d2f5d1d4","line":295,"updated":"2023-10-02 22:23:29.000000000","message":"```\n\u003e\u003e\u003e set(\u0027asdf\u0027)\n{\u0027f\u0027, \u0027d\u0027, \u0027s\u0027, \u0027a\u0027}\n```\n\n^ basically I want set to have an altnerative constructor that doesn\u0027t break apart strings just cause they\u0027re iterable.","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"bcb52432e997cd207866f8b22a7c55a1f0a34204","unresolved":false,"context_lines":[{"line_number":292,"context_line":"    if value is None:"},{"line_number":293,"context_line":"        return None"},{"line_number":294,"context_line":"    elif isinstance(value, (list, tuple, set)):"},{"line_number":295,"context_line":"        return set(value)"},{"line_number":296,"context_line":"    else:"},{"line_number":297,"context_line":"        return {value}"},{"line_number":298,"context_line":""}],"source_content_type":"text/x-python","patch_set":27,"id":"745241f0_2f8bc375","line":295,"in_reply_to":"fb1c110d_d2f5d1d4","updated":"2023-10-31 16:11:06.000000000","message":"it won\u0027t come down to this path, since the condition is:\n ``isinstance(value, (list, tuple, set)):``","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":true,"context_lines":[{"line_number":5069,"context_line":""},{"line_number":5070,"context_line":"    @classmethod"},{"line_number":5071,"context_line":"    def sort_key(cls, sr):"},{"line_number":5072,"context_line":"        return cls.sort_key_order(sr.name, sr.lower, sr.upper, sr.state)"},{"line_number":5073,"context_line":""},{"line_number":5074,"context_line":"    @staticmethod"},{"line_number":5075,"context_line":"    def sort_key_order(name, lower, upper, state):"}],"source_content_type":"text/x-python","patch_set":27,"id":"e87aa26e_0fcd1e36","line":5072,"updated":"2023-10-02 22:23:29.000000000","message":"as a classmethod this is already override-able","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"ab133fd538b104017d417ae79e5e983aa083a419","unresolved":false,"context_lines":[{"line_number":5069,"context_line":""},{"line_number":5070,"context_line":"    @classmethod"},{"line_number":5071,"context_line":"    def sort_key(cls, sr):"},{"line_number":5072,"context_line":"        return cls.sort_key_order(sr.name, sr.lower, sr.upper, sr.state)"},{"line_number":5073,"context_line":""},{"line_number":5074,"context_line":"    @staticmethod"},{"line_number":5075,"context_line":"    def sort_key_order(name, lower, upper, state):"}],"source_content_type":"text/x-python","patch_set":27,"id":"0b22e8ea_bf2ae8d0","line":5072,"in_reply_to":"e87aa26e_0fcd1e36","updated":"2023-10-04 04:32:17.000000000","message":"Namespace class doesn\u0027t have \"sr.state\", while the new get_namespaces function needs to sort by all four attributes.","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":true,"context_lines":[{"line_number":5080,"context_line":"        # note if this ever changes to *not* sort by upper first then it breaks"},{"line_number":5081,"context_line":"        # a key assumption for bisect, which is used by utils.find_namespace"},{"line_number":5082,"context_line":"        # with shard ranges."},{"line_number":5083,"context_line":"        return upper, state, lower, name"},{"line_number":5084,"context_line":""},{"line_number":5085,"context_line":"    def is_child_of(self, parent):"},{"line_number":5086,"context_line":"        \"\"\""}],"source_content_type":"text/x-python","patch_set":27,"id":"8cf681ae_e57e0b16","line":5083,"updated":"2023-10-02 22:23:29.000000000","message":"i\u0027m totally lost by this class hierarchy - I hate mostly hate staticmethod as a language feature","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"ab133fd538b104017d417ae79e5e983aa083a419","unresolved":false,"context_lines":[{"line_number":5080,"context_line":"        # note if this ever changes to *not* sort by upper first then it breaks"},{"line_number":5081,"context_line":"        # a key assumption for bisect, which is used by utils.find_namespace"},{"line_number":5082,"context_line":"        # with shard ranges."},{"line_number":5083,"context_line":"        return upper, state, lower, name"},{"line_number":5084,"context_line":""},{"line_number":5085,"context_line":"    def is_child_of(self, parent):"},{"line_number":5086,"context_line":"        \"\"\""}],"source_content_type":"text/x-python","patch_set":27,"id":"e6eb3906_153b9202","line":5083,"in_reply_to":"8cf681ae_e57e0b16","updated":"2023-10-04 04:32:17.000000000","message":"Another option will have get_namespaces function implement its own sort function and use the same sort order, but I feel that will make related code harder to maintain, since it\u0027s critical that they must use the same sort order. I feel it\u0027s better option to have two code paths to share one sort_key_order() function which defines the order.","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8a24e200d9cb30696c694717a3c20781843d2652","unresolved":true,"context_lines":[{"line_number":5079,"context_line":"    def sort_key_order(name, lower, upper, state):"},{"line_number":5080,"context_line":"        # Use Namespace.MaxBound() for upper bound \u0027\u0027, this will allow this"},{"line_number":5081,"context_line":"        # record to be sorted correctly by upper."},{"line_number":5082,"context_line":"        upper \u003d upper if upper else Namespace.MaxBound()"},{"line_number":5083,"context_line":"        # defines the sort order for shard ranges"},{"line_number":5084,"context_line":"        # note if this ever changes to *not* sort by upper first then it breaks"},{"line_number":5085,"context_line":"        # a key assumption for bisect, which is used by utils.find_namespace"}],"source_content_type":"text/x-python","patch_set":29,"id":"db325217_07282d18","line":5082,"updated":"2023-10-31 19:21:20.000000000","message":"this seems like a useful and significant change; I assume generally a ShardRange can\u0027t have self.upper \u003d \u0027\u0027 it\u0027s already been converted to MaxBound()\n\nit might be reasonable to have some stand-alone tests to demonstrate the design of this class hierarchy to enable easy sorting.","commit_id":"95dcd77bf6b77f83ab3471d40b0f658ee5c5014f"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"b263eb52a246427f54c4caf4fda57755aa96fa73","unresolved":false,"context_lines":[{"line_number":5079,"context_line":"    def sort_key_order(name, lower, upper, state):"},{"line_number":5080,"context_line":"        # Use Namespace.MaxBound() for upper bound \u0027\u0027, this will allow this"},{"line_number":5081,"context_line":"        # record to be sorted correctly by upper."},{"line_number":5082,"context_line":"        upper \u003d upper if upper else Namespace.MaxBound()"},{"line_number":5083,"context_line":"        # defines the sort order for shard ranges"},{"line_number":5084,"context_line":"        # note if this ever changes to *not* sort by upper first then it breaks"},{"line_number":5085,"context_line":"        # a key assumption for bisect, which is used by utils.find_namespace"}],"source_content_type":"text/x-python","patch_set":29,"id":"54df130e_8a2e646a","line":5082,"in_reply_to":"db325217_07282d18","updated":"2023-11-02 00:49:07.000000000","message":"Yes, ShardRange will have self.upper \u003d MaxBound(), this is added to support the new usage for the  ``get_namespaces`` in backend.py, will need call ``sort_key_order`` and will pass self.upper \u003d \u0027\u0027. Good point on test cases! Both new function ``sort_key_order`` and the existing function ``sort_key`` need test cases, got them added.","commit_id":"95dcd77bf6b77f83ab3471d40b0f658ee5c5014f"},{"author":{"_account_id":12620,"name":"Alexander Corwin","email":"ancorwin@gmail.com","username":"acorwin"},"change_message_id":"25332a0feddff6897254996114734720ed259083","unresolved":true,"context_lines":[{"line_number":281,"context_line":"def transform_to_set(value):"},{"line_number":282,"context_line":"    \"\"\""},{"line_number":283,"context_line":"    Transform either a single value or a collection of values to a set of"},{"line_number":284,"context_line":"    unique values."},{"line_number":285,"context_line":""},{"line_number":286,"context_line":"    :param value: An object to transform."},{"line_number":287,"context_line":"    :return: A set. If ``value`` is an instance of a ``list``, ``tuple`` or"}],"source_content_type":"text/x-python","patch_set":30,"id":"deeab342_9df72a59","line":284,"updated":"2023-11-01 18:57:52.000000000","message":"this summary is wrong (it does not return a set if the value passed in is None)\n\ni dont have the context here but Clay pointed me at this function and I was motivated to point out that the comment is incorrect.\n\nthe semantics also seem wild to me (why does `transform_to_set(None)` not produce `{None}`?) but honestly as long as it\u0027s documented correctly it\u0027s probably fine","commit_id":"5d7e2c08daf0745ba21548b07c1afefc9f4e6e92"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"b263eb52a246427f54c4caf4fda57755aa96fa73","unresolved":false,"context_lines":[{"line_number":281,"context_line":"def transform_to_set(value):"},{"line_number":282,"context_line":"    \"\"\""},{"line_number":283,"context_line":"    Transform either a single value or a collection of values to a set of"},{"line_number":284,"context_line":"    unique values."},{"line_number":285,"context_line":""},{"line_number":286,"context_line":"    :param value: An object to transform."},{"line_number":287,"context_line":"    :return: A set. If ``value`` is an instance of a ``list``, ``tuple`` or"}],"source_content_type":"text/x-python","patch_set":30,"id":"b3e34a0b_8b4bd978","line":284,"in_reply_to":"deeab342_9df72a59","updated":"2023-11-02 00:49:07.000000000","message":"Done","commit_id":"5d7e2c08daf0745ba21548b07c1afefc9f4e6e92"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"c52649a9d1d9fcf00c4b0e959afdc27c2366af8f","unresolved":true,"context_lines":[{"line_number":286,"context_line":"    :param value: An object to transform."},{"line_number":287,"context_line":"    :return: A set. If ``value`` is an instance of a ``list``, ``tuple`` or"},{"line_number":288,"context_line":"        ``set`` then a set containing the items in ``value`` is returned; if"},{"line_number":289,"context_line":"        value is None then None is returned; otherwise a set containing"},{"line_number":290,"context_line":"        the single ``value`` is returned."},{"line_number":291,"context_line":"    \"\"\""},{"line_number":292,"context_line":"    if value is None:"}],"source_content_type":"text/x-python","patch_set":31,"id":"37825441_d304e26d","line":289,"updated":"2023-11-01 18:52:06.000000000","message":"alexander noticed this docstring is out of date with the implementation regarding the behavior w/ value\u003dNone","commit_id":"9a03bb9447212dc1321393cbddca702272200bce"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"b263eb52a246427f54c4caf4fda57755aa96fa73","unresolved":false,"context_lines":[{"line_number":286,"context_line":"    :param value: An object to transform."},{"line_number":287,"context_line":"    :return: A set. If ``value`` is an instance of a ``list``, ``tuple`` or"},{"line_number":288,"context_line":"        ``set`` then a set containing the items in ``value`` is returned; if"},{"line_number":289,"context_line":"        value is None then None is returned; otherwise a set containing"},{"line_number":290,"context_line":"        the single ``value`` is returned."},{"line_number":291,"context_line":"    \"\"\""},{"line_number":292,"context_line":"    if value is None:"}],"source_content_type":"text/x-python","patch_set":31,"id":"1b58fad1_37ead15b","line":289,"in_reply_to":"37825441_d304e26d","updated":"2023-11-02 00:49:07.000000000","message":"Ack","commit_id":"9a03bb9447212dc1321393cbddca702272200bce"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"74eaf120af694826c670d061515def674745fb63","unresolved":true,"context_lines":[{"line_number":283,"context_line":"TRUE_VALUES \u003d {\u0027true\u0027, \u00271\u0027, \u0027yes\u0027, \u0027on\u0027, \u0027t\u0027, \u0027y\u0027}"},{"line_number":284,"context_line":""},{"line_number":285,"context_line":""},{"line_number":286,"context_line":"def transform_to_set(value):"},{"line_number":287,"context_line":"    \"\"\""},{"line_number":288,"context_line":"    Transform either a single value or a collection of values to a set of"},{"line_number":289,"context_line":"    unique values."}],"source_content_type":"text/x-python","patch_set":53,"id":"4b2478fd_0d1b42c5","line":286,"updated":"2024-01-05 15:04:39.000000000","message":"ugh, this isn\u0027t ageing well! how about we rip it out ?\nhttps://review.opendev.org/c/openstack/swift/+/904861","commit_id":"bae9654b8f540894c0fa8f5528d1e4c4e65baa44"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"b15761c0e0e2d7e69f93b97e5e7b7847078fdb4a","unresolved":false,"context_lines":[{"line_number":283,"context_line":"TRUE_VALUES \u003d {\u0027true\u0027, \u00271\u0027, \u0027yes\u0027, \u0027on\u0027, \u0027t\u0027, \u0027y\u0027}"},{"line_number":284,"context_line":""},{"line_number":285,"context_line":""},{"line_number":286,"context_line":"def transform_to_set(value):"},{"line_number":287,"context_line":"    \"\"\""},{"line_number":288,"context_line":"    Transform either a single value or a collection of values to a set of"},{"line_number":289,"context_line":"    unique values."}],"source_content_type":"text/x-python","patch_set":53,"id":"fc76c484_f8cf2b91","line":286,"in_reply_to":"4b2478fd_0d1b42c5","updated":"2024-01-09 15:23:00.000000000","message":"Done","commit_id":"bae9654b8f540894c0fa8f5528d1e4c4e65baa44"}],"swift/container/backend.py":[{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"35e5be292734d19503eaa3be35584fc308b9ac53","unresolved":true,"context_lines":[{"line_number":1922,"context_line":"        \"\"\""},{"line_number":1923,"context_line":"        Returns a list of persisted namespaces with state in raw format."},{"line_number":1924,"context_line":""},{"line_number":1925,"context_line":"        :param states: if specified, restricts the returned list to namespaces"},{"line_number":1926,"context_line":"            that have the given state(s); can be a list of ints or a single"},{"line_number":1927,"context_line":"            int."},{"line_number":1928,"context_line":"        :return: a list of tuples of (name, lower, upper, state)"}],"source_content_type":"text/x-python","patch_set":5,"id":"cdaae806_7ee1f4b4","line":1925,"range":{"start_line":1925,"start_character":23,"end_line":1925,"end_character":35},"updated":"2023-08-10 05:25:24.000000000","message":"If specified? looks like on line 1930 it must we specified or we don\u0027t get anything back.","commit_id":"5b1b5d986af7233b9c18a91e77cef6e1fbea8706"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"c03b18b305666b4f043dd6c764836f425e39f7fa","unresolved":false,"context_lines":[{"line_number":1922,"context_line":"        \"\"\""},{"line_number":1923,"context_line":"        Returns a list of persisted namespaces with state in raw format."},{"line_number":1924,"context_line":""},{"line_number":1925,"context_line":"        :param states: if specified, restricts the returned list to namespaces"},{"line_number":1926,"context_line":"            that have the given state(s); can be a list of ints or a single"},{"line_number":1927,"context_line":"            int."},{"line_number":1928,"context_line":"        :return: a list of tuples of (name, lower, upper, state)"}],"source_content_type":"text/x-python","patch_set":5,"id":"024b612a_b4c582ed","line":1925,"range":{"start_line":1925,"start_character":23,"end_line":1925,"end_character":35},"in_reply_to":"cdaae806_7ee1f4b4","updated":"2023-08-18 04:03:19.000000000","message":"thanks for pointing out the issue, yeah, we should support the case of states not specified and return all states by default.","commit_id":"5b1b5d986af7233b9c18a91e77cef6e1fbea8706"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"35e5be292734d19503eaa3be35584fc308b9ac53","unresolved":true,"context_lines":[{"line_number":1927,"context_line":"            int."},{"line_number":1928,"context_line":"        :return: a list of tuples of (name, lower, upper, state)"},{"line_number":1929,"context_line":"        \"\"\""},{"line_number":1930,"context_line":"        if not states:"},{"line_number":1931,"context_line":"            return []"},{"line_number":1932,"context_line":""},{"line_number":1933,"context_line":"        with self.get() as conn:"},{"line_number":1934,"context_line":"            params \u003d list(states)"}],"source_content_type":"text/x-python","patch_set":5,"id":"c68ab946_bc16699f","line":1931,"range":{"start_line":1930,"start_character":8,"end_line":1931,"end_character":21},"updated":"2023-08-10 05:25:24.000000000","message":"So by sending in no states we don\u0027t get all namespaces we actually get none?","commit_id":"5b1b5d986af7233b9c18a91e77cef6e1fbea8706"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"c03b18b305666b4f043dd6c764836f425e39f7fa","unresolved":false,"context_lines":[{"line_number":1927,"context_line":"            int."},{"line_number":1928,"context_line":"        :return: a list of tuples of (name, lower, upper, state)"},{"line_number":1929,"context_line":"        \"\"\""},{"line_number":1930,"context_line":"        if not states:"},{"line_number":1931,"context_line":"            return []"},{"line_number":1932,"context_line":""},{"line_number":1933,"context_line":"        with self.get() as conn:"},{"line_number":1934,"context_line":"            params \u003d list(states)"}],"source_content_type":"text/x-python","patch_set":5,"id":"b9822182_70276645","line":1931,"range":{"start_line":1930,"start_character":8,"end_line":1931,"end_character":21},"in_reply_to":"c68ab946_bc16699f","updated":"2023-08-18 04:03:19.000000000","message":"Ack","commit_id":"5b1b5d986af7233b9c18a91e77cef6e1fbea8706"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"35e5be292734d19503eaa3be35584fc308b9ac53","unresolved":true,"context_lines":[{"line_number":1947,"context_line":"                    raise"},{"line_number":1948,"context_line":""},{"line_number":1949,"context_line":"        def sort_key(namespace):"},{"line_number":1950,"context_line":"            # defines the sort order for shard ranges"},{"line_number":1951,"context_line":"            # note if this ever changes to *not* sort by upper first then it"},{"line_number":1952,"context_line":"            # breaks a key assumption for bisect, which is used by"},{"line_number":1953,"context_line":"            # utils.find_namespace with shard ranges."}],"source_content_type":"text/x-python","patch_set":5,"id":"1dff4ddb_5637dba7","line":1950,"range":{"start_line":1950,"start_character":41,"end_line":1950,"end_character":53},"updated":"2023-08-10 05:25:24.000000000","message":"namespaces","commit_id":"5b1b5d986af7233b9c18a91e77cef6e1fbea8706"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"c03b18b305666b4f043dd6c764836f425e39f7fa","unresolved":false,"context_lines":[{"line_number":1947,"context_line":"                    raise"},{"line_number":1948,"context_line":""},{"line_number":1949,"context_line":"        def sort_key(namespace):"},{"line_number":1950,"context_line":"            # defines the sort order for shard ranges"},{"line_number":1951,"context_line":"            # note if this ever changes to *not* sort by upper first then it"},{"line_number":1952,"context_line":"            # breaks a key assumption for bisect, which is used by"},{"line_number":1953,"context_line":"            # utils.find_namespace with shard ranges."}],"source_content_type":"text/x-python","patch_set":5,"id":"5ed2e0c5_8ec5e045","line":1950,"range":{"start_line":1950,"start_character":41,"end_line":1950,"end_character":53},"in_reply_to":"1dff4ddb_5637dba7","updated":"2023-08-18 04:03:19.000000000","message":"Done","commit_id":"5b1b5d986af7233b9c18a91e77cef6e1fbea8706"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"35e5be292734d19503eaa3be35584fc308b9ac53","unresolved":true,"context_lines":[{"line_number":1953,"context_line":"            # utils.find_namespace with shard ranges."},{"line_number":1954,"context_line":"            # sort order: upper, state, lower, name"},{"line_number":1955,"context_line":"            return namespace[2], namespace[3], namespace[2], namespace[0]"},{"line_number":1956,"context_line":"        namespaces.sort(key\u003dsort_key)"},{"line_number":1957,"context_line":""},{"line_number":1958,"context_line":"        # If there is a gap in the last, insert a modified copy of own shard"},{"line_number":1959,"context_line":"        # range to fill any gap between the end of any found and the upper"}],"source_content_type":"text/x-python","patch_set":5,"id":"3c5931b4_a716803c","line":1956,"range":{"start_line":1956,"start_character":8,"end_line":1956,"end_character":37},"updated":"2023-08-10 05:25:24.000000000","message":"I guess we do this sort outside SQL because we might have a smaller list to sort when we use states.. I do wonder though if we\u0027d get much speed improvement if we could get the sorting to happen in sqlite :thinking:","commit_id":"5b1b5d986af7233b9c18a91e77cef6e1fbea8706"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"bf33de1f5f106065568e0985e20cc76fff8f97e0","unresolved":false,"context_lines":[{"line_number":1953,"context_line":"            # utils.find_namespace with shard ranges."},{"line_number":1954,"context_line":"            # sort order: upper, state, lower, name"},{"line_number":1955,"context_line":"            return namespace[2], namespace[3], namespace[2], namespace[0]"},{"line_number":1956,"context_line":"        namespaces.sort(key\u003dsort_key)"},{"line_number":1957,"context_line":""},{"line_number":1958,"context_line":"        # If there is a gap in the last, insert a modified copy of own shard"},{"line_number":1959,"context_line":"        # range to fill any gap between the end of any found and the upper"}],"source_content_type":"text/x-python","patch_set":5,"id":"5784aa2b_02c407db","line":1956,"range":{"start_line":1956,"start_character":8,"end_line":1956,"end_character":37},"in_reply_to":"3b762c42_a5f1288d","updated":"2023-08-31 06:46:04.000000000","message":"because ordering by upper \u003d \u0027\u0027 doesn\u0027t work for us if we sort in SQLite so we need to sort in python.","commit_id":"5b1b5d986af7233b9c18a91e77cef6e1fbea8706"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"c03b18b305666b4f043dd6c764836f425e39f7fa","unresolved":true,"context_lines":[{"line_number":1953,"context_line":"            # utils.find_namespace with shard ranges."},{"line_number":1954,"context_line":"            # sort order: upper, state, lower, name"},{"line_number":1955,"context_line":"            return namespace[2], namespace[3], namespace[2], namespace[0]"},{"line_number":1956,"context_line":"        namespaces.sort(key\u003dsort_key)"},{"line_number":1957,"context_line":""},{"line_number":1958,"context_line":"        # If there is a gap in the last, insert a modified copy of own shard"},{"line_number":1959,"context_line":"        # range to fill any gap between the end of any found and the upper"}],"source_content_type":"text/x-python","patch_set":5,"id":"3b762c42_a5f1288d","line":1956,"range":{"start_line":1956,"start_character":8,"end_line":1956,"end_character":37},"in_reply_to":"3c5931b4_a716803c","updated":"2023-08-18 04:03:19.000000000","message":"good idea, I am testing it with multiple column order-by, will see.","commit_id":"5b1b5d986af7233b9c18a91e77cef6e1fbea8706"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"35e5be292734d19503eaa3be35584fc308b9ac53","unresolved":true,"context_lines":[{"line_number":1972,"context_line":"            namespaces.append((filler_sr.name, filler_sr.lower,"},{"line_number":1973,"context_line":"                              filler_sr.upper, filler_sr.state))"},{"line_number":1974,"context_line":""},{"line_number":1975,"context_line":"        return namespaces"},{"line_number":1976,"context_line":""},{"line_number":1977,"context_line":"    def get_own_shard_range(self, no_default\u003dFalse):"},{"line_number":1978,"context_line":"        \"\"\""}],"source_content_type":"text/x-python","patch_set":5,"id":"35f4a9a9_d798da13","line":1975,"range":{"start_line":1975,"start_character":15,"end_line":1975,"end_character":25},"updated":"2023-08-10 05:25:24.000000000","message":"WE probably don\u0027t, but did we want to return the timestamp from the DB too? this way we can include the last_modified in container server namespace listing responses, might be nice to know when a namespace was defined (in case of stale entries) :shrug:","commit_id":"5b1b5d986af7233b9c18a91e77cef6e1fbea8706"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"c03b18b305666b4f043dd6c764836f425e39f7fa","unresolved":true,"context_lines":[{"line_number":1972,"context_line":"            namespaces.append((filler_sr.name, filler_sr.lower,"},{"line_number":1973,"context_line":"                              filler_sr.upper, filler_sr.state))"},{"line_number":1974,"context_line":""},{"line_number":1975,"context_line":"        return namespaces"},{"line_number":1976,"context_line":""},{"line_number":1977,"context_line":"    def get_own_shard_range(self, no_default\u003dFalse):"},{"line_number":1978,"context_line":"        \"\"\""}],"source_content_type":"text/x-python","patch_set":5,"id":"e6758f18_0f2f7387","line":1975,"range":{"start_line":1975,"start_character":15,"end_line":1975,"end_character":25},"in_reply_to":"35f4a9a9_d798da13","updated":"2023-08-18 04:03:19.000000000","message":"can you explain more on this? why we should include the last_modified, since the caller only need namespace attribute lower and name.","commit_id":"5b1b5d986af7233b9c18a91e77cef6e1fbea8706"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"4e664f0e0766cb7f496d3aadc2f927ce29554976","unresolved":false,"context_lines":[{"line_number":1972,"context_line":"            namespaces.append((filler_sr.name, filler_sr.lower,"},{"line_number":1973,"context_line":"                              filler_sr.upper, filler_sr.state))"},{"line_number":1974,"context_line":""},{"line_number":1975,"context_line":"        return namespaces"},{"line_number":1976,"context_line":""},{"line_number":1977,"context_line":"    def get_own_shard_range(self, no_default\u003dFalse):"},{"line_number":1978,"context_line":"        \"\"\""}],"source_content_type":"text/x-python","patch_set":5,"id":"80364acc_4b7f369e","line":1975,"range":{"start_line":1975,"start_character":15,"end_line":1975,"end_character":25},"in_reply_to":"befb3850_d7ab4a08","updated":"2023-08-23 05:23:12.000000000","message":"Ack","commit_id":"5b1b5d986af7233b9c18a91e77cef6e1fbea8706"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"c802c485abf0a70ccedcf3fe2f8e77ab3517fdc2","unresolved":true,"context_lines":[{"line_number":1972,"context_line":"            namespaces.append((filler_sr.name, filler_sr.lower,"},{"line_number":1973,"context_line":"                              filler_sr.upper, filler_sr.state))"},{"line_number":1974,"context_line":""},{"line_number":1975,"context_line":"        return namespaces"},{"line_number":1976,"context_line":""},{"line_number":1977,"context_line":"    def get_own_shard_range(self, no_default\u003dFalse):"},{"line_number":1978,"context_line":"        \"\"\""}],"source_content_type":"text/x-python","patch_set":5,"id":"befb3850_d7ab4a08","line":1975,"range":{"start_line":1975,"start_character":15,"end_line":1975,"end_character":25},"in_reply_to":"e6758f18_0f2f7387","updated":"2023-08-18 14:45:34.000000000","message":"@Matt: think of the API returning Namespace objects - they don\u0027t have timestamp attribute\n\nBut, see my comment on commit message, Namespace objects DO have lower and upper.]","commit_id":"5b1b5d986af7233b9c18a91e77cef6e1fbea8706"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"c802c485abf0a70ccedcf3fe2f8e77ab3517fdc2","unresolved":true,"context_lines":[{"line_number":1932,"context_line":"        :param states: if specified, restricts the returned list to namespaces"},{"line_number":1933,"context_line":"            that have the given state(s); can be a list of ints or a single"},{"line_number":1934,"context_line":"            int."},{"line_number":1935,"context_line":"        :return: a list of tuples of (name, lower, upper, state)"},{"line_number":1936,"context_line":"        \"\"\""},{"line_number":1937,"context_line":"        with self.get() as conn:"},{"line_number":1938,"context_line":"            params \u003d []"}],"source_content_type":"text/x-python","patch_set":7,"id":"c72444ae_12629374","line":1935,"updated":"2023-08-18 14:45:34.000000000","message":"I\u0027m finding it confusing that here a namespace has (name, lower, upper, state), but the HTTP response body is a list of (name, lower) , but a Namespace has (name, lower, upper).\n\nI appreciate that there is a performance benefit to not instantiating Namespace instances here, but I think we should still reflect the Namespace \"data structure\".","commit_id":"1ff63be3bd812db97b3a319444346f5702457f59"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"4e664f0e0766cb7f496d3aadc2f927ce29554976","unresolved":false,"context_lines":[{"line_number":1932,"context_line":"        :param states: if specified, restricts the returned list to namespaces"},{"line_number":1933,"context_line":"            that have the given state(s); can be a list of ints or a single"},{"line_number":1934,"context_line":"            int."},{"line_number":1935,"context_line":"        :return: a list of tuples of (name, lower, upper, state)"},{"line_number":1936,"context_line":"        \"\"\""},{"line_number":1937,"context_line":"        with self.get() as conn:"},{"line_number":1938,"context_line":"            params \u003d []"}],"source_content_type":"text/x-python","patch_set":7,"id":"7448650e_ccc8a4f9","line":1935,"in_reply_to":"c72444ae_12629374","updated":"2023-08-23 05:23:12.000000000","message":"Ack","commit_id":"1ff63be3bd812db97b3a319444346f5702457f59"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"d7efa6e02d77ec38b860851868e2da2de3d45bb8","unresolved":true,"context_lines":[{"line_number":1925,"context_line":""},{"line_number":1926,"context_line":"        return shard_ranges"},{"line_number":1927,"context_line":""},{"line_number":1928,"context_line":"    def get_all_namespaces(self, states\u003dNone):"},{"line_number":1929,"context_line":"        \"\"\""},{"line_number":1930,"context_line":"        Returns a list of persisted namespaces with state in raw format."},{"line_number":1931,"context_line":""}],"source_content_type":"text/x-python","patch_set":8,"id":"53c80d0c_8ed803bf","line":1928,"updated":"2023-08-24 12:22:29.000000000","message":"nit: this lies between methods that deal in getting shard ranges, so breaks the file-order \"flow\". Perhaps relocate above _get_shard_range_rows ?","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"dba5cdc4d8314b4dc7c21ec88d42d98beac2cfbf","unresolved":false,"context_lines":[{"line_number":1925,"context_line":""},{"line_number":1926,"context_line":"        return shard_ranges"},{"line_number":1927,"context_line":""},{"line_number":1928,"context_line":"    def get_all_namespaces(self, states\u003dNone):"},{"line_number":1929,"context_line":"        \"\"\""},{"line_number":1930,"context_line":"        Returns a list of persisted namespaces with state in raw format."},{"line_number":1931,"context_line":""}],"source_content_type":"text/x-python","patch_set":8,"id":"fba2cb83_725549b5","line":1928,"in_reply_to":"53c80d0c_8ed803bf","updated":"2023-08-30 02:56:43.000000000","message":"Done","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"d7efa6e02d77ec38b860851868e2da2de3d45bb8","unresolved":true,"context_lines":[{"line_number":1927,"context_line":""},{"line_number":1928,"context_line":"    def get_all_namespaces(self, states\u003dNone):"},{"line_number":1929,"context_line":"        \"\"\""},{"line_number":1930,"context_line":"        Returns a list of persisted namespaces with state in raw format."},{"line_number":1931,"context_line":""},{"line_number":1932,"context_line":"        :param states: if specified, restricts the returned list to namespaces"},{"line_number":1933,"context_line":"            that have the given state(s); can be a list of ints or a single"}],"source_content_type":"text/x-python","patch_set":8,"id":"cb83a4af_636bf8b7","line":1930,"range":{"start_line":1930,"start_character":61,"end_line":1930,"end_character":71},"updated":"2023-08-24 12:22:29.000000000","message":"is \"raw format\" a hangover from when this method returned a list of dicts?","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"dba5cdc4d8314b4dc7c21ec88d42d98beac2cfbf","unresolved":false,"context_lines":[{"line_number":1927,"context_line":""},{"line_number":1928,"context_line":"    def get_all_namespaces(self, states\u003dNone):"},{"line_number":1929,"context_line":"        \"\"\""},{"line_number":1930,"context_line":"        Returns a list of persisted namespaces with state in raw format."},{"line_number":1931,"context_line":""},{"line_number":1932,"context_line":"        :param states: if specified, restricts the returned list to namespaces"},{"line_number":1933,"context_line":"            that have the given state(s); can be a list of ints or a single"}],"source_content_type":"text/x-python","patch_set":8,"id":"a9d1cca8_a91d414d","line":1930,"range":{"start_line":1930,"start_character":61,"end_line":1930,"end_character":71},"in_reply_to":"cb83a4af_636bf8b7","updated":"2023-08-30 02:56:43.000000000","message":"Ack","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"d7efa6e02d77ec38b860851868e2da2de3d45bb8","unresolved":true,"context_lines":[{"line_number":1938,"context_line":"            # The sort order used by SQL query: upper, state, lower, name."},{"line_number":1939,"context_line":"            # Note if this ever changes to *not* sort by upper first then it"},{"line_number":1940,"context_line":"            # breaks a key assumption for bisect, which is used by"},{"line_number":1941,"context_line":"            # utils.find_namespace with shard ranges."},{"line_number":1942,"context_line":"            params \u003d []"},{"line_number":1943,"context_line":"            if states is None:"},{"line_number":1944,"context_line":"                # TODO: add test case for this."}],"source_content_type":"text/x-python","patch_set":8,"id":"3dcfbbf3_dba43f77","line":1941,"updated":"2023-08-24 12:22:29.000000000","message":"great! thanks for adding this comment\n\ncould also include \"the sort order matches that defined by ShardRange.sort_key\"","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"dba5cdc4d8314b4dc7c21ec88d42d98beac2cfbf","unresolved":false,"context_lines":[{"line_number":1938,"context_line":"            # The sort order used by SQL query: upper, state, lower, name."},{"line_number":1939,"context_line":"            # Note if this ever changes to *not* sort by upper first then it"},{"line_number":1940,"context_line":"            # breaks a key assumption for bisect, which is used by"},{"line_number":1941,"context_line":"            # utils.find_namespace with shard ranges."},{"line_number":1942,"context_line":"            params \u003d []"},{"line_number":1943,"context_line":"            if states is None:"},{"line_number":1944,"context_line":"                # TODO: add test case for this."}],"source_content_type":"text/x-python","patch_set":8,"id":"34895693_77c4f3c4","line":1941,"in_reply_to":"3dcfbbf3_dba43f77","updated":"2023-08-30 02:56:43.000000000","message":"I added a helper function \"ShardRange.sort_key_order\" to be used by two places.","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"d7efa6e02d77ec38b860851868e2da2de3d45bb8","unresolved":true,"context_lines":[{"line_number":1940,"context_line":"            # breaks a key assumption for bisect, which is used by"},{"line_number":1941,"context_line":"            # utils.find_namespace with shard ranges."},{"line_number":1942,"context_line":"            params \u003d []"},{"line_number":1943,"context_line":"            if states is None:"},{"line_number":1944,"context_line":"                # TODO: add test case for this."},{"line_number":1945,"context_line":"                sql \u003d \u0027\u0027\u0027"},{"line_number":1946,"context_line":"                SELECT name, lower, upper FROM %s"}],"source_content_type":"text/x-python","patch_set":8,"id":"f069a1f2_fa3489d2","line":1943,"updated":"2023-08-24 12:22:29.000000000","message":"nit: params \u003d [] could be moved to here\n\nin fact, the params to pass to conn.execute could be fully defined here and at line  1951 e.g.\n\n1944: params \u003d [self.path]\n\n1951: params \u003d [self.path] + list(states)","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"dba5cdc4d8314b4dc7c21ec88d42d98beac2cfbf","unresolved":false,"context_lines":[{"line_number":1940,"context_line":"            # breaks a key assumption for bisect, which is used by"},{"line_number":1941,"context_line":"            # utils.find_namespace with shard ranges."},{"line_number":1942,"context_line":"            params \u003d []"},{"line_number":1943,"context_line":"            if states is None:"},{"line_number":1944,"context_line":"                # TODO: add test case for this."},{"line_number":1945,"context_line":"                sql \u003d \u0027\u0027\u0027"},{"line_number":1946,"context_line":"                SELECT name, lower, upper FROM %s"}],"source_content_type":"text/x-python","patch_set":8,"id":"1bd99278_271c641d","line":1943,"in_reply_to":"f069a1f2_fa3489d2","updated":"2023-08-30 02:56:43.000000000","message":"Done","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"4143d4bd5bbabf83ce395cd1329155481e56a09b","unresolved":true,"context_lines":[{"line_number":1945,"context_line":"                sql \u003d \u0027\u0027\u0027"},{"line_number":1946,"context_line":"                SELECT name, lower, upper FROM %s"},{"line_number":1947,"context_line":"                WHERE deleted \u003d 0 AND name !\u003d ?"},{"line_number":1948,"context_line":"                ORDER BY upper, state, lower, name"},{"line_number":1949,"context_line":"                \u0027\u0027\u0027 % (SHARD_RANGE_TABLE)"},{"line_number":1950,"context_line":"            else:"},{"line_number":1951,"context_line":"                params \u003d list(states)"}],"source_content_type":"text/x-python","patch_set":8,"id":"e2705a84_9961d5e9","line":1948,"range":{"start_line":1948,"start_character":16,"end_line":1948,"end_character":31},"updated":"2023-08-25 15:13:14.000000000","message":"I don\u0027t think this works for upper\u003d\"\"\n\nA Namespace converts upper\u003d\"\" to a Namespace.MaxBound which always sorts after any other string, whereas \"\" sorts before any non-empty string.","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"dba5cdc4d8314b4dc7c21ec88d42d98beac2cfbf","unresolved":false,"context_lines":[{"line_number":1945,"context_line":"                sql \u003d \u0027\u0027\u0027"},{"line_number":1946,"context_line":"                SELECT name, lower, upper FROM %s"},{"line_number":1947,"context_line":"                WHERE deleted \u003d 0 AND name !\u003d ?"},{"line_number":1948,"context_line":"                ORDER BY upper, state, lower, name"},{"line_number":1949,"context_line":"                \u0027\u0027\u0027 % (SHARD_RANGE_TABLE)"},{"line_number":1950,"context_line":"            else:"},{"line_number":1951,"context_line":"                params \u003d list(states)"}],"source_content_type":"text/x-python","patch_set":8,"id":"bfc66fcc_bd2958fa","line":1948,"range":{"start_line":1948,"start_character":16,"end_line":1948,"end_character":31},"in_reply_to":"e2705a84_9961d5e9","updated":"2023-08-30 02:56:43.000000000","message":"fixed, thanks for pointing it out!","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"d7efa6e02d77ec38b860851868e2da2de3d45bb8","unresolved":true,"context_lines":[{"line_number":1948,"context_line":"                ORDER BY upper, state, lower, name"},{"line_number":1949,"context_line":"                \u0027\u0027\u0027 % (SHARD_RANGE_TABLE)"},{"line_number":1950,"context_line":"            else:"},{"line_number":1951,"context_line":"                params \u003d list(states)"},{"line_number":1952,"context_line":"                sql \u003d \u0027\u0027\u0027"},{"line_number":1953,"context_line":"                SELECT name, lower, upper FROM %s"},{"line_number":1954,"context_line":"                WHERE deleted \u003d 0 AND name !\u003d ? AND state in (%s)"}],"source_content_type":"text/x-python","patch_set":8,"id":"2e847fcf_d74da5ce","line":1951,"range":{"start_line":1951,"start_character":25,"end_line":1951,"end_character":37},"updated":"2023-08-24 12:22:29.000000000","message":"boom! states may be just an int\n```\n\u003e               params \u003d list(states)\nE               TypeError: \u0027int\u0027 object is not iterable\n```\n\nwe need to re-use the states sanitisation code in _get_shard_range_rows - see my follow up for an idea to create a helper in utils","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"dba5cdc4d8314b4dc7c21ec88d42d98beac2cfbf","unresolved":false,"context_lines":[{"line_number":1948,"context_line":"                ORDER BY upper, state, lower, name"},{"line_number":1949,"context_line":"                \u0027\u0027\u0027 % (SHARD_RANGE_TABLE)"},{"line_number":1950,"context_line":"            else:"},{"line_number":1951,"context_line":"                params \u003d list(states)"},{"line_number":1952,"context_line":"                sql \u003d \u0027\u0027\u0027"},{"line_number":1953,"context_line":"                SELECT name, lower, upper FROM %s"},{"line_number":1954,"context_line":"                WHERE deleted \u003d 0 AND name !\u003d ? AND state in (%s)"}],"source_content_type":"text/x-python","patch_set":8,"id":"bbbf3930_a3f1d16d","line":1951,"range":{"start_line":1951,"start_character":25,"end_line":1951,"end_character":37},"in_reply_to":"2e847fcf_d74da5ce","updated":"2023-08-30 02:56:43.000000000","message":"Done","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"f35d2431a428cfec4da64577f186cb5c684ba018","unresolved":true,"context_lines":[{"line_number":1952,"context_line":"                sql \u003d \u0027\u0027\u0027"},{"line_number":1953,"context_line":"                SELECT name, lower, upper FROM %s"},{"line_number":1954,"context_line":"                WHERE deleted \u003d 0 AND name !\u003d ? AND state in (%s)"},{"line_number":1955,"context_line":"                ORDER BY upper, state, lower, name"},{"line_number":1956,"context_line":"                \u0027\u0027\u0027 % (SHARD_RANGE_TABLE, \u0027,\u0027.join(\u0027?\u0027 * len(states)))"},{"line_number":1957,"context_line":"            try:"},{"line_number":1958,"context_line":"                data \u003d conn.execute(sql, [self.path] + params)"}],"source_content_type":"text/x-python","patch_set":8,"id":"1f3dbee9_4fe9c6ac","line":1955,"updated":"2023-08-24 07:12:15.000000000","message":"ohh, nice! moved into sqlite!","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"d9f2b72057b8621e8971a513b7a78e1d82a5689a","unresolved":false,"context_lines":[{"line_number":1952,"context_line":"                sql \u003d \u0027\u0027\u0027"},{"line_number":1953,"context_line":"                SELECT name, lower, upper FROM %s"},{"line_number":1954,"context_line":"                WHERE deleted \u003d 0 AND name !\u003d ? AND state in (%s)"},{"line_number":1955,"context_line":"                ORDER BY upper, state, lower, name"},{"line_number":1956,"context_line":"                \u0027\u0027\u0027 % (SHARD_RANGE_TABLE, \u0027,\u0027.join(\u0027?\u0027 * len(states)))"},{"line_number":1957,"context_line":"            try:"},{"line_number":1958,"context_line":"                data \u003d conn.execute(sql, [self.path] + params)"}],"source_content_type":"text/x-python","patch_set":8,"id":"84954f9b_f2e6f06a","line":1955,"in_reply_to":"1f3dbee9_4fe9c6ac","updated":"2023-08-25 04:21:08.000000000","message":"Ack","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"f35d2431a428cfec4da64577f186cb5c684ba018","unresolved":true,"context_lines":[{"line_number":1957,"context_line":"            try:"},{"line_number":1958,"context_line":"                data \u003d conn.execute(sql, [self.path] + params)"},{"line_number":1959,"context_line":"                data.row_factory \u003d None"},{"line_number":1960,"context_line":"                namespaces \u003d [Namespace(*row) for row in data]"},{"line_number":1961,"context_line":"            except sqlite3.OperationalError as err:"},{"line_number":1962,"context_line":"                if (\u0027no such table: %s\u0027 % SHARD_RANGE_TABLE) in str(err):"},{"line_number":1963,"context_line":"                    return False"}],"source_content_type":"text/x-python","patch_set":8,"id":"6e83176c_b60d3375","line":1960,"updated":"2023-08-24 07:12:15.000000000","message":"Oh cool, they\u0027re Namespace objects now. Does that slow anything down?","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"d9f2b72057b8621e8971a513b7a78e1d82a5689a","unresolved":false,"context_lines":[{"line_number":1957,"context_line":"            try:"},{"line_number":1958,"context_line":"                data \u003d conn.execute(sql, [self.path] + params)"},{"line_number":1959,"context_line":"                data.row_factory \u003d None"},{"line_number":1960,"context_line":"                namespaces \u003d [Namespace(*row) for row in data]"},{"line_number":1961,"context_line":"            except sqlite3.OperationalError as err:"},{"line_number":1962,"context_line":"                if (\u0027no such table: %s\u0027 % SHARD_RANGE_TABLE) in str(err):"},{"line_number":1963,"context_line":"                    return False"}],"source_content_type":"text/x-python","patch_set":8,"id":"f3e49972_7c9fcbfb","line":1960,"in_reply_to":"6e83176c_b60d3375","updated":"2023-08-25 04:21:08.000000000","message":"It did a tiny little bit, only ~2%, so it\u0027s okay to initialize the Namespace objects here. And moving sorting to sqlite makes it ~5% faster, I will have detail numbers shared in another doc.","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"d7efa6e02d77ec38b860851868e2da2de3d45bb8","unresolved":true,"context_lines":[{"line_number":1960,"context_line":"                namespaces \u003d [Namespace(*row) for row in data]"},{"line_number":1961,"context_line":"            except sqlite3.OperationalError as err:"},{"line_number":1962,"context_line":"                if (\u0027no such table: %s\u0027 % SHARD_RANGE_TABLE) in str(err):"},{"line_number":1963,"context_line":"                    return False"},{"line_number":1964,"context_line":"                else:"},{"line_number":1965,"context_line":"                    raise"},{"line_number":1966,"context_line":""}],"source_content_type":"text/x-python","patch_set":8,"id":"230ff874_af24e3c7","line":1963,"range":{"start_line":1963,"start_character":20,"end_line":1963,"end_character":32},"updated":"2023-08-24 12:22:29.000000000","message":"return []\n\nwe have to return an empty list, right?","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"dba5cdc4d8314b4dc7c21ec88d42d98beac2cfbf","unresolved":false,"context_lines":[{"line_number":1960,"context_line":"                namespaces \u003d [Namespace(*row) for row in data]"},{"line_number":1961,"context_line":"            except sqlite3.OperationalError as err:"},{"line_number":1962,"context_line":"                if (\u0027no such table: %s\u0027 % SHARD_RANGE_TABLE) in str(err):"},{"line_number":1963,"context_line":"                    return False"},{"line_number":1964,"context_line":"                else:"},{"line_number":1965,"context_line":"                    raise"},{"line_number":1966,"context_line":""}],"source_content_type":"text/x-python","patch_set":8,"id":"73816a79_40c939f0","line":1963,"range":{"start_line":1963,"start_character":20,"end_line":1963,"end_character":32},"in_reply_to":"230ff874_af24e3c7","updated":"2023-08-30 02:56:43.000000000","message":"Done","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"d7efa6e02d77ec38b860851868e2da2de3d45bb8","unresolved":true,"context_lines":[{"line_number":1964,"context_line":"                else:"},{"line_number":1965,"context_line":"                    raise"},{"line_number":1966,"context_line":""},{"line_number":1967,"context_line":"        if not namespaces:"},{"line_number":1968,"context_line":"            return namespaces"},{"line_number":1969,"context_line":""},{"line_number":1970,"context_line":"        # If there is a gap in the last, insert a modified copy of own shard"},{"line_number":1971,"context_line":"        # range to fill any gap between the end of any found and the upper"}],"source_content_type":"text/x-python","patch_set":8,"id":"485e2a9a_e2196ddb","line":1968,"range":{"start_line":1967,"start_character":8,"end_line":1968,"end_character":29},"updated":"2023-08-24 12:22:29.000000000","message":"I\u0027m not sure this is correct: get_shard_ranges will return a filler for the entire namespace","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"dba5cdc4d8314b4dc7c21ec88d42d98beac2cfbf","unresolved":false,"context_lines":[{"line_number":1964,"context_line":"                else:"},{"line_number":1965,"context_line":"                    raise"},{"line_number":1966,"context_line":""},{"line_number":1967,"context_line":"        if not namespaces:"},{"line_number":1968,"context_line":"            return namespaces"},{"line_number":1969,"context_line":""},{"line_number":1970,"context_line":"        # If there is a gap in the last, insert a modified copy of own shard"},{"line_number":1971,"context_line":"        # range to fill any gap between the end of any found and the upper"}],"source_content_type":"text/x-python","patch_set":8,"id":"74ec43ba_618a3360","line":1968,"range":{"start_line":1967,"start_character":8,"end_line":1968,"end_character":29},"in_reply_to":"485e2a9a_e2196ddb","updated":"2023-08-30 02:56:43.000000000","message":"Done","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"d7efa6e02d77ec38b860851868e2da2de3d45bb8","unresolved":true,"context_lines":[{"line_number":1967,"context_line":"        if not namespaces:"},{"line_number":1968,"context_line":"            return namespaces"},{"line_number":1969,"context_line":""},{"line_number":1970,"context_line":"        # If there is a gap in the last, insert a modified copy of own shard"},{"line_number":1971,"context_line":"        # range to fill any gap between the end of any found and the upper"},{"line_number":1972,"context_line":"        # bound of own shard range. Gaps enclosed within the found shard"},{"line_number":1973,"context_line":"        # ranges are not filled."}],"source_content_type":"text/x-python","patch_set":8,"id":"22baf728_1729057c","line":1970,"updated":"2023-08-24 12:22:29.000000000","message":"I think this should be conditional on a fill_gaps argument, same as get_shard_ranges. Otherwise, the interface only really makes sense for listing or updating states, but actually supports arbitrary states.","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"dba5cdc4d8314b4dc7c21ec88d42d98beac2cfbf","unresolved":false,"context_lines":[{"line_number":1967,"context_line":"        if not namespaces:"},{"line_number":1968,"context_line":"            return namespaces"},{"line_number":1969,"context_line":""},{"line_number":1970,"context_line":"        # If there is a gap in the last, insert a modified copy of own shard"},{"line_number":1971,"context_line":"        # range to fill any gap between the end of any found and the upper"},{"line_number":1972,"context_line":"        # bound of own shard range. Gaps enclosed within the found shard"},{"line_number":1973,"context_line":"        # ranges are not filled."}],"source_content_type":"text/x-python","patch_set":8,"id":"0dfe4ffd_e033bb84","line":1970,"in_reply_to":"22baf728_1729057c","updated":"2023-08-30 02:56:43.000000000","message":"Added a fill_gaps argument and related test cases.","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"d7efa6e02d77ec38b860851868e2da2de3d45bb8","unresolved":true,"context_lines":[{"line_number":1974,"context_line":"        own_shard_range \u003d self.get_own_shard_range()"},{"line_number":1975,"context_line":"        if namespaces:"},{"line_number":1976,"context_line":"            last_upper \u003d namespaces[-1].upper"},{"line_number":1977,"context_line":"        else:"},{"line_number":1978,"context_line":"            last_upper \u003d own_shard_range.lower"},{"line_number":1979,"context_line":"        required_upper \u003d own_shard_range.upper"},{"line_number":1980,"context_line":"        if required_upper \u003e last_upper:"}],"source_content_type":"text/x-python","patch_set":8,"id":"7e65f59f_b39814d5","line":1977,"range":{"start_line":1977,"start_character":8,"end_line":1977,"end_character":13},"updated":"2023-08-24 12:22:29.000000000","message":"line 1967 prevents this else clause being reached - but I think we *do* want to reach here","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"dba5cdc4d8314b4dc7c21ec88d42d98beac2cfbf","unresolved":false,"context_lines":[{"line_number":1974,"context_line":"        own_shard_range \u003d self.get_own_shard_range()"},{"line_number":1975,"context_line":"        if namespaces:"},{"line_number":1976,"context_line":"            last_upper \u003d namespaces[-1].upper"},{"line_number":1977,"context_line":"        else:"},{"line_number":1978,"context_line":"            last_upper \u003d own_shard_range.lower"},{"line_number":1979,"context_line":"        required_upper \u003d own_shard_range.upper"},{"line_number":1980,"context_line":"        if required_upper \u003e last_upper:"}],"source_content_type":"text/x-python","patch_set":8,"id":"302b079f_39539818","line":1977,"range":{"start_line":1977,"start_character":8,"end_line":1977,"end_character":13},"in_reply_to":"7e65f59f_b39814d5","updated":"2023-08-30 02:56:43.000000000","message":"Ack","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"bf33de1f5f106065568e0985e20cc76fff8f97e0","unresolved":true,"context_lines":[{"line_number":1720,"context_line":"                data \u003d conn.execute(sql, params)"},{"line_number":1721,"context_line":"                data.row_factory \u003d None"},{"line_number":1722,"context_line":"                # convert upper bound \u0027\u0027 to MaxBound to be sorted later."},{"line_number":1723,"context_line":"                namespaces \u003d [(row[0], row[1], Namespace.MaxBound, row[3]) if ("},{"line_number":1724,"context_line":"                    row[2] \u003d\u003d \u0027\u0027) else row for row in data]"},{"line_number":1725,"context_line":"            except sqlite3.OperationalError as err:"},{"line_number":1726,"context_line":"                if (\u0027no such table: %s\u0027 % SHARD_RANGE_TABLE) in str(err):"}],"source_content_type":"text/x-python","patch_set":12,"id":"3bd5cfb0_87550295","line":1723,"range":{"start_line":1723,"start_character":47,"end_line":1723,"end_character":65},"updated":"2023-08-31 06:46:04.000000000","message":"Nice.\n\nOn line 1721 we set the row_factory to None, which basically returns a tuple. I wonder if it would be at all quicker to use a custom row_factory rather doing another loop through to make the namespaces. Might end up being a much for muchness.\nSomething like: https://paste.opendev.org/show/821404/\n\nIf that even works :)","commit_id":"4638670d9e4dc6850b5447e2d5756be79b4c1aff"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"fd158f484a16e6ddbe812130fa865b4ebe5e7a8b","unresolved":false,"context_lines":[{"line_number":1720,"context_line":"                data \u003d conn.execute(sql, params)"},{"line_number":1721,"context_line":"                data.row_factory \u003d None"},{"line_number":1722,"context_line":"                # convert upper bound \u0027\u0027 to MaxBound to be sorted later."},{"line_number":1723,"context_line":"                namespaces \u003d [(row[0], row[1], Namespace.MaxBound, row[3]) if ("},{"line_number":1724,"context_line":"                    row[2] \u003d\u003d \u0027\u0027) else row for row in data]"},{"line_number":1725,"context_line":"            except sqlite3.OperationalError as err:"},{"line_number":1726,"context_line":"                if (\u0027no such table: %s\u0027 % SHARD_RANGE_TABLE) in str(err):"}],"source_content_type":"text/x-python","patch_set":12,"id":"60718bcd_0dcc5737","line":1723,"range":{"start_line":1723,"start_character":47,"end_line":1723,"end_character":65},"in_reply_to":"3bd5cfb0_87550295","updated":"2023-08-31 21:06:25.000000000","message":"Great idea, I did benchmarking and saw an additional improvement of 3.7 X. Awesome!","commit_id":"4638670d9e4dc6850b5447e2d5756be79b4c1aff"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"c15c91b11bc946d098b3669e8b7532fa4a96d0e1","unresolved":true,"context_lines":[{"line_number":1685,"context_line":"            if (\u0027no such table: %s\u0027 % SHARD_RANGE_TABLE) not in str(err):"},{"line_number":1686,"context_line":"                raise"},{"line_number":1687,"context_line":""},{"line_number":1688,"context_line":"    def get_namespaces(self, states\u003dNone, fill_gaps\u003dFalse):"},{"line_number":1689,"context_line":"        \"\"\""},{"line_number":1690,"context_line":"        Returns a list of persisted namespaces per input parameters."},{"line_number":1691,"context_line":""}],"source_content_type":"text/x-python","patch_set":17,"id":"88de62f0_ef265e84","line":1688,"updated":"2023-09-05 04:59:38.000000000","message":"From the proxy side, I\u0027m looking to use namespaces for basically everything. On an object put we may attempt to get only shards that \"includes\" an object. Which isn\u0027t supported by this or the container server change.\n\nWe do container listing, which use marker and end_marker, but I\u0027m pretty sure we only end up using that to find the objects in a shard, not applied to getting shards themselves. So I think those are ok.\n\nMaybe the includes option to further filter namespaces would be good too.. This doesn\u0027t have to be a blocker for this patch, it could be a follow up, or I could just add it from the server patch side. Sorry for late notice, only something I stumbled upon while attempting the proxy side of the patch.\n\nMaybe the obj side of the proxy server could just keep using record_type\u003dshard, includes should limit the response so not be too expensive. Unless we\u0027re intercepting that and populating the cache, which would mean no, we should always do it with namespaces.","commit_id":"6b34ca53368256d5f7109d8f184fe4a5fc6abf2b"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"3919b5f331df82d39b944f6ec52506ac2ae65a31","unresolved":true,"context_lines":[{"line_number":1685,"context_line":"            if (\u0027no such table: %s\u0027 % SHARD_RANGE_TABLE) not in str(err):"},{"line_number":1686,"context_line":"                raise"},{"line_number":1687,"context_line":""},{"line_number":1688,"context_line":"    def get_namespaces(self, states\u003dNone, fill_gaps\u003dFalse):"},{"line_number":1689,"context_line":"        \"\"\""},{"line_number":1690,"context_line":"        Returns a list of persisted namespaces per input parameters."},{"line_number":1691,"context_line":""}],"source_content_type":"text/x-python","patch_set":17,"id":"78e003a4_6e1ee6ca","line":1688,"in_reply_to":"0b3d8cd4_2ac2dde1","updated":"2023-09-06 00:57:44.000000000","message":"Matt and Alistair, when you talked about \"this method should support includes\", are you talking about that this method should support \"marker\" and \"end_marker\"? also, the most pain we are seeing right now is caused by getting \"updating\" shard ranges from container, I feel we can attack \"updating\" shard ranges first, and then add back other supports later?","commit_id":"6b34ca53368256d5f7109d8f184fe4a5fc6abf2b"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"aceacc504d338a7772301488c765f1f0333a9b1a","unresolved":true,"context_lines":[{"line_number":1685,"context_line":"            if (\u0027no such table: %s\u0027 % SHARD_RANGE_TABLE) not in str(err):"},{"line_number":1686,"context_line":"                raise"},{"line_number":1687,"context_line":""},{"line_number":1688,"context_line":"    def get_namespaces(self, states\u003dNone, fill_gaps\u003dFalse):"},{"line_number":1689,"context_line":"        \"\"\""},{"line_number":1690,"context_line":"        Returns a list of persisted namespaces per input parameters."},{"line_number":1691,"context_line":""}],"source_content_type":"text/x-python","patch_set":17,"id":"8d792c7c_d90fc10f","line":1688,"in_reply_to":"4db69bc0_c163a092","updated":"2023-09-11 07:05:03.000000000","message":"Well I was hoping for includes and the other support so my patch for fix up the obj controller side too.","commit_id":"6b34ca53368256d5f7109d8f184fe4a5fc6abf2b"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"4621264a31586fe757b1846045e261e71f449a32","unresolved":true,"context_lines":[{"line_number":1685,"context_line":"            if (\u0027no such table: %s\u0027 % SHARD_RANGE_TABLE) not in str(err):"},{"line_number":1686,"context_line":"                raise"},{"line_number":1687,"context_line":""},{"line_number":1688,"context_line":"    def get_namespaces(self, states\u003dNone, fill_gaps\u003dFalse):"},{"line_number":1689,"context_line":"        \"\"\""},{"line_number":1690,"context_line":"        Returns a list of persisted namespaces per input parameters."},{"line_number":1691,"context_line":""}],"source_content_type":"text/x-python","patch_set":17,"id":"4db69bc0_c163a092","line":1688,"in_reply_to":"78e003a4_6e1ee6ca","updated":"2023-09-11 03:07:01.000000000","message":"okay, understood what you mean now. Since we came to this patch mostly for performance optimization of proxy getting \u0027updating\u0027 shard ranges, I will implement the complete support for \u0027listing\u0027 case in the follow-up patch.\nAdded comments \"Note, for state \u0027listing\u0027, only Sharded container is supported for now, since \u0027marker\u0027 and \u0027end_marker\u0027 will need to be implemented.\"","commit_id":"6b34ca53368256d5f7109d8f184fe4a5fc6abf2b"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"3ae9f087bde3d7f7c2bb592b87773a81dbd5fe1c","unresolved":true,"context_lines":[{"line_number":1685,"context_line":"            if (\u0027no such table: %s\u0027 % SHARD_RANGE_TABLE) not in str(err):"},{"line_number":1686,"context_line":"                raise"},{"line_number":1687,"context_line":""},{"line_number":1688,"context_line":"    def get_namespaces(self, states\u003dNone, fill_gaps\u003dFalse):"},{"line_number":1689,"context_line":"        \"\"\""},{"line_number":1690,"context_line":"        Returns a list of persisted namespaces per input parameters."},{"line_number":1691,"context_line":""}],"source_content_type":"text/x-python","patch_set":17,"id":"0b3d8cd4_2ac2dde1","line":1688,"in_reply_to":"88de62f0_ef265e84","updated":"2023-09-05 14:53:40.000000000","message":"@Matt are you referring to this case? https://github.com/openstack/swift/blob/e5d730dc56c0c75c9f3a8b7f6119dcc638ccded0/swift/proxy/controllers/obj.py#L324-L342\n\nWe could leave that as it is (proxy fetches a single shard range from backend) but that would be unfortunate IMHO : two conditions/request constructs to code for in proxy 😞\n\nSo I think I agree that this method should support ``includes``.\n\nAlso, the listing case does use marker/end_marker *sometimes*... When requesting shard ranges for a listing, the proxy sends ``x-backend-override-shard-name-filter\u003dsharded``. If, and only if, the container DB is in ``sharded`` state, that causes marker and end_marker to be ignored. But if the container is sharding then marker and end_marker are respected (and the proxy will not cache the results).\n\nWe could change that so that the backend always returns all namespaces, even when sharding. We don\u0027t expect containers to be in sharding state for long, and we don\u0027t them to have many shard ranges while they are, so the extra bytes on the wire is not significant, and the proxy could just drop the unwanted namespaces that lie outside (marker, end_marker). But, that is more disruptive than having get_namespaces handle marker \u0026 end_marker: we know how to write that code (copy from get_shard_ranges!).\n\nFurthermore, if recheck_listing_shard_ranges \u003d 0 or there is no memcache, then the proxy never sends ``x-backend-override-shard-name-filter\u003dsharded`` , so we need to support marker/end_marker for that case.","commit_id":"6b34ca53368256d5f7109d8f184fe4a5fc6abf2b"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bff0a54e68a76090810057a6fac976d6d5e3e745","unresolved":false,"context_lines":[{"line_number":1685,"context_line":"            if (\u0027no such table: %s\u0027 % SHARD_RANGE_TABLE) not in str(err):"},{"line_number":1686,"context_line":"                raise"},{"line_number":1687,"context_line":""},{"line_number":1688,"context_line":"    def get_namespaces(self, states\u003dNone, fill_gaps\u003dFalse):"},{"line_number":1689,"context_line":"        \"\"\""},{"line_number":1690,"context_line":"        Returns a list of persisted namespaces per input parameters."},{"line_number":1691,"context_line":""}],"source_content_type":"text/x-python","patch_set":17,"id":"6e8db1c0_2e48ba3f","line":1688,"in_reply_to":"8d792c7c_d90fc10f","updated":"2024-01-04 16:20:07.000000000","message":"Done","commit_id":"6b34ca53368256d5f7109d8f184fe4a5fc6abf2b"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"3ae9f087bde3d7f7c2bb592b87773a81dbd5fe1c","unresolved":true,"context_lines":[{"line_number":1701,"context_line":"        included_states \u003d transform_to_set(states)"},{"line_number":1702,"context_line":"        with self.get() as conn:"},{"line_number":1703,"context_line":"            # Namespace only needs \u0027name\u0027, \u0027lower\u0027 and \u0027upper\u0027, but the query"},{"line_number":1704,"context_line":"            # also need to include \u0027state\u0027 to be used the sorting after. And"},{"line_number":1705,"context_line":"            # sorting can\u0027t be done within SQLite since the value for maximum"},{"line_number":1706,"context_line":"            # upper bound is an empty string."},{"line_number":1707,"context_line":"            if included_states is None:"}],"source_content_type":"text/x-python","patch_set":17,"id":"6e6acdb4_bfa82eff","line":1704,"range":{"start_line":1704,"start_character":43,"end_line":1704,"end_character":71},"updated":"2023-09-05 14:53:40.000000000","message":"nit: wrong grammar.\n\n\"to be used when sorting after\", or \"to be used when subesequently sorting the rows\"","commit_id":"6b34ca53368256d5f7109d8f184fe4a5fc6abf2b"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"3919b5f331df82d39b944f6ec52506ac2ae65a31","unresolved":false,"context_lines":[{"line_number":1701,"context_line":"        included_states \u003d transform_to_set(states)"},{"line_number":1702,"context_line":"        with self.get() as conn:"},{"line_number":1703,"context_line":"            # Namespace only needs \u0027name\u0027, \u0027lower\u0027 and \u0027upper\u0027, but the query"},{"line_number":1704,"context_line":"            # also need to include \u0027state\u0027 to be used the sorting after. And"},{"line_number":1705,"context_line":"            # sorting can\u0027t be done within SQLite since the value for maximum"},{"line_number":1706,"context_line":"            # upper bound is an empty string."},{"line_number":1707,"context_line":"            if included_states is None:"}],"source_content_type":"text/x-python","patch_set":17,"id":"ba89844f_cd8bc324","line":1704,"range":{"start_line":1704,"start_character":43,"end_line":1704,"end_character":71},"in_reply_to":"6e6acdb4_bfa82eff","updated":"2023-09-06 00:57:44.000000000","message":"Done","commit_id":"6b34ca53368256d5f7109d8f184fe4a5fc6abf2b"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"3ae9f087bde3d7f7c2bb592b87773a81dbd5fe1c","unresolved":true,"context_lines":[{"line_number":1731,"context_line":"        def sort_key(namespace):"},{"line_number":1732,"context_line":"            # Use Namespace.MaxBound() for upper bound \u0027\u0027, this will allow this"},{"line_number":1733,"context_line":"            # record to be sorted correctly per ShardRange.sort_key_order()."},{"line_number":1734,"context_line":"            upper_key \u003d namespace[2] if namespace[2] else Namespace.MaxBound()"},{"line_number":1735,"context_line":"            return ShardRange.sort_key_order(name\u003dnamespace[0],"},{"line_number":1736,"context_line":"                                             lower\u003dnamespace[1],"},{"line_number":1737,"context_line":"                                             upper\u003dupper_key,"}],"source_content_type":"text/x-python","patch_set":17,"id":"12f7e114_2dc71df2","line":1734,"updated":"2023-09-05 14:53:40.000000000","message":"maybe this could be implemented in ShardRange.sort_key_order - it is always the desired behaviour for upper","commit_id":"6b34ca53368256d5f7109d8f184fe4a5fc6abf2b"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"eb02e84623e5173d951d13bd70511677d112089e","unresolved":false,"context_lines":[{"line_number":1731,"context_line":"        def sort_key(namespace):"},{"line_number":1732,"context_line":"            # Use Namespace.MaxBound() for upper bound \u0027\u0027, this will allow this"},{"line_number":1733,"context_line":"            # record to be sorted correctly per ShardRange.sort_key_order()."},{"line_number":1734,"context_line":"            upper_key \u003d namespace[2] if namespace[2] else Namespace.MaxBound()"},{"line_number":1735,"context_line":"            return ShardRange.sort_key_order(name\u003dnamespace[0],"},{"line_number":1736,"context_line":"                                             lower\u003dnamespace[1],"},{"line_number":1737,"context_line":"                                             upper\u003dupper_key,"}],"source_content_type":"text/x-python","patch_set":17,"id":"3fd4180b_78b4bcf4","line":1734,"in_reply_to":"12f7e114_2dc71df2","updated":"2023-09-09 01:11:47.000000000","message":"Done","commit_id":"6b34ca53368256d5f7109d8f184fe4a5fc6abf2b"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"3ae9f087bde3d7f7c2bb592b87773a81dbd5fe1c","unresolved":true,"context_lines":[{"line_number":1740,"context_line":"        # Convert the record tuples to Namespace objects."},{"line_number":1741,"context_line":"        namespaces \u003d [Namespace(row[0], row[1], row[2]) for row in namespaces]"},{"line_number":1742,"context_line":""},{"line_number":1743,"context_line":"        if fill_gaps:"},{"line_number":1744,"context_line":"            # If there is a gap in the last, insert a modified copy of own"},{"line_number":1745,"context_line":"            # shard range to fill any gap between the end of any found and the"},{"line_number":1746,"context_line":"            # upper bound of own shard range. Gaps enclosed within the found"}],"source_content_type":"text/x-python","patch_set":17,"id":"4ae1b93a_498b9c18","line":1743,"updated":"2023-09-05 14:53:40.000000000","message":"off-topic/follow-up: there\u0027s an optimisation here, and in get_shard_ranges...\n\nif namespaces[-1].upper \u003d\u003d MaxBound then there is never going to be a filler_sr, so short-circuit to return and save doing a get_own_shard_range","commit_id":"6b34ca53368256d5f7109d8f184fe4a5fc6abf2b"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"eb02e84623e5173d951d13bd70511677d112089e","unresolved":false,"context_lines":[{"line_number":1740,"context_line":"        # Convert the record tuples to Namespace objects."},{"line_number":1741,"context_line":"        namespaces \u003d [Namespace(row[0], row[1], row[2]) for row in namespaces]"},{"line_number":1742,"context_line":""},{"line_number":1743,"context_line":"        if fill_gaps:"},{"line_number":1744,"context_line":"            # If there is a gap in the last, insert a modified copy of own"},{"line_number":1745,"context_line":"            # shard range to fill any gap between the end of any found and the"},{"line_number":1746,"context_line":"            # upper bound of own shard range. Gaps enclosed within the found"}],"source_content_type":"text/x-python","patch_set":17,"id":"d31bc322_6f2d0674","line":1743,"in_reply_to":"4ae1b93a_498b9c18","updated":"2023-09-09 01:11:47.000000000","message":"Done","commit_id":"6b34ca53368256d5f7109d8f184fe4a5fc6abf2b"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"dd2266c879ea38eae3d5fd5fe501e62a1982679c","unresolved":true,"context_lines":[{"line_number":1705,"context_line":"            # also need to include \u0027state\u0027 to be used when subesequently"},{"line_number":1706,"context_line":"            # sorting the rows. And the sorting can\u0027t be done within SQLite"},{"line_number":1707,"context_line":"            # since the value for maximum upper bound is an empty string."},{"line_number":1708,"context_line":"            if included_states is None:"},{"line_number":1709,"context_line":"                params \u003d [self.path]"},{"line_number":1710,"context_line":"                sql \u003d \u0027\u0027\u0027"},{"line_number":1711,"context_line":"                SELECT name, lower, upper, state FROM %s"}],"source_content_type":"text/x-python","patch_set":22,"id":"bea41d32_4b963792","line":1708,"updated":"2023-09-13 19:57:42.000000000","message":"I think we want `if included_states:` here -- what happens if we get an empty set?\n\nAlternatively, maybe that should trigger an early `return []`?","commit_id":"b105e043017904fcf520fd7cdad5c07aba47e76f"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"693fbccfb57938a9ccae311ba3d53701e72a9a9d","unresolved":false,"context_lines":[{"line_number":1705,"context_line":"            # also need to include \u0027state\u0027 to be used when subesequently"},{"line_number":1706,"context_line":"            # sorting the rows. And the sorting can\u0027t be done within SQLite"},{"line_number":1707,"context_line":"            # since the value for maximum upper bound is an empty string."},{"line_number":1708,"context_line":"            if included_states is None:"},{"line_number":1709,"context_line":"                params \u003d [self.path]"},{"line_number":1710,"context_line":"                sql \u003d \u0027\u0027\u0027"},{"line_number":1711,"context_line":"                SELECT name, lower, upper, state FROM %s"}],"source_content_type":"text/x-python","patch_set":22,"id":"062e38e0_c88d6691","line":1708,"in_reply_to":"bea41d32_4b963792","updated":"2023-09-21 04:37:28.000000000","message":"even if we get an empty set, we are going to insert the \"own shard range\" at here: https://review.opendev.org/c/openstack/swift/+/890470/23/swift/container/backend.py#1758","commit_id":"b105e043017904fcf520fd7cdad5c07aba47e76f"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"dd2266c879ea38eae3d5fd5fe501e62a1982679c","unresolved":false,"context_lines":[{"line_number":1736,"context_line":"                                             state\u003dnamespace[3])"},{"line_number":1737,"context_line":"        namespaces.sort(key\u003dsort_key)"},{"line_number":1738,"context_line":"        # Convert the record tuples to Namespace objects."},{"line_number":1739,"context_line":"        namespaces \u003d [Namespace(row[0], row[1], row[2]) for row in namespaces]"},{"line_number":1740,"context_line":""},{"line_number":1741,"context_line":"        if fill_gaps and not ("},{"line_number":1742,"context_line":"            namespaces and namespaces[-1].upper \u003d\u003d Namespace.MaxBound"}],"source_content_type":"text/x-python","patch_set":22,"id":"51a6e0d9_35868ef0","line":1739,"updated":"2023-09-13 19:57:42.000000000","message":"Right; there\u0027s no reference to `row[3]` since `Namespace`s don\u0027t track state. This is also why we want to sort *before* converting to `Namespace`s.\n\nHow crazy would it be for us to pass `state\u003dNone` or something to `sort_key_order` so we don\u0027t even need to pull `state` out of the DB? We shouldn\u0027t have any overlaps to resolve anyway, right? Even if there were, I\u0027m not sure a client would be able to decide which namespace was appropriate, since we don\u0027t even expose the state to them.","commit_id":"b105e043017904fcf520fd7cdad5c07aba47e76f"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"dd2266c879ea38eae3d5fd5fe501e62a1982679c","unresolved":true,"context_lines":[{"line_number":1744,"context_line":"            # If there is a gap in the last, insert a modified copy of own"},{"line_number":1745,"context_line":"            # shard range to fill any gap between the end of any found and the"},{"line_number":1746,"context_line":"            # upper bound of own shard range. Gaps enclosed within the found"},{"line_number":1747,"context_line":"            # shard ranges are not filled."},{"line_number":1748,"context_line":"            own_shard_range \u003d self.get_own_shard_range()"},{"line_number":1749,"context_line":"            if namespaces:"},{"line_number":1750,"context_line":"                last_upper \u003d namespaces[-1].upper"}],"source_content_type":"text/x-python","patch_set":22,"id":"676e3b0c_725a13d4","line":1747,"updated":"2023-09-13 19:57:42.000000000","message":"Interesting... this is a quite notable difference from getting shard ranges, no?","commit_id":"b105e043017904fcf520fd7cdad5c07aba47e76f"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"693fbccfb57938a9ccae311ba3d53701e72a9a9d","unresolved":false,"context_lines":[{"line_number":1744,"context_line":"            # If there is a gap in the last, insert a modified copy of own"},{"line_number":1745,"context_line":"            # shard range to fill any gap between the end of any found and the"},{"line_number":1746,"context_line":"            # upper bound of own shard range. Gaps enclosed within the found"},{"line_number":1747,"context_line":"            # shard ranges are not filled."},{"line_number":1748,"context_line":"            own_shard_range \u003d self.get_own_shard_range()"},{"line_number":1749,"context_line":"            if namespaces:"},{"line_number":1750,"context_line":"                last_upper \u003d namespaces[-1].upper"}],"source_content_type":"text/x-python","patch_set":22,"id":"9bd49a5b_4679034d","line":1747,"in_reply_to":"676e3b0c_725a13d4","updated":"2023-09-21 04:37:28.000000000","message":"it\u0027s same behavior as getting shard ranges, see here: https://github.com/NVIDIA/swift/blob/master/swift/container/backend.py#L1885","commit_id":"b105e043017904fcf520fd7cdad5c07aba47e76f"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"dd2266c879ea38eae3d5fd5fe501e62a1982679c","unresolved":false,"context_lines":[{"line_number":1864,"context_line":"        with self.maybe_get(connection) as conn:"},{"line_number":1865,"context_line":"            defaults \u003d set()"},{"line_number":1866,"context_line":"            attempts \u003d len(default_values) + 1"},{"line_number":1867,"context_line":"            while attempts:"},{"line_number":1868,"context_line":"                attempts -\u003d 1"},{"line_number":1869,"context_line":"                try:"},{"line_number":1870,"context_line":"                    return do_query(conn, defaults)"}],"source_content_type":"text/x-python","patch_set":22,"id":"95134616_8fa8e41f","line":1867,"updated":"2023-09-13 19:57:42.000000000","message":"Right; we don\u0027t need these multiple attempts when getting namespaces because the potentially-missing columns aren\u0027t used.","commit_id":"b105e043017904fcf520fd7cdad5c07aba47e76f"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":true,"context_lines":[{"line_number":1734,"context_line":"        if exclude_others and not include_own:"},{"line_number":1735,"context_line":"            return []"},{"line_number":1736,"context_line":""},{"line_number":1737,"context_line":"        included_states \u003d set()"},{"line_number":1738,"context_line":"        if isinstance(states, (list, tuple, set)):"},{"line_number":1739,"context_line":"            included_states.update(states)"},{"line_number":1740,"context_line":"        elif states is not None:"}],"source_content_type":"text/x-python","patch_set":27,"id":"40ec880b_ba9f3f7f","side":"PARENT","line":1737,"updated":"2023-10-02 22:23:29.000000000","message":"seems like empty set is totally reasonable","commit_id":"5555980fb5fce79140efc0c8ab8a730bd552733c"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"bcb52432e997cd207866f8b22a7c55a1f0a34204","unresolved":false,"context_lines":[{"line_number":1734,"context_line":"        if exclude_others and not include_own:"},{"line_number":1735,"context_line":"            return []"},{"line_number":1736,"context_line":""},{"line_number":1737,"context_line":"        included_states \u003d set()"},{"line_number":1738,"context_line":"        if isinstance(states, (list, tuple, set)):"},{"line_number":1739,"context_line":"            included_states.update(states)"},{"line_number":1740,"context_line":"        elif states is not None:"}],"source_content_type":"text/x-python","patch_set":27,"id":"5126d964_524ed8d7","side":"PARENT","line":1737,"in_reply_to":"40ec880b_ba9f3f7f","updated":"2023-10-31 16:11:06.000000000","message":"Ack","commit_id":"5555980fb5fce79140efc0c8ab8a730bd552733c"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":true,"context_lines":[{"line_number":1691,"context_line":""},{"line_number":1692,"context_line":"        :param states: if specified, restricts the returned list to namespaces"},{"line_number":1693,"context_line":"            that have the given state(s); can be a list of ints or a single"},{"line_number":1694,"context_line":"            int; For state \u0027listing\u0027, only Sharded container is supported for"},{"line_number":1695,"context_line":"            now, since \u0027marker\u0027 and \u0027end_marker\u0027 will need to be implemented."},{"line_number":1696,"context_line":"        :param fill_gaps: if True, insert a modified copy of own shard range to"},{"line_number":1697,"context_line":"            fill any gap between the end of any found shard ranges and the"}],"source_content_type":"text/x-python","patch_set":27,"id":"afc44e60_3ad1bd2e","line":1694,"updated":"2023-10-02 22:23:29.000000000","message":"we don\u0027t have to carry forward the stupid - do we actually USE both `states\u003d10` and `states\u003d[10, ...]`???","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"bcb52432e997cd207866f8b22a7c55a1f0a34204","unresolved":false,"context_lines":[{"line_number":1691,"context_line":""},{"line_number":1692,"context_line":"        :param states: if specified, restricts the returned list to namespaces"},{"line_number":1693,"context_line":"            that have the given state(s); can be a list of ints or a single"},{"line_number":1694,"context_line":"            int; For state \u0027listing\u0027, only Sharded container is supported for"},{"line_number":1695,"context_line":"            now, since \u0027marker\u0027 and \u0027end_marker\u0027 will need to be implemented."},{"line_number":1696,"context_line":"        :param fill_gaps: if True, insert a modified copy of own shard range to"},{"line_number":1697,"context_line":"            fill any gap between the end of any found shard ranges and the"}],"source_content_type":"text/x-python","patch_set":27,"id":"0a5448c5_d92d039d","line":1694,"in_reply_to":"afc44e60_3ad1bd2e","updated":"2023-10-31 16:11:06.000000000","message":"yeah, this comment is not relevant anymore after a few iterations.","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":true,"context_lines":[{"line_number":1711,"context_line":"                SELECT name, lower, upper, state FROM %s"},{"line_number":1712,"context_line":"                WHERE deleted \u003d 0 AND name !\u003d ?"},{"line_number":1713,"context_line":"                \u0027\u0027\u0027 % (SHARD_RANGE_TABLE)"},{"line_number":1714,"context_line":"            else:"},{"line_number":1715,"context_line":"                params \u003d [self.path] + list(included_states)"},{"line_number":1716,"context_line":"                sql \u003d \u0027\u0027\u0027"},{"line_number":1717,"context_line":"                SELECT name, lower, upper, state FROM %s"}],"source_content_type":"text/x-python","patch_set":27,"id":"06a1e51c_4848729d","line":1714,"updated":"2023-10-02 22:23:29.000000000","message":"maybe a little better if we always setup this query and params - and the conditionally *extend* params and \n\n```\nsql +\u003d \u0027 AND state in (%s)\u0027\n```\n... when there\u0027s included_states","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"bcb52432e997cd207866f8b22a7c55a1f0a34204","unresolved":false,"context_lines":[{"line_number":1711,"context_line":"                SELECT name, lower, upper, state FROM %s"},{"line_number":1712,"context_line":"                WHERE deleted \u003d 0 AND name !\u003d ?"},{"line_number":1713,"context_line":"                \u0027\u0027\u0027 % (SHARD_RANGE_TABLE)"},{"line_number":1714,"context_line":"            else:"},{"line_number":1715,"context_line":"                params \u003d [self.path] + list(included_states)"},{"line_number":1716,"context_line":"                sql \u003d \u0027\u0027\u0027"},{"line_number":1717,"context_line":"                SELECT name, lower, upper, state FROM %s"}],"source_content_type":"text/x-python","patch_set":27,"id":"fee3567f_02efb5e7","line":1714,"in_reply_to":"06a1e51c_4848729d","updated":"2023-10-31 16:11:06.000000000","message":"Done","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":true,"context_lines":[{"line_number":1719,"context_line":"                \u0027\u0027\u0027 % (SHARD_RANGE_TABLE, \u0027,\u0027.join(\u0027?\u0027 * len(included_states)))"},{"line_number":1720,"context_line":"            try:"},{"line_number":1721,"context_line":"                data \u003d conn.execute(sql, params)"},{"line_number":1722,"context_line":"                data.row_factory \u003d None"},{"line_number":1723,"context_line":"                namespaces \u003d [row for row in data]"},{"line_number":1724,"context_line":"            except sqlite3.OperationalError as err:"},{"line_number":1725,"context_line":"                if (\u0027no such table: %s\u0027 % SHARD_RANGE_TABLE) in str(err):"}],"source_content_type":"text/x-python","patch_set":27,"id":"e9b74988_070e9f89","line":1722,"updated":"2023-10-02 22:23:29.000000000","message":"and there\u0027s some big significant performance reason to avoid a row_factory \u003d Namespace?  Does that go away if NameSpace is a NamedTuple or some other simple primative?  \n\nmaybe we\u0027re just cargo-culting row_factory \u003d None from 1-year-old sqlite code elsewhere in the codebase:\nhttps://docs.python.org/2/library/sqlite3.html#sqlite3.Connection.row_factory","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"4fbcf228954a81390f569f70a531c5bff3f910ed","unresolved":true,"context_lines":[{"line_number":1719,"context_line":"                \u0027\u0027\u0027 % (SHARD_RANGE_TABLE, \u0027,\u0027.join(\u0027?\u0027 * len(included_states)))"},{"line_number":1720,"context_line":"            try:"},{"line_number":1721,"context_line":"                data \u003d conn.execute(sql, params)"},{"line_number":1722,"context_line":"                data.row_factory \u003d None"},{"line_number":1723,"context_line":"                namespaces \u003d [row for row in data]"},{"line_number":1724,"context_line":"            except sqlite3.OperationalError as err:"},{"line_number":1725,"context_line":"                if (\u0027no such table: %s\u0027 % SHARD_RANGE_TABLE) in str(err):"}],"source_content_type":"text/x-python","patch_set":27,"id":"efb1f103_cc7c09a6","line":1722,"in_reply_to":"e9b74988_070e9f89","updated":"2023-10-03 17:56:56.000000000","message":"\u003e maybe we\u0027re just cargo-culting row_factory \u003d None from 1-year-old sqlite code elsewhere in the codebase\n\nAlmost certainly.","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"ab133fd538b104017d417ae79e5e983aa083a419","unresolved":false,"context_lines":[{"line_number":1719,"context_line":"                \u0027\u0027\u0027 % (SHARD_RANGE_TABLE, \u0027,\u0027.join(\u0027?\u0027 * len(included_states)))"},{"line_number":1720,"context_line":"            try:"},{"line_number":1721,"context_line":"                data \u003d conn.execute(sql, params)"},{"line_number":1722,"context_line":"                data.row_factory \u003d None"},{"line_number":1723,"context_line":"                namespaces \u003d [row for row in data]"},{"line_number":1724,"context_line":"            except sqlite3.OperationalError as err:"},{"line_number":1725,"context_line":"                if (\u0027no such table: %s\u0027 % SHARD_RANGE_TABLE) in str(err):"}],"source_content_type":"text/x-python","patch_set":27,"id":"d30d164c_2fde5438","line":1722,"in_reply_to":"efb1f103_cc7c09a6","updated":"2023-10-04 04:32:17.000000000","message":"we have to sort the four-attribute tuple(name, lower, upper, state) before we generate Namespace Objects which don\u0027t have \"state\". But I did try a customized row factory at below to save one list looping, to my surprise its performance actually was lower than None row_factory.\nhttps://review.opendev.org/c/openstack/swift/+/890470/13/swift/container/backend.py#1761","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":true,"context_lines":[{"line_number":1730,"context_line":"        # Sort those namespaces in order, note that each namespace record also"},{"line_number":1731,"context_line":"        # include additional attribute \u0027state\u0027."},{"line_number":1732,"context_line":"        def sort_key(namespace):"},{"line_number":1733,"context_line":"            return ShardRange.sort_key_order(name\u003dnamespace[0],"},{"line_number":1734,"context_line":"                                             lower\u003dnamespace[1],"},{"line_number":1735,"context_line":"                                             upper\u003dnamespace[2],"},{"line_number":1736,"context_line":"                                             state\u003dnamespace[3])"}],"source_content_type":"text/x-python","patch_set":27,"id":"244ec47d_5fe7e94e","line":1733,"updated":"2023-10-02 22:23:29.000000000","message":"gah, this is why people like static methods - it\u0027s a namespace thing - but python already has module level functions\n\nif we had `common.utils.shard_range.ShardRange` it would seem totally reasonable to use `common.utils.shard_range.sort_key_order`\n\nThere\u0027s also a weird coupling now from the keys name, lower, upper, state - I don\u0027t *expect* that to change, but if it did we\u0027d have to update the signature of sort_key_order and all the callers.  What I think we\u0027d really want","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"e77e3479d3023573255ad217d4a8ef9b220111a7","unresolved":false,"context_lines":[{"line_number":1730,"context_line":"        # Sort those namespaces in order, note that each namespace record also"},{"line_number":1731,"context_line":"        # include additional attribute \u0027state\u0027."},{"line_number":1732,"context_line":"        def sort_key(namespace):"},{"line_number":1733,"context_line":"            return ShardRange.sort_key_order(name\u003dnamespace[0],"},{"line_number":1734,"context_line":"                                             lower\u003dnamespace[1],"},{"line_number":1735,"context_line":"                                             upper\u003dnamespace[2],"},{"line_number":1736,"context_line":"                                             state\u003dnamespace[3])"}],"source_content_type":"text/x-python","patch_set":27,"id":"9d7d5837_8ec26bd1","line":1733,"in_reply_to":"003e9612_0635097b","updated":"2024-01-12 14:29:07.000000000","message":"Acknowledged","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"75f2bbd98f81540e59137c0a128facb8e38d61f8","unresolved":false,"context_lines":[{"line_number":1730,"context_line":"        # Sort those namespaces in order, note that each namespace record also"},{"line_number":1731,"context_line":"        # include additional attribute \u0027state\u0027."},{"line_number":1732,"context_line":"        def sort_key(namespace):"},{"line_number":1733,"context_line":"            return ShardRange.sort_key_order(name\u003dnamespace[0],"},{"line_number":1734,"context_line":"                                             lower\u003dnamespace[1],"},{"line_number":1735,"context_line":"                                             upper\u003dnamespace[2],"},{"line_number":1736,"context_line":"                                             state\u003dnamespace[3])"}],"source_content_type":"text/x-python","patch_set":27,"id":"37b16334_1fe99347","line":1733,"in_reply_to":"003e9612_0635097b","updated":"2024-01-11 00:53:22.000000000","message":"thanks for benchmarking different alternatives. I also don\u0027t want to create another ``sort_key`` function here with duplicate logic, and those order logic is super important in order to return correct shard ranges.","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"e367c6989582ab6ef04e286014b46f4c1023eb77","unresolved":true,"context_lines":[{"line_number":1730,"context_line":"        # Sort those namespaces in order, note that each namespace record also"},{"line_number":1731,"context_line":"        # include additional attribute \u0027state\u0027."},{"line_number":1732,"context_line":"        def sort_key(namespace):"},{"line_number":1733,"context_line":"            return ShardRange.sort_key_order(name\u003dnamespace[0],"},{"line_number":1734,"context_line":"                                             lower\u003dnamespace[1],"},{"line_number":1735,"context_line":"                                             upper\u003dnamespace[2],"},{"line_number":1736,"context_line":"                                             state\u003dnamespace[3])"}],"source_content_type":"text/x-python","patch_set":27,"id":"412065c5_0d94ca38","line":1733,"in_reply_to":"244ec47d_5fe7e94e","updated":"2023-11-13 23:38:06.000000000","message":"I feel like there was more you wanted to say here...","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"74e9ae2e078e75459ac35188586733c97d906994","unresolved":true,"context_lines":[{"line_number":1730,"context_line":"        # Sort those namespaces in order, note that each namespace record also"},{"line_number":1731,"context_line":"        # include additional attribute \u0027state\u0027."},{"line_number":1732,"context_line":"        def sort_key(namespace):"},{"line_number":1733,"context_line":"            return ShardRange.sort_key_order(name\u003dnamespace[0],"},{"line_number":1734,"context_line":"                                             lower\u003dnamespace[1],"},{"line_number":1735,"context_line":"                                             upper\u003dnamespace[2],"},{"line_number":1736,"context_line":"                                             state\u003dnamespace[3])"}],"source_content_type":"text/x-python","patch_set":27,"id":"003e9612_0635097b","line":1733,"in_reply_to":"412065c5_0d94ca38","updated":"2024-01-10 14:27:08.000000000","message":"I played with a few alternative ways of avoiding the sort_key_order method, e.g. having a row factory that build namedtuples that can be passed to ShardRange.sort_key, or subclassing Namespace to add a state attr. In each case I saw performance penalties ~5%-10%.","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":true,"context_lines":[{"line_number":1736,"context_line":"                                             state\u003dnamespace[3])"},{"line_number":1737,"context_line":"        namespaces.sort(key\u003dsort_key)"},{"line_number":1738,"context_line":"        # Convert the record tuples to Namespace objects."},{"line_number":1739,"context_line":"        namespaces \u003d [Namespace(row[0], row[1], row[2]) for row in namespaces]"},{"line_number":1740,"context_line":""},{"line_number":1741,"context_line":"        if fill_gaps and not ("},{"line_number":1742,"context_line":"            namespaces and namespaces[-1].upper \u003d\u003d Namespace.MaxBound"}],"source_content_type":"text/x-python","patch_set":27,"id":"e31ae06a_c3fe13e1","line":1739,"updated":"2023-10-02 22:23:29.000000000","message":"we end up converting to Namespace after all!?   We SORT by state, but then throw it out.  This is the third time iterating this 10K+ object list in the last 10 lines.\n\nI\u0027m going to play with custom collate function:\n\nhttps://docs.python.org/2/library/sqlite3.html#sqlite3.Connection.create_collationo","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"75f2bbd98f81540e59137c0a128facb8e38d61f8","unresolved":false,"context_lines":[{"line_number":1736,"context_line":"                                             state\u003dnamespace[3])"},{"line_number":1737,"context_line":"        namespaces.sort(key\u003dsort_key)"},{"line_number":1738,"context_line":"        # Convert the record tuples to Namespace objects."},{"line_number":1739,"context_line":"        namespaces \u003d [Namespace(row[0], row[1], row[2]) for row in namespaces]"},{"line_number":1740,"context_line":""},{"line_number":1741,"context_line":"        if fill_gaps and not ("},{"line_number":1742,"context_line":"            namespaces and namespaces[-1].upper \u003d\u003d Namespace.MaxBound"}],"source_content_type":"text/x-python","patch_set":27,"id":"168a81c7_268c1959","line":1739,"in_reply_to":"aafe48fb_1b4c5e4e","updated":"2024-01-11 00:53:22.000000000","message":"Acknowledged","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"e77e3479d3023573255ad217d4a8ef9b220111a7","unresolved":false,"context_lines":[{"line_number":1736,"context_line":"                                             state\u003dnamespace[3])"},{"line_number":1737,"context_line":"        namespaces.sort(key\u003dsort_key)"},{"line_number":1738,"context_line":"        # Convert the record tuples to Namespace objects."},{"line_number":1739,"context_line":"        namespaces \u003d [Namespace(row[0], row[1], row[2]) for row in namespaces]"},{"line_number":1740,"context_line":""},{"line_number":1741,"context_line":"        if fill_gaps and not ("},{"line_number":1742,"context_line":"            namespaces and namespaces[-1].upper \u003d\u003d Namespace.MaxBound"}],"source_content_type":"text/x-python","patch_set":27,"id":"5ccf2849_28fbf8f0","line":1739,"in_reply_to":"aafe48fb_1b4c5e4e","updated":"2024-01-12 14:29:07.000000000","message":"Acknowledged","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"74e9ae2e078e75459ac35188586733c97d906994","unresolved":true,"context_lines":[{"line_number":1736,"context_line":"                                             state\u003dnamespace[3])"},{"line_number":1737,"context_line":"        namespaces.sort(key\u003dsort_key)"},{"line_number":1738,"context_line":"        # Convert the record tuples to Namespace objects."},{"line_number":1739,"context_line":"        namespaces \u003d [Namespace(row[0], row[1], row[2]) for row in namespaces]"},{"line_number":1740,"context_line":""},{"line_number":1741,"context_line":"        if fill_gaps and not ("},{"line_number":1742,"context_line":"            namespaces and namespaces[-1].upper \u003d\u003d Namespace.MaxBound"}],"source_content_type":"text/x-python","patch_set":27,"id":"aafe48fb_1b4c5e4e","line":1739,"in_reply_to":"b6be71fc_d3359f24","updated":"2024-01-10 14:27:08.000000000","message":"We can\u0027t use cmp_to_key *yet* https://docs.python.org/3/library/functools.html#functools.cmp_to_key\n\nNew in version 3.2.","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"4fbcf228954a81390f569f70a531c5bff3f910ed","unresolved":true,"context_lines":[{"line_number":1736,"context_line":"                                             state\u003dnamespace[3])"},{"line_number":1737,"context_line":"        namespaces.sort(key\u003dsort_key)"},{"line_number":1738,"context_line":"        # Convert the record tuples to Namespace objects."},{"line_number":1739,"context_line":"        namespaces \u003d [Namespace(row[0], row[1], row[2]) for row in namespaces]"},{"line_number":1740,"context_line":""},{"line_number":1741,"context_line":"        if fill_gaps and not ("},{"line_number":1742,"context_line":"            namespaces and namespaces[-1].upper \u003d\u003d Namespace.MaxBound"}],"source_content_type":"text/x-python","patch_set":27,"id":"b6be71fc_d3359f24","line":1739,"in_reply_to":"e31ae06a_c3fe13e1","updated":"2023-10-03 17:56:56.000000000","message":"I\u0027m digging the `create_collation` idea... and it has me wondering whether we actually need to sort shard ranges in all the places we currently do -- it seems like we do a decent bit of sorting as we\u0027re combining shard ranges just before writing to the DB, for example.\n\nI think us involving `upper` in the sorting might complicate things, though, since we write down `\u0027\u0027` for the max.\n\nOne downside: I\u0027m not sure we\u0027ll be able to get away from having our existing `sort_key`. Maybe we could at least use [`functools.cmp_to_key`](https://docs.python.org/3/library/functools.html) to avoid having two separate implementations?","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":true,"context_lines":[{"line_number":1740,"context_line":""},{"line_number":1741,"context_line":"        if fill_gaps and not ("},{"line_number":1742,"context_line":"            namespaces and namespaces[-1].upper \u003d\u003d Namespace.MaxBound"},{"line_number":1743,"context_line":"        ):"},{"line_number":1744,"context_line":"            # If there is a gap in the last, insert a modified copy of own"},{"line_number":1745,"context_line":"            # shard range to fill any gap between the end of any found and the"},{"line_number":1746,"context_line":"            # upper bound of own shard range. Gaps enclosed within the found"}],"source_content_type":"text/x-python","patch_set":27,"id":"bad9e67f_ff25cff7","line":1743,"updated":"2023-10-02 22:23:29.000000000","message":"is this the only place where fill_gaps has any effect?  the special case during initial sharding where the last shard-range isn\u0027t the end of the namespace?","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"bcb52432e997cd207866f8b22a7c55a1f0a34204","unresolved":false,"context_lines":[{"line_number":1740,"context_line":""},{"line_number":1741,"context_line":"        if fill_gaps and not ("},{"line_number":1742,"context_line":"            namespaces and namespaces[-1].upper \u003d\u003d Namespace.MaxBound"},{"line_number":1743,"context_line":"        ):"},{"line_number":1744,"context_line":"            # If there is a gap in the last, insert a modified copy of own"},{"line_number":1745,"context_line":"            # shard range to fill any gap between the end of any found and the"},{"line_number":1746,"context_line":"            # upper bound of own shard range. Gaps enclosed within the found"}],"source_content_type":"text/x-python","patch_set":27,"id":"2b0cb8fd_50ae8b01","line":1743,"in_reply_to":"bad9e67f_ff25cff7","updated":"2023-10-31 16:11:06.000000000","message":"yes, fill_gaps will be true if \u0027states\u0027 is either \u0027listing\u0027 or \u0027updating\u0027, then the returned ShardRanges/Namespaces have to fill in any gap if found, in order to have proxy PUT to function well and find a corresponding shard to upload objects.","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":true,"context_lines":[{"line_number":1786,"context_line":"            are applied (e.g. ``exclude_others`` and ``include_deleted``)."},{"line_number":1787,"context_line":"        :param include_deleted: include rows marked as deleted."},{"line_number":1788,"context_line":"        :param states: include only rows matching the given state(s); can be an"},{"line_number":1789,"context_line":"            int or a list of ints."},{"line_number":1790,"context_line":"        :param include_own: boolean that governs whether the row whose name"},{"line_number":1791,"context_line":"            matches the broker\u0027s path is included in the returned list. If"},{"line_number":1792,"context_line":"            True, that row is included unless it is excluded by other"}],"source_content_type":"text/x-python","patch_set":27,"id":"06b36766_b597264c","line":1789,"updated":"2023-10-02 22:23:29.000000000","message":"I guess that `states\u003d10` is pretty un-ambigious; but we should have just made you pass `states\u003d[10]` if that\u0027s what you wanted","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8a24e200d9cb30696c694717a3c20781843d2652","unresolved":false,"context_lines":[{"line_number":1786,"context_line":"            are applied (e.g. ``exclude_others`` and ``include_deleted``)."},{"line_number":1787,"context_line":"        :param include_deleted: include rows marked as deleted."},{"line_number":1788,"context_line":"        :param states: include only rows matching the given state(s); can be an"},{"line_number":1789,"context_line":"            int or a list of ints."},{"line_number":1790,"context_line":"        :param include_own: boolean that governs whether the row whose name"},{"line_number":1791,"context_line":"            matches the broker\u0027s path is included in the returned list. If"},{"line_number":1792,"context_line":"            True, that row is included unless it is excluded by other"}],"source_content_type":"text/x-python","patch_set":27,"id":"b404d210_bb99d538","line":1789,"in_reply_to":"06b36766_b597264c","updated":"2023-10-31 19:21:20.000000000","message":"I find this ambiguity very distasteful in dynamic languages where there\u0027s no first class support for polymorphic method dispatch\n\nIn the face of ambiguity, refuse the temptation to guess.\nThere should be one-- and preferably only one --obvious way to do it.\n\nI would suggest that it\u0027s more reasonable to break from precedence than carry forward a bad pattern.","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":true,"context_lines":[{"line_number":1810,"context_line":"        if exclude_others and not include_own:"},{"line_number":1811,"context_line":"            return []"},{"line_number":1812,"context_line":""},{"line_number":1813,"context_line":"        included_states \u003d transform_to_set(states)"},{"line_number":1814,"context_line":""},{"line_number":1815,"context_line":"        # defaults to be used when legacy db\u0027s are missing columns"},{"line_number":1816,"context_line":"        default_values \u003d {\u0027reported\u0027: 0,"}],"source_content_type":"text/x-python","patch_set":27,"id":"53bc1975_2af69cd7","line":1813,"updated":"2023-10-02 22:23:29.000000000","message":"it seems like this is really the only change to the existing shard-range record type code path","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"bcb52432e997cd207866f8b22a7c55a1f0a34204","unresolved":false,"context_lines":[{"line_number":1810,"context_line":"        if exclude_others and not include_own:"},{"line_number":1811,"context_line":"            return []"},{"line_number":1812,"context_line":""},{"line_number":1813,"context_line":"        included_states \u003d transform_to_set(states)"},{"line_number":1814,"context_line":""},{"line_number":1815,"context_line":"        # defaults to be used when legacy db\u0027s are missing columns"},{"line_number":1816,"context_line":"        default_values \u003d {\u0027reported\u0027: 0,"}],"source_content_type":"text/x-python","patch_set":27,"id":"b6d3b87d_9a2c7cb4","line":1813,"in_reply_to":"53bc1975_2af69cd7","updated":"2023-10-31 16:11:06.000000000","message":"Ack","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":true,"context_lines":[{"line_number":1822,"context_line":"            params \u003d []"},{"line_number":1823,"context_line":"            if not include_deleted:"},{"line_number":1824,"context_line":"                conditions.append(\u0027deleted\u003d0\u0027)"},{"line_number":1825,"context_line":"            if included_states:"},{"line_number":1826,"context_line":"                conditions.append(\u0027state in (%s)\u0027 % \u0027,\u0027.join("},{"line_number":1827,"context_line":"                    \u0027?\u0027 * len(included_states)))"},{"line_number":1828,"context_line":"                params.extend(included_states)"}],"source_content_type":"text/x-python","patch_set":27,"id":"6fbd5d38_7b975447","line":1825,"updated":"2023-10-02 22:23:29.000000000","message":"and I think having it turn `None \u003d\u003e None` instead of `None \u003d\u003e set()` is ok","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"bcb52432e997cd207866f8b22a7c55a1f0a34204","unresolved":false,"context_lines":[{"line_number":1822,"context_line":"            params \u003d []"},{"line_number":1823,"context_line":"            if not include_deleted:"},{"line_number":1824,"context_line":"                conditions.append(\u0027deleted\u003d0\u0027)"},{"line_number":1825,"context_line":"            if included_states:"},{"line_number":1826,"context_line":"                conditions.append(\u0027state in (%s)\u0027 % \u0027,\u0027.join("},{"line_number":1827,"context_line":"                    \u0027?\u0027 * len(included_states)))"},{"line_number":1828,"context_line":"                params.extend(included_states)"}],"source_content_type":"text/x-python","patch_set":27,"id":"f7fbd6a7_6b0a79c8","line":1825,"in_reply_to":"6fbd5d38_7b975447","updated":"2023-10-31 16:11:06.000000000","message":"will convert it to return set() which you suggested in other comments.","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":true,"context_lines":[{"line_number":1973,"context_line":"                marker\u003dmarker, end_marker\u003dend_marker, includes\u003dincludes,"},{"line_number":1974,"context_line":"                include_deleted\u003dinclude_deleted, states\u003dstates,"},{"line_number":1975,"context_line":"                include_own\u003dinclude_own, exclude_others\u003dexclude_others)]"},{"line_number":1976,"context_line":"        shard_ranges.sort(key\u003dShardRange.sort_key)"},{"line_number":1977,"context_line":"        if includes:"},{"line_number":1978,"context_line":"            return shard_ranges[:1] if shard_ranges else []"},{"line_number":1979,"context_line":""}],"source_content_type":"text/x-python","patch_set":27,"id":"06e14cff_44f9c527","line":1976,"updated":"2023-10-02 22:23:29.000000000","message":"it seems like get_namespaces is trying to avoid a sort(key\u003dNamespace.sort_key)","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"4fbcf228954a81390f569f70a531c5bff3f910ed","unresolved":true,"context_lines":[{"line_number":1973,"context_line":"                marker\u003dmarker, end_marker\u003dend_marker, includes\u003dincludes,"},{"line_number":1974,"context_line":"                include_deleted\u003dinclude_deleted, states\u003dstates,"},{"line_number":1975,"context_line":"                include_own\u003dinclude_own, exclude_others\u003dexclude_others)]"},{"line_number":1976,"context_line":"        shard_ranges.sort(key\u003dShardRange.sort_key)"},{"line_number":1977,"context_line":"        if includes:"},{"line_number":1978,"context_line":"            return shard_ranges[:1] if shard_ranges else []"},{"line_number":1979,"context_line":""}],"source_content_type":"text/x-python","patch_set":27,"id":"bf34b0f0_43966a29","line":1976,"in_reply_to":"06e14cff_44f9c527","updated":"2023-10-03 17:56:56.000000000","message":"I think the trouble is that we don\u0027t have (maybe *can\u0027t have*?) a `Namespace.sort_key` -- `Namespace`s don\u0027t have enough information attached. Trying to write one that doesn\u0027t include `state` would lead to some edge cases where getting shard ranges and getting namespaces would give you results with different orders -- which in turn could mean different listings, different update targets, etc.\n\nMaybe that\u0027s already somewhat possible, though -- `Namespace`s *do* have `__lt__`/`__gt__` already...","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"ab133fd538b104017d417ae79e5e983aa083a419","unresolved":false,"context_lines":[{"line_number":1973,"context_line":"                marker\u003dmarker, end_marker\u003dend_marker, includes\u003dincludes,"},{"line_number":1974,"context_line":"                include_deleted\u003dinclude_deleted, states\u003dstates,"},{"line_number":1975,"context_line":"                include_own\u003dinclude_own, exclude_others\u003dexclude_others)]"},{"line_number":1976,"context_line":"        shard_ranges.sort(key\u003dShardRange.sort_key)"},{"line_number":1977,"context_line":"        if includes:"},{"line_number":1978,"context_line":"            return shard_ranges[:1] if shard_ranges else []"},{"line_number":1979,"context_line":""}],"source_content_type":"text/x-python","patch_set":27,"id":"c95d5f71_5fe114ac","line":1976,"in_reply_to":"bf34b0f0_43966a29","updated":"2023-10-04 04:32:17.000000000","message":"Ack","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8a24e200d9cb30696c694717a3c20781843d2652","unresolved":true,"context_lines":[{"line_number":1691,"context_line":""},{"line_number":1692,"context_line":"        :param states: if specified, restricts the returned list to namespaces"},{"line_number":1693,"context_line":"            that have the given state(s); can be a list of ints or a single"},{"line_number":1694,"context_line":"            int."},{"line_number":1695,"context_line":"        :param fill_gaps: if True, insert a modified copy of own shard range to"},{"line_number":1696,"context_line":"            fill any gap between the end of any found shard ranges and the"},{"line_number":1697,"context_line":"            upper bound of own shard range. Gaps enclosed within the found"}],"source_content_type":"text/x-python","patch_set":29,"id":"6e898d75_2ba5dfa7","line":1694,"updated":"2023-10-31 19:21:20.000000000","message":"\u003e can be a list of ints or a single int\n\nI know that _get_shard_range_rows works this way; but this behavior is not necessary in this new method since we control *all* call *new* sites and we only ever pass in a collection of values.  The transform_to_set method is entirely unecessary, please stop this pattern and remove it.","commit_id":"95dcd77bf6b77f83ab3471d40b0f658ee5c5014f"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"7515a0f55dd5433e624c281ec73257d47b15060d","unresolved":true,"context_lines":[{"line_number":1691,"context_line":""},{"line_number":1692,"context_line":"        :param states: if specified, restricts the returned list to namespaces"},{"line_number":1693,"context_line":"            that have the given state(s); can be a list of ints or a single"},{"line_number":1694,"context_line":"            int."},{"line_number":1695,"context_line":"        :param fill_gaps: if True, insert a modified copy of own shard range to"},{"line_number":1696,"context_line":"            fill any gap between the end of any found shard ranges and the"},{"line_number":1697,"context_line":"            upper bound of own shard range. Gaps enclosed within the found"}],"source_content_type":"text/x-python","patch_set":29,"id":"9135b691_b6d45467","line":1694,"in_reply_to":"6e898d75_2ba5dfa7","updated":"2023-10-31 20:36:52.000000000","message":"\u003e I know that _get_shard_range_rows works this way\n\nBy extension, `get_shard_ranges` gets this behavior -- it seems weird to me that we would have it for one but not the other. I maybe wouldn\u0027t be opposed to us dropping it for both, though... I don\u0027t know that we ever actually use the `states\u003dsingle_value` calling style under `swift/`","commit_id":"95dcd77bf6b77f83ab3471d40b0f658ee5c5014f"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"a57812bb8061973f9c43b94ddf758ccac9d7d357","unresolved":false,"context_lines":[{"line_number":1691,"context_line":""},{"line_number":1692,"context_line":"        :param states: if specified, restricts the returned list to namespaces"},{"line_number":1693,"context_line":"            that have the given state(s); can be a list of ints or a single"},{"line_number":1694,"context_line":"            int."},{"line_number":1695,"context_line":"        :param fill_gaps: if True, insert a modified copy of own shard range to"},{"line_number":1696,"context_line":"            fill any gap between the end of any found shard ranges and the"},{"line_number":1697,"context_line":"            upper bound of own shard range. Gaps enclosed within the found"}],"source_content_type":"text/x-python","patch_set":29,"id":"b0271a28_130c373b","line":1694,"in_reply_to":"9135b691_b6d45467","updated":"2023-11-02 20:54:39.000000000","message":"I also don\u0027t believe it to be used, and having code we don\u0027t use is never helpful; it just slows us down.\n\nHowever, consistency is a sufficently reasonable justification given we\u0027re not going to pull out the behavior from get_shard_ranges.\n\nAlthough... I might explore what making them both only accept lists would look like as a diff (at least to this code base) and we can consider how worried we are somewhat public nature of ContainerBroker.get_shard_ranges WRT out-of-tree tools.","commit_id":"95dcd77bf6b77f83ab3471d40b0f658ee5c5014f"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"e367c6989582ab6ef04e286014b46f4c1023eb77","unresolved":false,"context_lines":[{"line_number":1737,"context_line":"        # Convert the record tuples to Namespace objects."},{"line_number":1738,"context_line":"        namespaces \u003d [Namespace(row[0], row[1], row[2]) for row in namespaces]"},{"line_number":1739,"context_line":""},{"line_number":1740,"context_line":"        if fill_gaps and not ("},{"line_number":1741,"context_line":"            namespaces and namespaces[-1].upper \u003d\u003d Namespace.MaxBound"},{"line_number":1742,"context_line":"        ):"},{"line_number":1743,"context_line":"            # If there is a gap in the last, insert a modified copy of own"}],"source_content_type":"text/x-python","patch_set":38,"id":"91579685_b35fb48c","line":1740,"updated":"2023-11-13 23:38:06.000000000","message":"It bugs me a little how similar-but-not-close-enough-for-any-reuse this is from the `fill_gaps` logic in `get_shard_ranges`... but like I said, I don\u0027t see any obvious way to unify them.","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"e367c6989582ab6ef04e286014b46f4c1023eb77","unresolved":true,"context_lines":[{"line_number":1738,"context_line":"        namespaces \u003d [Namespace(row[0], row[1], row[2]) for row in namespaces]"},{"line_number":1739,"context_line":""},{"line_number":1740,"context_line":"        if fill_gaps and not ("},{"line_number":1741,"context_line":"            namespaces and namespaces[-1].upper \u003d\u003d Namespace.MaxBound"},{"line_number":1742,"context_line":"        ):"},{"line_number":1743,"context_line":"            # If there is a gap in the last, insert a modified copy of own"},{"line_number":1744,"context_line":"            # shard range to fill any gap between the end of any found and the"}],"source_content_type":"text/x-python","patch_set":38,"id":"c0ab9e6a_dbe2e0d7","line":1741,"range":{"start_line":1741,"start_character":12,"end_line":1741,"end_character":69},"updated":"2023-11-13 23:38:06.000000000","message":"I was about to say\n\n\u003e I wonder how much of whatever performance improvement we\u0027re seeing is because we\u0027re avoiding the `get_own_shard_range()` call in the common, happy path...\n\n...but then I realized that the `upper` check will always be `False`! We want something more like `namespaces[-1].upper \u003d\u003d Namespace.MAX`","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"8d8195542db644352565bd617030845015b81ec1","unresolved":false,"context_lines":[{"line_number":1738,"context_line":"        namespaces \u003d [Namespace(row[0], row[1], row[2]) for row in namespaces]"},{"line_number":1739,"context_line":""},{"line_number":1740,"context_line":"        if fill_gaps and not ("},{"line_number":1741,"context_line":"            namespaces and namespaces[-1].upper \u003d\u003d Namespace.MaxBound"},{"line_number":1742,"context_line":"        ):"},{"line_number":1743,"context_line":"            # If there is a gap in the last, insert a modified copy of own"},{"line_number":1744,"context_line":"            # shard range to fill any gap between the end of any found and the"}],"source_content_type":"text/x-python","patch_set":38,"id":"415e5af3_d274dff3","line":1741,"range":{"start_line":1741,"start_character":12,"end_line":1741,"end_character":69},"in_reply_to":"0a281fa8_c6cab0ff","updated":"2023-11-14 21:25:35.000000000","message":"OIC, I would need to use type() or isinstance() for checking types. Thanks for verifying it!","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"3f4812df4982d0832906bdbd476dc32a24c0a01f","unresolved":true,"context_lines":[{"line_number":1738,"context_line":"        namespaces \u003d [Namespace(row[0], row[1], row[2]) for row in namespaces]"},{"line_number":1739,"context_line":""},{"line_number":1740,"context_line":"        if fill_gaps and not ("},{"line_number":1741,"context_line":"            namespaces and namespaces[-1].upper \u003d\u003d Namespace.MaxBound"},{"line_number":1742,"context_line":"        ):"},{"line_number":1743,"context_line":"            # If there is a gap in the last, insert a modified copy of own"},{"line_number":1744,"context_line":"            # shard range to fill any gap between the end of any found and the"}],"source_content_type":"text/x-python","patch_set":38,"id":"0a281fa8_c6cab0ff","line":1741,"range":{"start_line":1741,"start_character":12,"end_line":1741,"end_character":69},"in_reply_to":"46717b10_765ce7a4","updated":"2023-11-14 18:57:34.000000000","message":"Nope:\n```\n\u003e\u003e\u003e ShardRange.MAX \u003d\u003d ShardRange.MaxBound\nFalse\n```\n`MaxBound` is the type; `MAX` is the instance.","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"6641867462956b530bfcb3af9268e1a8f3f322eb","unresolved":true,"context_lines":[{"line_number":1738,"context_line":"        namespaces \u003d [Namespace(row[0], row[1], row[2]) for row in namespaces]"},{"line_number":1739,"context_line":""},{"line_number":1740,"context_line":"        if fill_gaps and not ("},{"line_number":1741,"context_line":"            namespaces and namespaces[-1].upper \u003d\u003d Namespace.MaxBound"},{"line_number":1742,"context_line":"        ):"},{"line_number":1743,"context_line":"            # If there is a gap in the last, insert a modified copy of own"},{"line_number":1744,"context_line":"            # shard range to fill any gap between the end of any found and the"}],"source_content_type":"text/x-python","patch_set":38,"id":"46717b10_765ce7a4","line":1741,"range":{"start_line":1741,"start_character":12,"end_line":1741,"end_character":69},"in_reply_to":"c0ab9e6a_dbe2e0d7","updated":"2023-11-14 01:59:11.000000000","message":"```namespaces[-1].upper \u003d\u003d Namespace.MaxBound``` will check if the type of ``namespaces[-1].upper`` is ``Namespace.MaxBound``, IIUC it should also work?\n\nI made this optimization after the main change, I remember benchmarking showed a small improvement.","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"e367c6989582ab6ef04e286014b46f4c1023eb77","unresolved":true,"context_lines":[{"line_number":1748,"context_line":"            if namespaces:"},{"line_number":1749,"context_line":"                last_upper \u003d namespaces[-1].upper"},{"line_number":1750,"context_line":"            else:"},{"line_number":1751,"context_line":"                last_upper \u003d own_shard_range.lower"},{"line_number":1752,"context_line":"            required_upper \u003d own_shard_range.upper"},{"line_number":1753,"context_line":"            if required_upper \u003e last_upper:"},{"line_number":1754,"context_line":"                filler_sr \u003d own_shard_range"}],"source_content_type":"text/x-python","patch_set":38,"id":"a40a7a2d_2e1e9228","line":1751,"updated":"2023-11-13 23:38:06.000000000","message":"Or `Namespace.MinBound`, or anything else that will make the comparison below `True`... would it maybe be more clear if we had something like\n```\nown_shard_range \u003d self.get_own_shard_range()\nif not namespaces:\n    namespaces.append(own_shard_range.to_namespace())\nelif namespaces[-1].upper \u003c own_shard_range.upper:\n    namespaces.append(Namespace(\n        own_shard_range.name,\n        lower\u003dnamespaces[-1].upper,\n        upper\u003down_shard_range.upper))\n# else, namespaces already go to the end of own_shard_range\n```\n? \\*shrug\\*","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"b15761c0e0e2d7e69f93b97e5e7b7847078fdb4a","unresolved":false,"context_lines":[{"line_number":1748,"context_line":"            if namespaces:"},{"line_number":1749,"context_line":"                last_upper \u003d namespaces[-1].upper"},{"line_number":1750,"context_line":"            else:"},{"line_number":1751,"context_line":"                last_upper \u003d own_shard_range.lower"},{"line_number":1752,"context_line":"            required_upper \u003d own_shard_range.upper"},{"line_number":1753,"context_line":"            if required_upper \u003e last_upper:"},{"line_number":1754,"context_line":"                filler_sr \u003d own_shard_range"}],"source_content_type":"text/x-python","patch_set":38,"id":"54184210_922dfdc5","line":1751,"in_reply_to":"0e4b90cd_bbdfd8bf","updated":"2024-01-09 15:23:00.000000000","message":"Done","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bff0a54e68a76090810057a6fac976d6d5e3e745","unresolved":true,"context_lines":[{"line_number":1748,"context_line":"            if namespaces:"},{"line_number":1749,"context_line":"                last_upper \u003d namespaces[-1].upper"},{"line_number":1750,"context_line":"            else:"},{"line_number":1751,"context_line":"                last_upper \u003d own_shard_range.lower"},{"line_number":1752,"context_line":"            required_upper \u003d own_shard_range.upper"},{"line_number":1753,"context_line":"            if required_upper \u003e last_upper:"},{"line_number":1754,"context_line":"                filler_sr \u003d own_shard_range"}],"source_content_type":"text/x-python","patch_set":38,"id":"0e4b90cd_bbdfd8bf","line":1751,"in_reply_to":"a40a7a2d_2e1e9228","updated":"2024-01-04 16:20:07.000000000","message":"this gap-filling code was lifted from get_shard_ranges, but in this patch marker and end_marker support are dropped for namespaces. Once we add that later in the patch chain (and also move this to a helper method) we end up with something much closer to the get_shard_ranges case, where it is more obvious why ``last_upper \u003d own_shard_range.lower``\n\n```\n    def _make_filler_shard_range(self, namespaces, marker, end_marker):\n        if namespaces and namespaces[-1].upper \u003d\u003d Namespace.MAX:\n            return None\n\n        # Insert a modified copy of own shard range to fill any gap between the\n        # end of any found and the upper bound of own shard range. Gaps\n        # enclosed within the found shard ranges are not filled.\n        own_shard_range \u003d self.get_own_shard_range()\n        if namespaces:\n            last_upper \u003d namespaces[-1].upper\n        else:\n            last_upper \u003d max(marker or own_shard_range.lower,\n                             own_shard_range.lower)\n        required_upper \u003d min(end_marker or own_shard_range.upper,\n                             own_shard_range.upper)\n        if required_upper \u003e last_upper:\n            filler_sr \u003d own_shard_range\n            filler_sr.lower \u003d last_upper\n            filler_sr.upper \u003d required_upper\n            return filler_sr\n        else:\n            return None\n            ```\n            \nsee 900740: container: add Namespace marker/end_marker/include/reverse support. | https://review.opendev.org/c/openstack/swift/+/900740\nwhich we plan to squash into this patch","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"b15761c0e0e2d7e69f93b97e5e7b7847078fdb4a","unresolved":true,"context_lines":[{"line_number":1785,"context_line":"            return ShardRange.sort_key_order(name\u003dnamespace[0],"},{"line_number":1786,"context_line":"                                             lower\u003dnamespace[1],"},{"line_number":1787,"context_line":"                                             upper\u003dnamespace[2],"},{"line_number":1788,"context_line":"                                             state\u003dnamespace[3])"},{"line_number":1789,"context_line":"        namespaces.sort(key\u003dsort_key)"},{"line_number":1790,"context_line":"        # Convert the record tuples to Namespace objects."},{"line_number":1791,"context_line":"        namespaces \u003d [Namespace(row[0], row[1], row[2]) for row in namespaces]"}],"source_content_type":"text/x-python","patch_set":56,"id":"26f5245a_3763aa09","line":1788,"updated":"2024-01-09 15:23:00.000000000","message":"nit: this could be written ``ShardRange.sort_key_order(*namespace)``","commit_id":"e3676eff802dcde5eeabbda8d5707025f40412ad"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"75f2bbd98f81540e59137c0a128facb8e38d61f8","unresolved":false,"context_lines":[{"line_number":1785,"context_line":"            return ShardRange.sort_key_order(name\u003dnamespace[0],"},{"line_number":1786,"context_line":"                                             lower\u003dnamespace[1],"},{"line_number":1787,"context_line":"                                             upper\u003dnamespace[2],"},{"line_number":1788,"context_line":"                                             state\u003dnamespace[3])"},{"line_number":1789,"context_line":"        namespaces.sort(key\u003dsort_key)"},{"line_number":1790,"context_line":"        # Convert the record tuples to Namespace objects."},{"line_number":1791,"context_line":"        namespaces \u003d [Namespace(row[0], row[1], row[2]) for row in namespaces]"}],"source_content_type":"text/x-python","patch_set":56,"id":"a2438d3c_742d5839","line":1788,"in_reply_to":"26f5245a_3763aa09","updated":"2024-01-11 00:53:22.000000000","message":"yeah. since namespace is just a list, what if later the order of data inside that list gets changed; also, keyword arguments is more explicit.","commit_id":"e3676eff802dcde5eeabbda8d5707025f40412ad"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"b15761c0e0e2d7e69f93b97e5e7b7847078fdb4a","unresolved":true,"context_lines":[{"line_number":1802,"context_line":"        if reverse:"},{"line_number":1803,"context_line":"            namespaces.reverse()"},{"line_number":1804,"context_line":""},{"line_number":1805,"context_line":"        return namespaces"},{"line_number":1806,"context_line":""},{"line_number":1807,"context_line":"    def _get_shard_range_rows(self, connection\u003dNone, marker\u003dNone,"},{"line_number":1808,"context_line":"                              end_marker\u003dNone, includes\u003dNone,"}],"source_content_type":"text/x-python","patch_set":56,"id":"ec423128_37c43d64","line":1805,"updated":"2024-01-09 15:23:00.000000000","message":"did we ever consider returning a list of dicts (and then not needing to transform back to dicts when serializing the listing in the ContainerServer?)\n\nWe\u0027d need to convince ourselves that the Namespace._encode() wasn\u0027t necessary. AFAICT that is the one thing that constructing Namespaces gives us, and maybe some lower/upper convenience stuff in _make_filler_shard_ranges.\n\nI did a very quick\u0027n\u0027dirty benchmark that suggests there is performance improvement to gain. Might be interesting to explore? as a follow on??","commit_id":"e3676eff802dcde5eeabbda8d5707025f40412ad"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"74e9ae2e078e75459ac35188586733c97d906994","unresolved":false,"context_lines":[{"line_number":1802,"context_line":"        if reverse:"},{"line_number":1803,"context_line":"            namespaces.reverse()"},{"line_number":1804,"context_line":""},{"line_number":1805,"context_line":"        return namespaces"},{"line_number":1806,"context_line":""},{"line_number":1807,"context_line":"    def _get_shard_range_rows(self, connection\u003dNone, marker\u003dNone,"},{"line_number":1808,"context_line":"                              end_marker\u003dNone, includes\u003dNone,"}],"source_content_type":"text/x-python","patch_set":56,"id":"8489aebc_380ac46a","line":1805,"in_reply_to":"5540b0e5_241318ee","updated":"2024-01-10 14:27:08.000000000","message":"Acknowledged","commit_id":"e3676eff802dcde5eeabbda8d5707025f40412ad"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"7f7419d81f176f1e13b447e561bb43ebf264f128","unresolved":true,"context_lines":[{"line_number":1802,"context_line":"        if reverse:"},{"line_number":1803,"context_line":"            namespaces.reverse()"},{"line_number":1804,"context_line":""},{"line_number":1805,"context_line":"        return namespaces"},{"line_number":1806,"context_line":""},{"line_number":1807,"context_line":"    def _get_shard_range_rows(self, connection\u003dNone, marker\u003dNone,"},{"line_number":1808,"context_line":"                              end_marker\u003dNone, includes\u003dNone,"}],"source_content_type":"text/x-python","patch_set":56,"id":"5540b0e5_241318ee","line":1805,"in_reply_to":"ec423128_37c43d64","updated":"2024-01-09 18:59:48.000000000","message":"An earlier version of this patch did try a similar approach: ``get_namespaces`` returns a list of [namespace name, lower, upper, state] and pass it directly to json encoding before sending out to proxy server. I saw a good amount of performance improvement, 5.8X improvement v.s. 3.5X improvement of current standing patch. But the interface was not well received at that time, so I abandoned that idea. Yeah, maybe a follow up work.","commit_id":"e3676eff802dcde5eeabbda8d5707025f40412ad"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"74e9ae2e078e75459ac35188586733c97d906994","unresolved":true,"context_lines":[{"line_number":1771,"context_line":"            \u0027\u0027\u0027 % (SHARD_RANGE_TABLE, condition)"},{"line_number":1772,"context_line":"            try:"},{"line_number":1773,"context_line":"                data \u003d conn.execute(sql, params)"},{"line_number":1774,"context_line":"                data.row_factory \u003d None"},{"line_number":1775,"context_line":"                namespaces \u003d [row for row in data]"},{"line_number":1776,"context_line":"            except sqlite3.OperationalError as err:"},{"line_number":1777,"context_line":"                if (\u0027no such table: %s\u0027 % SHARD_RANGE_TABLE) in str(err):"}],"source_content_type":"text/x-python","patch_set":57,"id":"09fea404_b31cb250","line":1774,"updated":"2024-01-10 14:27:08.000000000","message":"this line is significant and gives ~5% performance gain at the API for a db with ~12k shard","commit_id":"7be1763455236ebf184d6be483798bad5e2acc9d"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"e77e3479d3023573255ad217d4a8ef9b220111a7","unresolved":true,"context_lines":[{"line_number":1771,"context_line":"            \u0027\u0027\u0027 % (SHARD_RANGE_TABLE, condition)"},{"line_number":1772,"context_line":"            try:"},{"line_number":1773,"context_line":"                data \u003d conn.execute(sql, params)"},{"line_number":1774,"context_line":"                data.row_factory \u003d None"},{"line_number":1775,"context_line":"                namespaces \u003d [row for row in data]"},{"line_number":1776,"context_line":"            except sqlite3.OperationalError as err:"},{"line_number":1777,"context_line":"                if (\u0027no such table: %s\u0027 % SHARD_RANGE_TABLE) in str(err):"}],"source_content_type":"text/x-python","patch_set":57,"id":"bdfb82c4_d243e7b8","line":1774,"in_reply_to":"09fea404_b31cb250","updated":"2024-01-12 14:29:07.000000000","message":"so the default \"is a dict and a tuple\" Row is not free, GTK!\n\nif we used a create collation to get state sorting right without selecting it we could make the row factory Namespace!\n\nhttps://gist.github.com/clayg/778c88da0ae891e4d96529d0a4cc33c8\n\n... but that turns out to be slower than the sort in pyhton 😂\n\n... although I guess we could try the row_factory \u003d lambda trick.","commit_id":"7be1763455236ebf184d6be483798bad5e2acc9d"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"75f2bbd98f81540e59137c0a128facb8e38d61f8","unresolved":false,"context_lines":[{"line_number":1771,"context_line":"            \u0027\u0027\u0027 % (SHARD_RANGE_TABLE, condition)"},{"line_number":1772,"context_line":"            try:"},{"line_number":1773,"context_line":"                data \u003d conn.execute(sql, params)"},{"line_number":1774,"context_line":"                data.row_factory \u003d None"},{"line_number":1775,"context_line":"                namespaces \u003d [row for row in data]"},{"line_number":1776,"context_line":"            except sqlite3.OperationalError as err:"},{"line_number":1777,"context_line":"                if (\u0027no such table: %s\u0027 % SHARD_RANGE_TABLE) in str(err):"}],"source_content_type":"text/x-python","patch_set":57,"id":"271508f5_b3c9b7a7","line":1774,"in_reply_to":"09fea404_b31cb250","updated":"2024-01-11 00:53:22.000000000","message":"yes, I tried other optimized row factory as well. and surprisingly, ``None`` is the best.","commit_id":"7be1763455236ebf184d6be483798bad5e2acc9d"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"74e9ae2e078e75459ac35188586733c97d906994","unresolved":true,"context_lines":[{"line_number":1786,"context_line":"                                             lower\u003dnamespace[1],"},{"line_number":1787,"context_line":"                                             upper\u003dnamespace[2],"},{"line_number":1788,"context_line":"                                             state\u003dnamespace[3])"},{"line_number":1789,"context_line":"        namespaces.sort(key\u003dsort_key)"},{"line_number":1790,"context_line":"        # Convert the record tuples to Namespace objects."},{"line_number":1791,"context_line":"        namespaces \u003d [Namespace(row[0], row[1], row[2]) for row in namespaces]"},{"line_number":1792,"context_line":"        if includes:"}],"source_content_type":"text/x-python","patch_set":57,"id":"2460e836_3fd62ca7","line":1789,"updated":"2024-01-10 14:27:08.000000000","message":"nit: we could write:\n\n```\nnamespace \u003d sorted(data, key\u003dsort_key)\n```\n and delete line 1775\n\nbut I couldn\u0027t detect much difference when I benchmarked that change on ~10000 entry list - the sort is probably dominating.","commit_id":"7be1763455236ebf184d6be483798bad5e2acc9d"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"e77e3479d3023573255ad217d4a8ef9b220111a7","unresolved":true,"context_lines":[{"line_number":1786,"context_line":"                                             lower\u003dnamespace[1],"},{"line_number":1787,"context_line":"                                             upper\u003dnamespace[2],"},{"line_number":1788,"context_line":"                                             state\u003dnamespace[3])"},{"line_number":1789,"context_line":"        namespaces.sort(key\u003dsort_key)"},{"line_number":1790,"context_line":"        # Convert the record tuples to Namespace objects."},{"line_number":1791,"context_line":"        namespaces \u003d [Namespace(row[0], row[1], row[2]) for row in namespaces]"},{"line_number":1792,"context_line":"        if includes:"}],"source_content_type":"text/x-python","patch_set":57,"id":"d84c5837_b9d1b03d","line":1789,"in_reply_to":"2460e836_3fd62ca7","updated":"2024-01-12 14:29:07.000000000","message":"+1","commit_id":"7be1763455236ebf184d6be483798bad5e2acc9d"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"75f2bbd98f81540e59137c0a128facb8e38d61f8","unresolved":false,"context_lines":[{"line_number":1786,"context_line":"                                             lower\u003dnamespace[1],"},{"line_number":1787,"context_line":"                                             upper\u003dnamespace[2],"},{"line_number":1788,"context_line":"                                             state\u003dnamespace[3])"},{"line_number":1789,"context_line":"        namespaces.sort(key\u003dsort_key)"},{"line_number":1790,"context_line":"        # Convert the record tuples to Namespace objects."},{"line_number":1791,"context_line":"        namespaces \u003d [Namespace(row[0], row[1], row[2]) for row in namespaces]"},{"line_number":1792,"context_line":"        if includes:"}],"source_content_type":"text/x-python","patch_set":57,"id":"ccf481cd_b872d429","line":1789,"in_reply_to":"2460e836_3fd62ca7","updated":"2024-01-11 00:53:22.000000000","message":"``namespaces.sort(key\u003dsort_key)`` does in place sorting, while ``sorted(data, key\u003dsort_key)`` will do sorting and create a new list. so performance should be same in theory.","commit_id":"7be1763455236ebf184d6be483798bad5e2acc9d"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"e21248dd559c344333f1b3fec3ea14b01d2e0ac0","unresolved":false,"context_lines":[{"line_number":1786,"context_line":"                                             lower\u003dnamespace[1],"},{"line_number":1787,"context_line":"                                             upper\u003dnamespace[2],"},{"line_number":1788,"context_line":"                                             state\u003dnamespace[3])"},{"line_number":1789,"context_line":"        namespaces.sort(key\u003dsort_key)"},{"line_number":1790,"context_line":"        # Convert the record tuples to Namespace objects."},{"line_number":1791,"context_line":"        namespaces \u003d [Namespace(row[0], row[1], row[2]) for row in namespaces]"},{"line_number":1792,"context_line":"        if includes:"}],"source_content_type":"text/x-python","patch_set":57,"id":"e1485b77_2162fa02","line":1789,"in_reply_to":"ccf481cd_b872d429","updated":"2024-01-11 10:47:57.000000000","message":"good point, that explains why I saw no significant difference","commit_id":"7be1763455236ebf184d6be483798bad5e2acc9d"}],"swift/container/server.py":[{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"35e5be292734d19503eaa3be35584fc308b9ac53","unresolved":true,"context_lines":[{"line_number":731,"context_line":"            record_type \u003d \u0027shard\u0027"},{"line_number":732,"context_line":"        if record_type \u003d\u003d \u0027namespace\u0027:"},{"line_number":733,"context_line":"            # For record type of \u0027namespace\u0027, we only support listing of full"},{"line_number":734,"context_line":"            # updating namespaces, since this is the only case for now."},{"line_number":735,"context_line":"            resp_headers \u003d gen_resp_headers(info, is_deleted\u003dis_deleted)"},{"line_number":736,"context_line":"            if is_deleted:"},{"line_number":737,"context_line":"                return HTTPNotFound(request\u003dreq, headers\u003dresp_headers)"}],"source_content_type":"text/x-python","patch_set":5,"id":"77d1930c_708f88f1","line":734,"range":{"start_line":734,"start_character":14,"end_line":734,"end_character":33},"updated":"2023-08-10 05:25:24.000000000","message":"OK so is this why in the backend if we don\u0027t specify states we get back []?\n\nWe don\u0027t seem to be actaully filtering to only support updating state, this code seems to allow us to pass though any state or none.. but the backend will only return namespaces if any valid state is supplied.","commit_id":"5b1b5d986af7233b9c18a91e77cef6e1fbea8706"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"c03b18b305666b4f043dd6c764836f425e39f7fa","unresolved":false,"context_lines":[{"line_number":731,"context_line":"            record_type \u003d \u0027shard\u0027"},{"line_number":732,"context_line":"        if record_type \u003d\u003d \u0027namespace\u0027:"},{"line_number":733,"context_line":"            # For record type of \u0027namespace\u0027, we only support listing of full"},{"line_number":734,"context_line":"            # updating namespaces, since this is the only case for now."},{"line_number":735,"context_line":"            resp_headers \u003d gen_resp_headers(info, is_deleted\u003dis_deleted)"},{"line_number":736,"context_line":"            if is_deleted:"},{"line_number":737,"context_line":"                return HTTPNotFound(request\u003dreq, headers\u003dresp_headers)"}],"source_content_type":"text/x-python","patch_set":5,"id":"4a97b735_e3fdea24","line":734,"range":{"start_line":734,"start_character":14,"end_line":734,"end_character":33},"in_reply_to":"77d1930c_708f88f1","updated":"2023-08-18 04:03:19.000000000","message":"Done","commit_id":"5b1b5d986af7233b9c18a91e77cef6e1fbea8706"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"35e5be292734d19503eaa3be35584fc308b9ac53","unresolved":true,"context_lines":[{"line_number":808,"context_line":"                          is_sys_or_user_meta(\u0027container\u0027, key)):"},{"line_number":809,"context_line":"                resp_headers[str_to_wsgi(key)] \u003d str_to_wsgi(value)"},{"line_number":810,"context_line":""},{"line_number":811,"context_line":"        if (resp_headers[\u0027X-Backend-Record-Type\u0027] \u003d\u003d \u0027namespace\u0027):"},{"line_number":812,"context_line":"            listing \u003d [{\u0027lower\u0027: record[1], \u0027name\u0027: record[0]}"},{"line_number":813,"context_line":"                       for record in container_list]"},{"line_number":814,"context_line":"        else:"},{"line_number":815,"context_line":"            listing \u003d [self.update_data_record(record)"},{"line_number":816,"context_line":"                       for record in container_list]"}],"source_content_type":"text/x-python","patch_set":5,"id":"a0b562f7_f96993b0","line":813,"range":{"start_line":811,"start_character":10,"end_line":813,"end_character":52},"updated":"2023-08-10 05:25:24.000000000","message":"We could roll this into self.update_data_record.\nOK maybe we don\u0027t want to pass in a Namespace object like we do for shardranges (and maybe we should change that). Maybe we should just pass in the record_type to update_data_record or check to see if there are only 4 elements in each record?\n\nSomething like: https://paste.opendev.org/show/821053/ (although I forgot about the last_modifid in here)\n\nOr by passing in the record_type: https://paste.opendev.org/show/821054/","commit_id":"5b1b5d986af7233b9c18a91e77cef6e1fbea8706"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"c03b18b305666b4f043dd6c764836f425e39f7fa","unresolved":false,"context_lines":[{"line_number":808,"context_line":"                          is_sys_or_user_meta(\u0027container\u0027, key)):"},{"line_number":809,"context_line":"                resp_headers[str_to_wsgi(key)] \u003d str_to_wsgi(value)"},{"line_number":810,"context_line":""},{"line_number":811,"context_line":"        if (resp_headers[\u0027X-Backend-Record-Type\u0027] \u003d\u003d \u0027namespace\u0027):"},{"line_number":812,"context_line":"            listing \u003d [{\u0027lower\u0027: record[1], \u0027name\u0027: record[0]}"},{"line_number":813,"context_line":"                       for record in container_list]"},{"line_number":814,"context_line":"        else:"},{"line_number":815,"context_line":"            listing \u003d [self.update_data_record(record)"},{"line_number":816,"context_line":"                       for record in container_list]"}],"source_content_type":"text/x-python","patch_set":5,"id":"9d4e1986_a8d03a2e","line":813,"range":{"start_line":811,"start_character":10,"end_line":813,"end_character":52},"in_reply_to":"a0b562f7_f96993b0","updated":"2023-08-18 04:03:19.000000000","message":"the new record_type looks great, I squashed in your patch.","commit_id":"5b1b5d986af7233b9c18a91e77cef6e1fbea8706"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"f35d2431a428cfec4da64577f186cb5c684ba018","unresolved":true,"context_lines":[{"line_number":613,"context_line":"                            value determines how it\u0027ll build the response dict."},{"line_number":614,"context_line":"        :returns: modified record"},{"line_number":615,"context_line":"        \"\"\""},{"line_number":616,"context_line":"        if record_type \u003d\u003d \u0027namespace_list\u0027:"},{"line_number":617,"context_line":"            # namespace bound list"},{"line_number":618,"context_line":"            response \u003d dict(name\u003drecord.name, lower\u003drecord.lower)"},{"line_number":619,"context_line":"        elif record_type \u003d\u003d \u0027shard\u0027:"}],"source_content_type":"text/x-python","patch_set":8,"id":"ecf3ccf9_2eeaf6c7","line":616,"updated":"2023-08-24 07:12:15.000000000","message":"So namespace_list over namespace, what was your thoughts here? Is it because its name and lower (so that same as a namespace list rather then a list of namespace objects)?\n\nI know the former is what we put into memcache, so that makes sense I think.","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"d9f2b72057b8621e8971a513b7a78e1d82a5689a","unresolved":false,"context_lines":[{"line_number":613,"context_line":"                            value determines how it\u0027ll build the response dict."},{"line_number":614,"context_line":"        :returns: modified record"},{"line_number":615,"context_line":"        \"\"\""},{"line_number":616,"context_line":"        if record_type \u003d\u003d \u0027namespace_list\u0027:"},{"line_number":617,"context_line":"            # namespace bound list"},{"line_number":618,"context_line":"            response \u003d dict(name\u003drecord.name, lower\u003drecord.lower)"},{"line_number":619,"context_line":"        elif record_type \u003d\u003d \u0027shard\u0027:"}],"source_content_type":"text/x-python","patch_set":8,"id":"0dc93bff_c1e448be","line":616,"in_reply_to":"ecf3ccf9_2eeaf6c7","updated":"2023-08-25 04:21:08.000000000","message":"yes, the former is what we put into memcache and we have a wrapper data structure called NamespaceBoundList to store those list of name and lower. So that\u0027s why Al and I think \u0027namespace_list\u0027 indicates NamespaceBoundList instead of real Namespace. but maybe \u0027namespace_bound_list\u0027 is a even better name, just a little too long?","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"d7efa6e02d77ec38b860851868e2da2de3d45bb8","unresolved":true,"context_lines":[{"line_number":615,"context_line":"        \"\"\""},{"line_number":616,"context_line":"        if record_type \u003d\u003d \u0027namespace_list\u0027:"},{"line_number":617,"context_line":"            # namespace bound list"},{"line_number":618,"context_line":"            response \u003d dict(name\u003drecord.name, lower\u003drecord.lower)"},{"line_number":619,"context_line":"        elif record_type \u003d\u003d \u0027shard\u0027:"},{"line_number":620,"context_line":"            created \u003d record.timestamp"},{"line_number":621,"context_line":"            response \u003d dict(record)"}],"source_content_type":"text/x-python","patch_set":8,"id":"904cb977_25310502","line":618,"updated":"2023-08-24 12:22:29.000000000","message":"This only makes sense if the container is a root container that spans the entire namespace. The proxy-server will only use this API for root containers, but that cannot justify the interface returning useless data for a non-root container. So I think we do need to somehow include, as a minimum, the lower and upper outer bounds of the list... or return \n\n```\nresponse \u003d dict(record)\n```","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"4143d4bd5bbabf83ce395cd1329155481e56a09b","unresolved":true,"context_lines":[{"line_number":615,"context_line":"        \"\"\""},{"line_number":616,"context_line":"        if record_type \u003d\u003d \u0027namespace_list\u0027:"},{"line_number":617,"context_line":"            # namespace bound list"},{"line_number":618,"context_line":"            response \u003d dict(name\u003drecord.name, lower\u003drecord.lower)"},{"line_number":619,"context_line":"        elif record_type \u003d\u003d \u0027shard\u0027:"},{"line_number":620,"context_line":"            created \u003d record.timestamp"},{"line_number":621,"context_line":"            response \u003d dict(record)"}],"source_content_type":"text/x-python","patch_set":8,"id":"207507ee_5eb74c7f","line":618,"range":{"start_line":618,"start_character":52,"end_line":618,"end_character":64},"updated":"2023-08-25 15:13:14.000000000","message":"we need to use record.lower_str so that Namespace.[Min|Max]\nBound is converted to an empty string","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"dba5cdc4d8314b4dc7c21ec88d42d98beac2cfbf","unresolved":false,"context_lines":[{"line_number":615,"context_line":"        \"\"\""},{"line_number":616,"context_line":"        if record_type \u003d\u003d \u0027namespace_list\u0027:"},{"line_number":617,"context_line":"            # namespace bound list"},{"line_number":618,"context_line":"            response \u003d dict(name\u003drecord.name, lower\u003drecord.lower)"},{"line_number":619,"context_line":"        elif record_type \u003d\u003d \u0027shard\u0027:"},{"line_number":620,"context_line":"            created \u003d record.timestamp"},{"line_number":621,"context_line":"            response \u003d dict(record)"}],"source_content_type":"text/x-python","patch_set":8,"id":"064c62cc_9d6f62f5","line":618,"range":{"start_line":618,"start_character":52,"end_line":618,"end_character":64},"in_reply_to":"207507ee_5eb74c7f","updated":"2023-08-30 02:56:43.000000000","message":"Ack","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"dba5cdc4d8314b4dc7c21ec88d42d98beac2cfbf","unresolved":false,"context_lines":[{"line_number":615,"context_line":"        \"\"\""},{"line_number":616,"context_line":"        if record_type \u003d\u003d \u0027namespace_list\u0027:"},{"line_number":617,"context_line":"            # namespace bound list"},{"line_number":618,"context_line":"            response \u003d dict(name\u003drecord.name, lower\u003drecord.lower)"},{"line_number":619,"context_line":"        elif record_type \u003d\u003d \u0027shard\u0027:"},{"line_number":620,"context_line":"            created \u003d record.timestamp"},{"line_number":621,"context_line":"            response \u003d dict(record)"}],"source_content_type":"text/x-python","patch_set":8,"id":"098e65f1_a6afd90b","line":618,"in_reply_to":"904cb977_25310502","updated":"2023-08-30 02:56:43.000000000","message":"Ack","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"d7efa6e02d77ec38b860851868e2da2de3d45bb8","unresolved":true,"context_lines":[{"line_number":619,"context_line":"        elif record_type \u003d\u003d \u0027shard\u0027:"},{"line_number":620,"context_line":"            created \u003d record.timestamp"},{"line_number":621,"context_line":"            response \u003d dict(record)"},{"line_number":622,"context_line":"            response[\u0027last_modified\u0027] \u003d Timestamp(created).isoformat"},{"line_number":623,"context_line":"        else:"},{"line_number":624,"context_line":"            (name, created, size, content_type, etag) \u003d record[:5]"},{"line_number":625,"context_line":"            name_ \u003d name.decode(\u0027utf8\u0027) if six.PY2 else name"}],"source_content_type":"text/x-python","patch_set":8,"id":"60eaf67f_55ea9487","line":622,"updated":"2023-08-24 12:22:29.000000000","message":"off-topic: I\u0027m not sure this is necessary - a ShardRange doesn\u0027t consume a last_modified arg: https://github.com/openstack/swift/blob/1edf7df75528196d464470199d67df08149dfb04/swift/common/utils/__init__.py#L5523-L5539","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bff0a54e68a76090810057a6fac976d6d5e3e745","unresolved":false,"context_lines":[{"line_number":619,"context_line":"        elif record_type \u003d\u003d \u0027shard\u0027:"},{"line_number":620,"context_line":"            created \u003d record.timestamp"},{"line_number":621,"context_line":"            response \u003d dict(record)"},{"line_number":622,"context_line":"            response[\u0027last_modified\u0027] \u003d Timestamp(created).isoformat"},{"line_number":623,"context_line":"        else:"},{"line_number":624,"context_line":"            (name, created, size, content_type, etag) \u003d record[:5]"},{"line_number":625,"context_line":"            name_ \u003d name.decode(\u0027utf8\u0027) if six.PY2 else name"}],"source_content_type":"text/x-python","patch_set":8,"id":"c4208e0b_3ef66351","line":622,"in_reply_to":"60eaf67f_55ea9487","updated":"2024-01-04 16:20:07.000000000","message":"Acknowledged","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"bf33de1f5f106065568e0985e20cc76fff8f97e0","unresolved":true,"context_lines":[{"line_number":623,"context_line":"                \u0027bytes\u0027: size, \u0027hash\u0027: etag, \u0027name\u0027: name_,"},{"line_number":624,"context_line":"                \u0027content_type\u0027: content_type}"},{"line_number":625,"context_line":"            override_bytes_from_content_type(response, logger\u003dself.logger)"},{"line_number":626,"context_line":"        response[\u0027last_modified\u0027] \u003d Timestamp(created).isoformat"},{"line_number":627,"context_line":"        return response"},{"line_number":628,"context_line":""},{"line_number":629,"context_line":"    @public"}],"source_content_type":"text/x-python","patch_set":12,"id":"6d6eca21_d0a84ff3","side":"PARENT","line":626,"updated":"2023-08-31 06:46:04.000000000","message":"I\u0027d need to remember why we had this. Maybe for object listing when getting by json or xml. And what did we add it to shardranges to have a similar API?","commit_id":"ac524832e99b63edbad2f0984a6ced56d9dae49e"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"a57812bb8061973f9c43b94ddf758ccac9d7d357","unresolved":false,"context_lines":[{"line_number":623,"context_line":"                \u0027bytes\u0027: size, \u0027hash\u0027: etag, \u0027name\u0027: name_,"},{"line_number":624,"context_line":"                \u0027content_type\u0027: content_type}"},{"line_number":625,"context_line":"            override_bytes_from_content_type(response, logger\u003dself.logger)"},{"line_number":626,"context_line":"        response[\u0027last_modified\u0027] \u003d Timestamp(created).isoformat"},{"line_number":627,"context_line":"        return response"},{"line_number":628,"context_line":""},{"line_number":629,"context_line":"    @public"}],"source_content_type":"text/x-python","patch_set":12,"id":"46ba3f34_3017b2bc","side":"PARENT","line":626,"in_reply_to":"40c96494_998b517f","updated":"2023-11-02 20:54:39.000000000","message":"object listings have a \u0027created\u0027 column, but in sqlite it\u0027s a timestamp string - returning last_modified as an iso format string is part of the object listing api\n\nwe probably added it to internal shard listing just cause - but we mostly only consume shard listings in our code; where the timestamp strings are most useful.  It\u0027d be easier and more obvious to drop if we split up the record handling:\n\nhttps://review.opendev.org/c/openstack/swift/+/899983\n\n... but it\u0027s probalby useful occasionally when humans are inpecting shard-range json output, and if it ever slows us down we can just ask for the more lean namespaces!","commit_id":"ac524832e99b63edbad2f0984a6ced56d9dae49e"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"fd158f484a16e6ddbe812130fa865b4ebe5e7a8b","unresolved":true,"context_lines":[{"line_number":623,"context_line":"                \u0027bytes\u0027: size, \u0027hash\u0027: etag, \u0027name\u0027: name_,"},{"line_number":624,"context_line":"                \u0027content_type\u0027: content_type}"},{"line_number":625,"context_line":"            override_bytes_from_content_type(response, logger\u003dself.logger)"},{"line_number":626,"context_line":"        response[\u0027last_modified\u0027] \u003d Timestamp(created).isoformat"},{"line_number":627,"context_line":"        return response"},{"line_number":628,"context_line":""},{"line_number":629,"context_line":"    @public"}],"source_content_type":"text/x-python","patch_set":12,"id":"40c96494_998b517f","side":"PARENT","line":626,"in_reply_to":"6d6eca21_d0a84ff3","updated":"2023-08-31 21:06:25.000000000","message":"yeah, Alistair also pointed out: a ShardRange doesn\u0027t consume a last_modified arg: https://github.com/openstack/swift/blob/1edf7df75528196d464470199d67df08149dfb04/swift/common/utils/__init__.py#L5523-L5539","commit_id":"ac524832e99b63edbad2f0984a6ced56d9dae49e"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"bf33de1f5f106065568e0985e20cc76fff8f97e0","unresolved":true,"context_lines":[{"line_number":738,"context_line":"        if record_type \u003d\u003d \u0027auto\u0027 and db_state in (SHARDING, SHARDED):"},{"line_number":739,"context_line":"            record_type \u003d \u0027shard\u0027"},{"line_number":740,"context_line":"        if record_type \u003d\u003d \u0027namespace\u0027:"},{"line_number":741,"context_line":"            # For record type of \u0027namespace\u0027, we only support listing of full"},{"line_number":742,"context_line":"            # updating namespaces, since this is the only case for now."},{"line_number":743,"context_line":"            resp_headers \u003d gen_resp_headers(info, is_deleted\u003dis_deleted)"},{"line_number":744,"context_line":"            if is_deleted:"},{"line_number":745,"context_line":"                return HTTPNotFound(request\u003dreq, headers\u003dresp_headers)"}],"source_content_type":"text/x-python","patch_set":12,"id":"eed18f2d_d7253629","line":742,"range":{"start_line":741,"start_character":12,"end_line":742,"end_character":71},"updated":"2023-08-31 06:46:04.000000000","message":"Hmm, I might have changed things so this comment is lying in the new patch in the chain. I am now passing record_type \u003d namespace for non cached object put so we get a list of namespaces to find out where to send an update.\nI guess I should probably check that.\n\nIn anycase this code does allow namespace calls that aren\u0027t updateing stated calls so is this strictly true?","commit_id":"4638670d9e4dc6850b5447e2d5756be79b4c1aff"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"fd158f484a16e6ddbe812130fa865b4ebe5e7a8b","unresolved":false,"context_lines":[{"line_number":738,"context_line":"        if record_type \u003d\u003d \u0027auto\u0027 and db_state in (SHARDING, SHARDED):"},{"line_number":739,"context_line":"            record_type \u003d \u0027shard\u0027"},{"line_number":740,"context_line":"        if record_type \u003d\u003d \u0027namespace\u0027:"},{"line_number":741,"context_line":"            # For record type of \u0027namespace\u0027, we only support listing of full"},{"line_number":742,"context_line":"            # updating namespaces, since this is the only case for now."},{"line_number":743,"context_line":"            resp_headers \u003d gen_resp_headers(info, is_deleted\u003dis_deleted)"},{"line_number":744,"context_line":"            if is_deleted:"},{"line_number":745,"context_line":"                return HTTPNotFound(request\u003dreq, headers\u003dresp_headers)"}],"source_content_type":"text/x-python","patch_set":12,"id":"38287f02_f0e92cae","line":742,"range":{"start_line":741,"start_character":12,"end_line":742,"end_character":71},"in_reply_to":"eed18f2d_d7253629","updated":"2023-08-31 21:06:25.000000000","message":"Good eye, comments updated.","commit_id":"4638670d9e4dc6850b5447e2d5756be79b4c1aff"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"3ae9f087bde3d7f7c2bb592b87773a81dbd5fe1c","unresolved":true,"context_lines":[{"line_number":737,"context_line":"        db_state \u003d info.get(\u0027db_state\u0027)"},{"line_number":738,"context_line":"        if record_type \u003d\u003d \u0027auto\u0027 and db_state in (SHARDING, SHARDED):"},{"line_number":739,"context_line":"            record_type \u003d \u0027shard\u0027"},{"line_number":740,"context_line":"        if record_type \u003d\u003d \u0027namespace\u0027:"},{"line_number":741,"context_line":"            # For record type of \u0027namespace\u0027, we only support listing of states"},{"line_number":742,"context_line":"            # using \u0027updating\u0027 or \u0027listing\u0027. \u0027auditing\u0027 is not supported, for"},{"line_number":743,"context_line":"            # which caller should use record type of \u0027shard\u0027 instead."}],"source_content_type":"text/x-python","patch_set":17,"id":"3021203f_5bfee7c1","line":740,"updated":"2023-09-05 14:53:40.000000000","message":"this requires an upgraded proxy to send x-backend-record-type \u003d namespace, which means that an upgraded proxy will get back objects from a not-upgraded container server.\n\nAlternatives would be to use x-backend-record-type-v2 or to use a request query string param for the new version","commit_id":"6b34ca53368256d5f7109d8f184fe4a5fc6abf2b"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"2067216d369c4814babfac5e8edab055268bb263","unresolved":true,"context_lines":[{"line_number":737,"context_line":"        db_state \u003d info.get(\u0027db_state\u0027)"},{"line_number":738,"context_line":"        if record_type \u003d\u003d \u0027auto\u0027 and db_state in (SHARDING, SHARDED):"},{"line_number":739,"context_line":"            record_type \u003d \u0027shard\u0027"},{"line_number":740,"context_line":"        if record_type \u003d\u003d \u0027namespace\u0027:"},{"line_number":741,"context_line":"            # For record type of \u0027namespace\u0027, we only support listing of states"},{"line_number":742,"context_line":"            # using \u0027updating\u0027 or \u0027listing\u0027. \u0027auditing\u0027 is not supported, for"},{"line_number":743,"context_line":"            # which caller should use record type of \u0027shard\u0027 instead."}],"source_content_type":"text/x-python","patch_set":17,"id":"f0fb0db9_d18dd5b8","line":740,"in_reply_to":"033f2690_68cc6450","updated":"2023-09-07 17:34:59.000000000","message":"I\u0027ve realised that using a query_string param is not simple, because we do not filter the query string that comes from a client, but we do not want clients to be able to specify record_type\u003dnamespace. So if we start to use query string for backend-only params, then we need to enhance the gatekeeper middleware *first* - and then there\u0027s another upgrade path to figure out!!\n\nSo, reluctantly, I think we need to stick with using X-Backend-* headers. But we do, I think, need an additional new header so that the old container servers will continue to receive and understand X-Backend-Record-Type\u003dshard.\n\nWe have choices including:\n\nX-Backend-Record-Type-v2 \u003d [namespace|shard|object]\nX-Backend-Record-Format \u003d [namespace_json|json] \u003c- I\u0027m currently preferring this because it offers a new dimension - could imagine it being applied to new object row formats in future\n\nor maybe we can (ab)use the Accept header - but again, clients can set that :/\n\nI\u0027ll think on it some more.","commit_id":"6b34ca53368256d5f7109d8f184fe4a5fc6abf2b"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"e735d8c2861a8cd59022823cd8c9a3662f71e67d","unresolved":true,"context_lines":[{"line_number":737,"context_line":"        db_state \u003d info.get(\u0027db_state\u0027)"},{"line_number":738,"context_line":"        if record_type \u003d\u003d \u0027auto\u0027 and db_state in (SHARDING, SHARDED):"},{"line_number":739,"context_line":"            record_type \u003d \u0027shard\u0027"},{"line_number":740,"context_line":"        if record_type \u003d\u003d \u0027namespace\u0027:"},{"line_number":741,"context_line":"            # For record type of \u0027namespace\u0027, we only support listing of states"},{"line_number":742,"context_line":"            # using \u0027updating\u0027 or \u0027listing\u0027. \u0027auditing\u0027 is not supported, for"},{"line_number":743,"context_line":"            # which caller should use record type of \u0027shard\u0027 instead."}],"source_content_type":"text/x-python","patch_set":17,"id":"b768dd34_2bc885d6","line":740,"in_reply_to":"1d9e8aa2_882ba527","updated":"2023-09-08 00:26:09.000000000","message":"Yeah on the surface I don\u0027t love mixing the content-type and encoding format into one param\n\nProbably ok to have the proxy send:\n\nX-Backend-Record-Type: [shard|auto]\nX-Backend-Record-Shard-Format: namespace[,...]\n\nwhile sharder continues to send only\n\nX-Backend-Record-Type: shard\n\ncurrently it does seem to me like the container server shard API tries to respect the format-query-param and accept headers for xml/text/json\n\n\tvagrant@saio:~$ curl http://127.0.0.4:6041/sdb4/54/AUTH_test/lots-of-files -H \u0027x-backend-record-type: shard\u0027\n\t.shards_AUTH_test/lots-of-files-9933d06bdb1de26a4adfa0a7692feae9-1694125864.15718-0\n\t.shards_AUTH_test/lots-of-files-9933d06bdb1de26a4adfa0a7692feae9-1694125864.15718-1\n\t.shards_AUTH_test/lots-of-files-9933d06bdb1de26a4adfa0a7692feae9-1694125864.15718-2\n\t.shards_AUTH_test/lots-of-files-9933d06bdb1de26a4adfa0a7692feae9-1694125864.15718-3\n\t.shards_AUTH_test/lots-of-files-9933d06bdb1de26a4adfa0a7692feae9-1694125864.15718-4\n\t.shards_AUTH_test/lots-of-files-9933d06bdb1de26a4adfa0a7692feae9-1694125864.15718-5\n\t.shards_AUTH_test/lots-of-files-9933d06bdb1de26a4adfa0a7692feae9-1694125864.15718-6\n\t.shards_AUTH_test/lots-of-files-9933d06bdb1de26a4adfa0a7692feae9-1694125864.15718-7\n\t.shards_AUTH_test/lots-of-files-9933d06bdb1de26a4adfa0a7692feae9-1694125864.15718-8\n\t.shards_AUTH_test/lots-of-files-9933d06bdb1de26a4adfa0a7692feae9-1694125864.15718-9\n\tvagrant@saio:~$ curl http://127.0.0.4:6041/sdb4/54/AUTH_test/lots-of-files -H \u0027x-backend-record-type: shard\u0027 -H \u0027accept: application/xml\u0027\n\tTraceback (most recent call last):\n\t  File \"/vagrant/swift/swift/container/server.py\", line 906, in __call__\n\t    res \u003d getattr(self, req.method)(req)\n\t  File \"/vagrant/swift/swift/common/utils/__init__.py\", line 1529, in _timing_stats\n\t    resp \u003d func(ctrl, *args, **kwargs)\n\t  File \"/vagrant/swift/swift/container/server.py\", line 786, in GET\n\t    return self.create_listing(req, out_content_type, info, resp_headers,\n\t  File \"/vagrant/swift/swift/container/server.py\", line 798, in create_listing\n\t    body \u003d listing_formats.container_to_xml(listing, container)\n\t  File \"/vagrant/swift/swift/common/middleware/listing_formats.py\", line 104, in container_to_xml\n\t    record.pop(field))\n\tKeyError: \u0027hash\u0027\n\t\n\t\n... and maybe that\u0027s not obviously only a great thing","commit_id":"6b34ca53368256d5f7109d8f184fe4a5fc6abf2b"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"b25484713ceac3020ccd2e6c7d4fa98819f1499f","unresolved":true,"context_lines":[{"line_number":737,"context_line":"        db_state \u003d info.get(\u0027db_state\u0027)"},{"line_number":738,"context_line":"        if record_type \u003d\u003d \u0027auto\u0027 and db_state in (SHARDING, SHARDED):"},{"line_number":739,"context_line":"            record_type \u003d \u0027shard\u0027"},{"line_number":740,"context_line":"        if record_type \u003d\u003d \u0027namespace\u0027:"},{"line_number":741,"context_line":"            # For record type of \u0027namespace\u0027, we only support listing of states"},{"line_number":742,"context_line":"            # using \u0027updating\u0027 or \u0027listing\u0027. \u0027auditing\u0027 is not supported, for"},{"line_number":743,"context_line":"            # which caller should use record type of \u0027shard\u0027 instead."}],"source_content_type":"text/x-python","patch_set":17,"id":"4c65c9ba_80460884","line":740,"in_reply_to":"3021203f_5bfee7c1","updated":"2023-09-06 06:31:56.000000000","message":"OK, so I like sending in a querystring to get an upgrade path. But not 100% sold on the query param being \u0027size\u0027. I mean I get it size\u003dfull or size\u003dnamespace. But size feels a little close to limit, which could be a little confusing.\n\nCould we just move to a record_type or rtype or something. Maybe type if we ever want to overload with something else later.\n\nMaybe then we can have:\n\n    record_type \u003d params.get(\u0027record_type\u0027, req.headers.get(\u0027x-backend-record-type\u0027, \u0027\u0027)).lower()\n \nThe proxy can start senfing the record_type (or something) param but also throw in an \u0027x-record-type: shard\u0027 and the param will take precidence?","commit_id":"6b34ca53368256d5f7109d8f184fe4a5fc6abf2b"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"eb2b31677cd82186d386efc853e0db387d7164ae","unresolved":true,"context_lines":[{"line_number":737,"context_line":"        db_state \u003d info.get(\u0027db_state\u0027)"},{"line_number":738,"context_line":"        if record_type \u003d\u003d \u0027auto\u0027 and db_state in (SHARDING, SHARDED):"},{"line_number":739,"context_line":"            record_type \u003d \u0027shard\u0027"},{"line_number":740,"context_line":"        if record_type \u003d\u003d \u0027namespace\u0027:"},{"line_number":741,"context_line":"            # For record type of \u0027namespace\u0027, we only support listing of states"},{"line_number":742,"context_line":"            # using \u0027updating\u0027 or \u0027listing\u0027. \u0027auditing\u0027 is not supported, for"},{"line_number":743,"context_line":"            # which caller should use record type of \u0027shard\u0027 instead."}],"source_content_type":"text/x-python","patch_set":17,"id":"c703bf71_2df7938e","line":740,"in_reply_to":"4c65c9ba_80460884","updated":"2023-09-06 16:31:47.000000000","message":"I agree with Matt.\n\n1. \u0027size\u0027 doesn\u0027t feel like the right label for what is in essence a choice of data type.\n\n2. I\u0027d hoped this change to using params to result in the URL alone indicating if the GET was for object, shard or namespace records by having ``record_type\u003d[object|shard|namespace]``. OK, it can be inferred that if there is a size param then the GET is for a shard-like type, but if there is NOT a size param, I cannot infer from the URL alone if it is a new proxy requesting an object record type, or an old proxy requesting shard record type.\n\n3. Subjectively I suspect it will be easier to understand a single param that defines the record type with a fallback to the existing header that is clearly commented \"this is for backwards compatibility only\".","commit_id":"6b34ca53368256d5f7109d8f184fe4a5fc6abf2b"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"eb02e84623e5173d951d13bd70511677d112089e","unresolved":false,"context_lines":[{"line_number":737,"context_line":"        db_state \u003d info.get(\u0027db_state\u0027)"},{"line_number":738,"context_line":"        if record_type \u003d\u003d \u0027auto\u0027 and db_state in (SHARDING, SHARDED):"},{"line_number":739,"context_line":"            record_type \u003d \u0027shard\u0027"},{"line_number":740,"context_line":"        if record_type \u003d\u003d \u0027namespace\u0027:"},{"line_number":741,"context_line":"            # For record type of \u0027namespace\u0027, we only support listing of states"},{"line_number":742,"context_line":"            # using \u0027updating\u0027 or \u0027listing\u0027. \u0027auditing\u0027 is not supported, for"},{"line_number":743,"context_line":"            # which caller should use record type of \u0027shard\u0027 instead."}],"source_content_type":"text/x-python","patch_set":17,"id":"a2457414_ad52dbda","line":740,"in_reply_to":"5d516d3a_f6cef231","updated":"2023-09-09 01:11:47.000000000","message":"Ack","commit_id":"6b34ca53368256d5f7109d8f184fe4a5fc6abf2b"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"0f534ac097e0f90a9366bd636cb83df71679897b","unresolved":true,"context_lines":[{"line_number":737,"context_line":"        db_state \u003d info.get(\u0027db_state\u0027)"},{"line_number":738,"context_line":"        if record_type \u003d\u003d \u0027auto\u0027 and db_state in (SHARDING, SHARDED):"},{"line_number":739,"context_line":"            record_type \u003d \u0027shard\u0027"},{"line_number":740,"context_line":"        if record_type \u003d\u003d \u0027namespace\u0027:"},{"line_number":741,"context_line":"            # For record type of \u0027namespace\u0027, we only support listing of states"},{"line_number":742,"context_line":"            # using \u0027updating\u0027 or \u0027listing\u0027. \u0027auditing\u0027 is not supported, for"},{"line_number":743,"context_line":"            # which caller should use record type of \u0027shard\u0027 instead."}],"source_content_type":"text/x-python","patch_set":17,"id":"033f2690_68cc6450","line":740,"in_reply_to":"87d36fdd_7d895f06","updated":"2023-09-07 15:58:05.000000000","message":"\"At some point maybe we can drop the header from a proxy.\" \n\nIf we drop the header \u0027X-Backend-Record-Type\u0027 from our future proxy upstream release, what will happen if our upstream users upgrade across versions, from old version to that future upstream release? then they will still face the same issue.\n\nAlso, we have a lot of other headers other than \u0027X-Backend-Record-Type\u0027, if we move one to query params, how about others in future? I\u0027d like to have something consistent across the board if we can.","commit_id":"6b34ca53368256d5f7109d8f184fe4a5fc6abf2b"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"63f877bc97fa23150a29caaae804a272af87693f","unresolved":true,"context_lines":[{"line_number":737,"context_line":"        db_state \u003d info.get(\u0027db_state\u0027)"},{"line_number":738,"context_line":"        if record_type \u003d\u003d \u0027auto\u0027 and db_state in (SHARDING, SHARDED):"},{"line_number":739,"context_line":"            record_type \u003d \u0027shard\u0027"},{"line_number":740,"context_line":"        if record_type \u003d\u003d \u0027namespace\u0027:"},{"line_number":741,"context_line":"            # For record type of \u0027namespace\u0027, we only support listing of states"},{"line_number":742,"context_line":"            # using \u0027updating\u0027 or \u0027listing\u0027. \u0027auditing\u0027 is not supported, for"},{"line_number":743,"context_line":"            # which caller should use record type of \u0027shard\u0027 instead."}],"source_content_type":"text/x-python","patch_set":17,"id":"5d516d3a_f6cef231","line":740,"in_reply_to":"b768dd34_2bc885d6","updated":"2023-09-08 07:07:27.000000000","message":"Yeah I can get on board with a:\n\n    X-Backend-Record-Type: [shard|auto]\n    X-Backend-Record-Shard-Format: namespace[,...]\n    \nAt least it gives us an upgrade path that works","commit_id":"6b34ca53368256d5f7109d8f184fe4a5fc6abf2b"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"acaec4aa4c186c496cc9e5f3ec6384b00cb1cfee","unresolved":true,"context_lines":[{"line_number":737,"context_line":"        db_state \u003d info.get(\u0027db_state\u0027)"},{"line_number":738,"context_line":"        if record_type \u003d\u003d \u0027auto\u0027 and db_state in (SHARDING, SHARDED):"},{"line_number":739,"context_line":"            record_type \u003d \u0027shard\u0027"},{"line_number":740,"context_line":"        if record_type \u003d\u003d \u0027namespace\u0027:"},{"line_number":741,"context_line":"            # For record type of \u0027namespace\u0027, we only support listing of states"},{"line_number":742,"context_line":"            # using \u0027updating\u0027 or \u0027listing\u0027. \u0027auditing\u0027 is not supported, for"},{"line_number":743,"context_line":"            # which caller should use record type of \u0027shard\u0027 instead."}],"source_content_type":"text/x-python","patch_set":17,"id":"efd2a28b_e3978ca1","line":740,"in_reply_to":"c703bf71_2df7938e","updated":"2023-09-07 01:49:05.000000000","message":"for this question \"if there is NOT a size param, I cannot infer from the URL alone if it is a new proxy requesting an object record type, or an old proxy requesting shard record type\", will container server (old or new) still be able to tell this by checking \u0027db_state\u0027?\nhttps://github.com/NVIDIA/swift/blob/master/swift/container/server.py#L730C11-L730C11","commit_id":"6b34ca53368256d5f7109d8f184fe4a5fc6abf2b"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"28bb07dd7e36e391037ac1855c0e40edded4a9d6","unresolved":true,"context_lines":[{"line_number":737,"context_line":"        db_state \u003d info.get(\u0027db_state\u0027)"},{"line_number":738,"context_line":"        if record_type \u003d\u003d \u0027auto\u0027 and db_state in (SHARDING, SHARDED):"},{"line_number":739,"context_line":"            record_type \u003d \u0027shard\u0027"},{"line_number":740,"context_line":"        if record_type \u003d\u003d \u0027namespace\u0027:"},{"line_number":741,"context_line":"            # For record type of \u0027namespace\u0027, we only support listing of states"},{"line_number":742,"context_line":"            # using \u0027updating\u0027 or \u0027listing\u0027. \u0027auditing\u0027 is not supported, for"},{"line_number":743,"context_line":"            # which caller should use record type of \u0027shard\u0027 instead."}],"source_content_type":"text/x-python","patch_set":17,"id":"87d36fdd_7d895f06","line":740,"in_reply_to":"efd2a28b_e3978ca1","updated":"2023-09-07 04:10:09.000000000","message":"The way I imagine it would be that old proxies still send \u0027X-Backend-Record-Type: shard\u0027, so the container server still checks that. A new proxy can send it as a query param `GET /v1/account/container?record_type\u003dnamespace`, so the container server should look for that.\n\nA new proxy, so we can talk to old container servers, should send header \u0027X-Backend-Record-Type: shard\u0027 and \u0027?record_type\u003dnamespace\u0027 and mention in code that this is for backwards compat. At some point maybe we can drop the header from a proxy.\n\nIn the code, we\u0027d just need to:\n\n    record_type \u003d params.get(\u0027record_type\u0027, req.headers.get(\u0027x-backend-record-type\u0027, \u0027\u0027)).lower()\n    \nOr something, and just keep treating record_type as we did before. It also means we\u0027d see the ?record_type query string on log lines.\n\nThe proxy will look for the X-Backend-Record-Type response and now what it\u0027s getting from the container server.\n\nAs for Auto, I think keeping it as is for now is ok. If there isn\u0027t a record_type (header or query param) then we can do what we currently do, and if that\u0027s \"auto\" then returning object or shard based off db_state is fine. We might just have to be explicit when we want namespaces? Although we could flip this and default to namespaces and be explicit about shards (As it\u0027ll only be the sharder that\u0027ll be asking for these).","commit_id":"6b34ca53368256d5f7109d8f184fe4a5fc6abf2b"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"b1653e9d53d095ff42423ad8cc33df36169edf94","unresolved":true,"context_lines":[{"line_number":737,"context_line":"        db_state \u003d info.get(\u0027db_state\u0027)"},{"line_number":738,"context_line":"        if record_type \u003d\u003d \u0027auto\u0027 and db_state in (SHARDING, SHARDED):"},{"line_number":739,"context_line":"            record_type \u003d \u0027shard\u0027"},{"line_number":740,"context_line":"        if record_type \u003d\u003d \u0027namespace\u0027:"},{"line_number":741,"context_line":"            # For record type of \u0027namespace\u0027, we only support listing of states"},{"line_number":742,"context_line":"            # using \u0027updating\u0027 or \u0027listing\u0027. \u0027auditing\u0027 is not supported, for"},{"line_number":743,"context_line":"            # which caller should use record type of \u0027shard\u0027 instead."}],"source_content_type":"text/x-python","patch_set":17,"id":"1d9e8aa2_882ba527","line":740,"in_reply_to":"f0fb0db9_d18dd5b8","updated":"2023-09-07 22:15:39.000000000","message":"For this option \"X-Backend-Record-Format \u003d [namespace_json|json]\", will the new option be mixed up with another layer of configs to define the return encoding format? https://github.com/NVIDIA/swift/blob/master/swift/common/middleware/listing_formats.py#L40-L62\n\nI understand that\u0027ll be great if the newly added header can be extended for other purposes later, but it seems to me that configs of content type and encoding format are for different purposes.\n\nHow about the new header config is very specific to \"Shard\" related record type?\nX-Backend-ShardRange-Format \u003d [namespace|full]","commit_id":"6b34ca53368256d5f7109d8f184fe4a5fc6abf2b"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"3ae9f087bde3d7f7c2bb592b87773a81dbd5fe1c","unresolved":true,"context_lines":[{"line_number":764,"context_line":"                return HTTPNotFound(request\u003dreq, headers\u003dresp_headers)"},{"line_number":765,"context_line":"            resp_headers[\u0027X-Backend-Record-Type\u0027] \u003d \u0027shard\u0027"},{"line_number":766,"context_line":"            includes \u003d params.get(\u0027includes\u0027)"},{"line_number":767,"context_line":"            override_filter_hdr \u003d req.headers.get("},{"line_number":768,"context_line":"                \u0027x-backend-override-shard-name-filter\u0027, \u0027\u0027).lower()"},{"line_number":769,"context_line":"            if override_filter_hdr \u003d\u003d db_state \u003d\u003d \u0027sharded\u0027:"},{"line_number":770,"context_line":"                # respect the request to send back *all* ranges if the db is in"},{"line_number":771,"context_line":"                # sharded state"},{"line_number":772,"context_line":"                resp_headers[\u0027X-Backend-Override-Shard-Name-Filter\u0027] \u003d \u0027true\u0027"},{"line_number":773,"context_line":"                marker \u003d end_marker \u003d includes \u003d None"},{"line_number":774,"context_line":"                reverse \u003d False"},{"line_number":775,"context_line":"            states \u003d params.get(\u0027states\u0027)"},{"line_number":776,"context_line":"            fill_gaps \u003d include_own \u003d False"},{"line_number":777,"context_line":"            if states:"},{"line_number":778,"context_line":"                states \u003d list_from_csv(states)"}],"source_content_type":"text/x-python","patch_set":17,"id":"56f9f84f_03d0b339","line":775,"range":{"start_line":767,"start_character":0,"end_line":775,"end_character":0},"updated":"2023-09-05 14:53:40.000000000","message":"the proxy uses this logic for listings","commit_id":"6b34ca53368256d5f7109d8f184fe4a5fc6abf2b"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"4621264a31586fe757b1846045e261e71f449a32","unresolved":false,"context_lines":[{"line_number":764,"context_line":"                return HTTPNotFound(request\u003dreq, headers\u003dresp_headers)"},{"line_number":765,"context_line":"            resp_headers[\u0027X-Backend-Record-Type\u0027] \u003d \u0027shard\u0027"},{"line_number":766,"context_line":"            includes \u003d params.get(\u0027includes\u0027)"},{"line_number":767,"context_line":"            override_filter_hdr \u003d req.headers.get("},{"line_number":768,"context_line":"                \u0027x-backend-override-shard-name-filter\u0027, \u0027\u0027).lower()"},{"line_number":769,"context_line":"            if override_filter_hdr \u003d\u003d db_state \u003d\u003d \u0027sharded\u0027:"},{"line_number":770,"context_line":"                # respect the request to send back *all* ranges if the db is in"},{"line_number":771,"context_line":"                # sharded state"},{"line_number":772,"context_line":"                resp_headers[\u0027X-Backend-Override-Shard-Name-Filter\u0027] \u003d \u0027true\u0027"},{"line_number":773,"context_line":"                marker \u003d end_marker \u003d includes \u003d None"},{"line_number":774,"context_line":"                reverse \u003d False"},{"line_number":775,"context_line":"            states \u003d params.get(\u0027states\u0027)"},{"line_number":776,"context_line":"            fill_gaps \u003d include_own \u003d False"},{"line_number":777,"context_line":"            if states:"},{"line_number":778,"context_line":"                states \u003d list_from_csv(states)"}],"source_content_type":"text/x-python","patch_set":17,"id":"f1e954e8_c404003f","line":775,"range":{"start_line":767,"start_character":0,"end_line":775,"end_character":0},"in_reply_to":"56f9f84f_03d0b339","updated":"2023-09-11 03:07:01.000000000","message":"Ack","commit_id":"6b34ca53368256d5f7109d8f184fe4a5fc6abf2b"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"aceacc504d338a7772301488c765f1f0333a9b1a","unresolved":true,"context_lines":[{"line_number":742,"context_line":"        record_type \u003d req.headers.get(\u0027x-backend-record-type\u0027, \u0027\u0027).lower()"},{"line_number":743,"context_line":"        db_state \u003d info.get(\u0027db_state\u0027)"},{"line_number":744,"context_line":"        if record_type \u003d\u003d \u0027auto\u0027 and db_state in (SHARDING, SHARDED):"},{"line_number":745,"context_line":"            record_type \u003d \u0027shard\u0027"},{"line_number":746,"context_line":"        if record_type \u003d\u003d \u0027shard\u0027:"},{"line_number":747,"context_line":"            shard_format \u003d req.headers.get("},{"line_number":748,"context_line":"                \u0027x-backend-record-shard-format\u0027, \u0027full\u0027).lower()"}],"source_content_type":"text/x-python","patch_set":21,"id":"0c4edfbf_1d5bb80d","line":745,"range":{"start_line":745,"start_character":12,"end_line":745,"end_character":33},"updated":"2023-09-11 07:05:03.000000000","message":"The proxy container listing basically defaults to \u0027auto\u0027 for record_type if one isn\u0027t provided. And when we\u0027re doing a container GET in the proxy we basically just let it add an \u0027auto\u0027 because it\u0027ll return either a shard or object record_type in the respose we can use on how to build the listing.\n\nWhen we\u0027re doing GET\u0027s from the object controller we tent to provide the record_type, so less likely to use \u0027auto\u0027.\n\nBecause we\u0027re defaulting auto to shard here, we\u0027re basically going to request shards. I guess I\u0027ll just always have to add \u0027x-backend-record-shard-format: namespace\u0027 on the container controller side so we get the speed up.\n\nThe question is, do I only add container controller support to match this, so should we work on the follow up to add marker, end_marker and includes now so my patch can take advantage of the speed up everywhere... I think a bunch of the work I\u0027ve currently done was done attempting to use namespaces everywhere.. I guess I could break that down into two steps too.\n\nI\u0027ll have a play with that I guess.","commit_id":"99291123a8eedfbcd665f570f4133ea44e55d16a"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"0dad8d14fe8a0ea9a7905e7fe47b9439d0567051","unresolved":false,"context_lines":[{"line_number":742,"context_line":"        record_type \u003d req.headers.get(\u0027x-backend-record-type\u0027, \u0027\u0027).lower()"},{"line_number":743,"context_line":"        db_state \u003d info.get(\u0027db_state\u0027)"},{"line_number":744,"context_line":"        if record_type \u003d\u003d \u0027auto\u0027 and db_state in (SHARDING, SHARDED):"},{"line_number":745,"context_line":"            record_type \u003d \u0027shard\u0027"},{"line_number":746,"context_line":"        if record_type \u003d\u003d \u0027shard\u0027:"},{"line_number":747,"context_line":"            shard_format \u003d req.headers.get("},{"line_number":748,"context_line":"                \u0027x-backend-record-shard-format\u0027, \u0027full\u0027).lower()"}],"source_content_type":"text/x-python","patch_set":21,"id":"58f225b8_67cb015a","line":745,"range":{"start_line":745,"start_character":12,"end_line":745,"end_character":33},"in_reply_to":"0c4edfbf_1d5bb80d","updated":"2023-09-13 06:08:45.000000000","message":"Ack","commit_id":"99291123a8eedfbcd665f570f4133ea44e55d16a"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"52eef4ab78afe8da7fe5de0a541c3b1255dfe896","unresolved":true,"context_lines":[{"line_number":753,"context_line":"            # Namespace yet during upgrade."},{"line_number":754,"context_line":"            if shard_format \u003d\u003d \u0027namespace\u0027:"},{"line_number":755,"context_line":"                record_type \u003d \u0027namespace\u0027"},{"line_number":756,"context_line":"        if record_type \u003d\u003d \u0027namespace\u0027:"},{"line_number":757,"context_line":"            # For \u0027namespace\u0027 record type, container server only supports"},{"line_number":758,"context_line":"            # listing of states using \u0027updating\u0027 or \u0027listing\u0027, \u0027auditing\u0027 is"},{"line_number":759,"context_line":"            # not supported, for which caller should set query param \u0027size\u0027 to"}],"source_content_type":"text/x-python","patch_set":21,"id":"9d170672_dfd37dbb","line":756,"updated":"2023-09-12 15:58:56.000000000","message":"thinking about the next upgrade path (this patch has merged, but we have another change so that container-server supports marker/end_marker and includes for get_namespaces) - if the proxy sends ``shard_format \u003d\u003d namespace`` and ``includes`` to *this* version, it will get back the whole list. Is that OK? I don\u0027t think it is. https://github.com/openstack/swift/blob/9f385c07f3ea3a55c8922c65124d14202e168351/swift/proxy/controllers/obj.py#L340-L346\n\nShould we make this condition also test for ``not (includes or marker or end_marker)``?\n\nThen we would remove that extra condition in the next upgrade to the container server.","commit_id":"99291123a8eedfbcd665f570f4133ea44e55d16a"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"e54ab333db264444f68335f6b41a2910cc866988","unresolved":false,"context_lines":[{"line_number":753,"context_line":"            # Namespace yet during upgrade."},{"line_number":754,"context_line":"            if shard_format \u003d\u003d \u0027namespace\u0027:"},{"line_number":755,"context_line":"                record_type \u003d \u0027namespace\u0027"},{"line_number":756,"context_line":"        if record_type \u003d\u003d \u0027namespace\u0027:"},{"line_number":757,"context_line":"            # For \u0027namespace\u0027 record type, container server only supports"},{"line_number":758,"context_line":"            # listing of states using \u0027updating\u0027 or \u0027listing\u0027, \u0027auditing\u0027 is"},{"line_number":759,"context_line":"            # not supported, for which caller should set query param \u0027size\u0027 to"}],"source_content_type":"text/x-python","patch_set":21,"id":"09446904_342452b2","line":756,"in_reply_to":"9d170672_dfd37dbb","updated":"2023-09-13 06:07:17.000000000","message":"Done, thanks! For Namespace GET requests with \u0027includes\u0027 or \u0027marker\u0027 or \u0027end_marker\u0027, the new patch will return ShardRange instead. So the second step proxy side patch will need implement the same thing: check the header of the response, it may be ShardRange even if you are asking for Namespace.","commit_id":"99291123a8eedfbcd665f570f4133ea44e55d16a"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"d325f8091eb41df7731be6675c7fcde317b4a090","unresolved":true,"context_lines":[{"line_number":710,"context_line":"          only attributes (name, lower and upper) if and only if the"},{"line_number":711,"context_line":"          ``X-Backend-Record-Shard-Format`` header value is \u0027namespace\u0027. If"},{"line_number":712,"context_line":"          this header doesn\u0027t exist or the value is \u0027full\u0027, the listings will"},{"line_number":713,"context_line":"          default to include all attributes of shard ranges."},{"line_number":714,"context_line":""},{"line_number":715,"context_line":"        * Listings are not normally returned from a deleted container. However,"},{"line_number":716,"context_line":"          the ``X-Backend-Override-Deleted`` header may be used with a value in"}],"source_content_type":"text/x-python","patch_set":22,"id":"f5a00671_a65e6640","line":713,"updated":"2023-09-13 16:42:15.000000000","message":"X-Backend-Record-Shard-Format can be sent by the proxy as a hint that it would prefer namespaces, but if params has includes/marker/end_marker then the response will be full shard ranges, regardless of X-Backend-Record-Shard-Format.\n\nThe response X-Backend-Record-Type header will tell the proxy what type it is getting back.","commit_id":"b105e043017904fcf520fd7cdad5c07aba47e76f"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"e0de10c09424e156f49ac8d7a4ac31497acef492","unresolved":false,"context_lines":[{"line_number":710,"context_line":"          only attributes (name, lower and upper) if and only if the"},{"line_number":711,"context_line":"          ``X-Backend-Record-Shard-Format`` header value is \u0027namespace\u0027. If"},{"line_number":712,"context_line":"          this header doesn\u0027t exist or the value is \u0027full\u0027, the listings will"},{"line_number":713,"context_line":"          default to include all attributes of shard ranges."},{"line_number":714,"context_line":""},{"line_number":715,"context_line":"        * Listings are not normally returned from a deleted container. However,"},{"line_number":716,"context_line":"          the ``X-Backend-Override-Deleted`` header may be used with a value in"}],"source_content_type":"text/x-python","patch_set":22,"id":"35e68885_af8f4e46","line":713,"in_reply_to":"f5a00671_a65e6640","updated":"2023-09-13 19:42:38.000000000","message":"Done","commit_id":"b105e043017904fcf520fd7cdad5c07aba47e76f"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"dd2266c879ea38eae3d5fd5fe501e62a1982679c","unresolved":true,"context_lines":[{"line_number":766,"context_line":"            # \u0027x-backend-record-shard-format\u0027 to \u0027full\u0027 instead, or remove it"},{"line_number":767,"context_line":"            # from headers."},{"line_number":768,"context_line":"            resp_headers \u003d gen_resp_headers(info, is_deleted\u003dis_deleted)"},{"line_number":769,"context_line":"            if is_deleted:"},{"line_number":770,"context_line":"                return HTTPNotFound(request\u003dreq, headers\u003dresp_headers)"},{"line_number":771,"context_line":"            # Send back record type as \u0027namespace\u0027."},{"line_number":772,"context_line":"            resp_headers[\u0027X-Backend-Record-Type\u0027] \u003d \u0027namespace\u0027"}],"source_content_type":"text/x-python","patch_set":22,"id":"be15e6e1_676e1b16","line":769,"updated":"2023-09-13 19:57:42.000000000","message":"No ability to override? I guess that was only used by the sharder, which wants full records?","commit_id":"b105e043017904fcf520fd7cdad5c07aba47e76f"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"693fbccfb57938a9ccae311ba3d53701e72a9a9d","unresolved":false,"context_lines":[{"line_number":766,"context_line":"            # \u0027x-backend-record-shard-format\u0027 to \u0027full\u0027 instead, or remove it"},{"line_number":767,"context_line":"            # from headers."},{"line_number":768,"context_line":"            resp_headers \u003d gen_resp_headers(info, is_deleted\u003dis_deleted)"},{"line_number":769,"context_line":"            if is_deleted:"},{"line_number":770,"context_line":"                return HTTPNotFound(request\u003dreq, headers\u003dresp_headers)"},{"line_number":771,"context_line":"            # Send back record type as \u0027namespace\u0027."},{"line_number":772,"context_line":"            resp_headers[\u0027X-Backend-Record-Type\u0027] \u003d \u0027namespace\u0027"}],"source_content_type":"text/x-python","patch_set":22,"id":"82f96ce5_acdf615d","line":769,"in_reply_to":"be15e6e1_676e1b16","updated":"2023-09-21 04:37:28.000000000","message":"Ack","commit_id":"b105e043017904fcf520fd7cdad5c07aba47e76f"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"dd2266c879ea38eae3d5fd5fe501e62a1982679c","unresolved":true,"context_lines":[{"line_number":775,"context_line":"            if override_filter_hdr \u003d\u003d db_state \u003d\u003d \u0027sharded\u0027:"},{"line_number":776,"context_line":"                # Only support sending back *all* ranges if the db is in"},{"line_number":777,"context_line":"                # sharded state, add the header for proxy checking purpose."},{"line_number":778,"context_line":"                resp_headers[\u0027X-Backend-Override-Shard-Name-Filter\u0027] \u003d \u0027true\u0027"},{"line_number":779,"context_line":"            fill_gaps \u003d False"},{"line_number":780,"context_line":"            if states:"},{"line_number":781,"context_line":"                states \u003d list_from_csv(states)"}],"source_content_type":"text/x-python","patch_set":22,"id":"d0a1a89d_c7b6b42d","line":778,"updated":"2023-09-13 19:57:42.000000000","message":"I\u0027m a little torn about this -- we don\u0027t support name filtering for namespaces anyway, so I\u0027m tempted to say that `X-Backend-Record-Type: namespace` is sufficient to determine that the proxy got back a full, cacheable set. But, if we later *do* want to allow filtering...","commit_id":"b105e043017904fcf520fd7cdad5c07aba47e76f"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"693fbccfb57938a9ccae311ba3d53701e72a9a9d","unresolved":true,"context_lines":[{"line_number":775,"context_line":"            if override_filter_hdr \u003d\u003d db_state \u003d\u003d \u0027sharded\u0027:"},{"line_number":776,"context_line":"                # Only support sending back *all* ranges if the db is in"},{"line_number":777,"context_line":"                # sharded state, add the header for proxy checking purpose."},{"line_number":778,"context_line":"                resp_headers[\u0027X-Backend-Override-Shard-Name-Filter\u0027] \u003d \u0027true\u0027"},{"line_number":779,"context_line":"            fill_gaps \u003d False"},{"line_number":780,"context_line":"            if states:"},{"line_number":781,"context_line":"                states \u003d list_from_csv(states)"}],"source_content_type":"text/x-python","patch_set":22,"id":"d68e7204_788acd65","line":778,"in_reply_to":"d0a1a89d_c7b6b42d","updated":"2023-09-21 04:37:28.000000000","message":"yeah, this header \"X-Backend-Override-Shard-Name-Filter\" is added only to keep proxy-server happy, who check this header after retrieving the listing shard ranges back from backend. https://github.com/NVIDIA/swift/blob/master/swift/proxy/controllers/container.py#L281","commit_id":"b105e043017904fcf520fd7cdad5c07aba47e76f"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bff0a54e68a76090810057a6fac976d6d5e3e745","unresolved":false,"context_lines":[{"line_number":775,"context_line":"            if override_filter_hdr \u003d\u003d db_state \u003d\u003d \u0027sharded\u0027:"},{"line_number":776,"context_line":"                # Only support sending back *all* ranges if the db is in"},{"line_number":777,"context_line":"                # sharded state, add the header for proxy checking purpose."},{"line_number":778,"context_line":"                resp_headers[\u0027X-Backend-Override-Shard-Name-Filter\u0027] \u003d \u0027true\u0027"},{"line_number":779,"context_line":"            fill_gaps \u003d False"},{"line_number":780,"context_line":"            if states:"},{"line_number":781,"context_line":"                states \u003d list_from_csv(states)"}],"source_content_type":"text/x-python","patch_set":22,"id":"4e433201_5392aae4","line":778,"in_reply_to":"d68e7204_788acd65","updated":"2024-01-04 16:20:07.000000000","message":"Done","commit_id":"b105e043017904fcf520fd7cdad5c07aba47e76f"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"aef4a35e9f00ec228a3f60bfba95722c8d3e2bd6","unresolved":true,"context_lines":[{"line_number":762,"context_line":"            if shard_format \u003d\u003d \"namespace\" and not ("},{"line_number":763,"context_line":"                includes or marker or end_marker or reverse"},{"line_number":764,"context_line":"            ):"},{"line_number":765,"context_line":"                record_type \u003d \u0027namespace\u0027"},{"line_number":766,"context_line":"        if record_type \u003d\u003d \u0027namespace\u0027:"},{"line_number":767,"context_line":"            # For \u0027namespace\u0027 record type, container server only supports"},{"line_number":768,"context_line":"            # listing of states using \u0027updating\u0027 or \u0027listing\u0027, \u0027auditing\u0027 is"}],"source_content_type":"text/x-python","patch_set":25,"id":"1b974a54_b19af036","line":765,"range":{"start_line":765,"start_character":16,"end_line":765,"end_character":41},"updated":"2023-09-29 16:43:17.000000000","message":"it\u0027s a shame that the var name does not match the header - can we pass shard_format around instead?","commit_id":"11803c122b27001d9365fe3dc6686ce271bfa372"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"a64253c0362cf5bfcb3cff22de1fc93e52b2a75b","unresolved":false,"context_lines":[{"line_number":762,"context_line":"            if shard_format \u003d\u003d \"namespace\" and not ("},{"line_number":763,"context_line":"                includes or marker or end_marker or reverse"},{"line_number":764,"context_line":"            ):"},{"line_number":765,"context_line":"                record_type \u003d \u0027namespace\u0027"},{"line_number":766,"context_line":"        if record_type \u003d\u003d \u0027namespace\u0027:"},{"line_number":767,"context_line":"            # For \u0027namespace\u0027 record type, container server only supports"},{"line_number":768,"context_line":"            # listing of states using \u0027updating\u0027 or \u0027listing\u0027, \u0027auditing\u0027 is"}],"source_content_type":"text/x-python","patch_set":25,"id":"a2290b1d_8c6bb085","line":765,"range":{"start_line":765,"start_character":16,"end_line":765,"end_character":41},"in_reply_to":"1b974a54_b19af036","updated":"2023-09-29 19:18:17.000000000","message":"Ack","commit_id":"11803c122b27001d9365fe3dc6686ce271bfa372"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"aef4a35e9f00ec228a3f60bfba95722c8d3e2bd6","unresolved":true,"context_lines":[{"line_number":773,"context_line":"            if is_deleted:"},{"line_number":774,"context_line":"                return HTTPNotFound(request\u003dreq, headers\u003dresp_headers)"},{"line_number":775,"context_line":"            resp_headers[\u0027X-Backend-Record-Type\u0027] \u003d \u0027shard\u0027"},{"line_number":776,"context_line":"            resp_headers[\u0027X-Backend-Record-Shard-Format\u0027] \u003d \u0027namespace\u0027"},{"line_number":777,"context_line":"            override_filter_hdr \u003d req.headers.get("},{"line_number":778,"context_line":"                \u0027x-backend-override-shard-name-filter\u0027, \u0027\u0027).lower()"},{"line_number":779,"context_line":"            if override_filter_hdr \u003d\u003d db_state \u003d\u003d \u0027sharded\u0027:"}],"source_content_type":"text/x-python","patch_set":25,"id":"6be718f0_e401cc8d","line":776,"updated":"2023-09-29 16:43:17.000000000","message":"+1 this looks better to me, but we should return same header for full shard_format too","commit_id":"11803c122b27001d9365fe3dc6686ce271bfa372"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"a64253c0362cf5bfcb3cff22de1fc93e52b2a75b","unresolved":false,"context_lines":[{"line_number":773,"context_line":"            if is_deleted:"},{"line_number":774,"context_line":"                return HTTPNotFound(request\u003dreq, headers\u003dresp_headers)"},{"line_number":775,"context_line":"            resp_headers[\u0027X-Backend-Record-Type\u0027] \u003d \u0027shard\u0027"},{"line_number":776,"context_line":"            resp_headers[\u0027X-Backend-Record-Shard-Format\u0027] \u003d \u0027namespace\u0027"},{"line_number":777,"context_line":"            override_filter_hdr \u003d req.headers.get("},{"line_number":778,"context_line":"                \u0027x-backend-override-shard-name-filter\u0027, \u0027\u0027).lower()"},{"line_number":779,"context_line":"            if override_filter_hdr \u003d\u003d db_state \u003d\u003d \u0027sharded\u0027:"}],"source_content_type":"text/x-python","patch_set":25,"id":"d5ab4e20_a9567e17","line":776,"in_reply_to":"6be718f0_e401cc8d","updated":"2023-09-29 19:18:17.000000000","message":"Ack","commit_id":"11803c122b27001d9365fe3dc6686ce271bfa372"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"aef4a35e9f00ec228a3f60bfba95722c8d3e2bd6","unresolved":true,"context_lines":[{"line_number":780,"context_line":"                # Only support sending back *all* ranges if the db is in"},{"line_number":781,"context_line":"                # sharded state, add the header for proxy checking purpose."},{"line_number":782,"context_line":"                resp_headers[\u0027X-Backend-Override-Shard-Name-Filter\u0027] \u003d \u0027true\u0027"},{"line_number":783,"context_line":"            fill_gaps \u003d False"},{"line_number":784,"context_line":"            if states:"},{"line_number":785,"context_line":"                states \u003d list_from_csv(states)"},{"line_number":786,"context_line":"                fill_gaps \u003d any((\u0027listing\u0027 in states, \u0027updating\u0027 in states))"},{"line_number":787,"context_line":"                try:"},{"line_number":788,"context_line":"                    states \u003d broker.resolve_shard_range_states(states)"},{"line_number":789,"context_line":"                except ValueError:"},{"line_number":790,"context_line":"                    return HTTPBadRequest(request\u003dreq, body\u003d\u0027Bad state\u0027)"},{"line_number":791,"context_line":"            container_list \u003d broker.get_namespaces(states, fill_gaps)"},{"line_number":792,"context_line":"        elif record_type \u003d\u003d \u0027shard\u0027:"},{"line_number":793,"context_line":"            override_deleted \u003d info and config_true_value("}],"source_content_type":"text/x-python","patch_set":25,"id":"f79e34fc_060bc72a","line":790,"range":{"start_line":783,"start_character":1,"end_line":790,"end_character":72},"updated":"2023-09-29 16:43:17.000000000","message":"there\u0027s plenty of duplicated code vs shard record_type","commit_id":"11803c122b27001d9365fe3dc6686ce271bfa372"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"a64253c0362cf5bfcb3cff22de1fc93e52b2a75b","unresolved":false,"context_lines":[{"line_number":780,"context_line":"                # Only support sending back *all* ranges if the db is in"},{"line_number":781,"context_line":"                # sharded state, add the header for proxy checking purpose."},{"line_number":782,"context_line":"                resp_headers[\u0027X-Backend-Override-Shard-Name-Filter\u0027] \u003d \u0027true\u0027"},{"line_number":783,"context_line":"            fill_gaps \u003d False"},{"line_number":784,"context_line":"            if states:"},{"line_number":785,"context_line":"                states \u003d list_from_csv(states)"},{"line_number":786,"context_line":"                fill_gaps \u003d any((\u0027listing\u0027 in states, \u0027updating\u0027 in states))"},{"line_number":787,"context_line":"                try:"},{"line_number":788,"context_line":"                    states \u003d broker.resolve_shard_range_states(states)"},{"line_number":789,"context_line":"                except ValueError:"},{"line_number":790,"context_line":"                    return HTTPBadRequest(request\u003dreq, body\u003d\u0027Bad state\u0027)"},{"line_number":791,"context_line":"            container_list \u003d broker.get_namespaces(states, fill_gaps)"},{"line_number":792,"context_line":"        elif record_type \u003d\u003d \u0027shard\u0027:"},{"line_number":793,"context_line":"            override_deleted \u003d info and config_true_value("}],"source_content_type":"text/x-python","patch_set":25,"id":"c42fd108_1b5b284f","line":790,"range":{"start_line":783,"start_character":1,"end_line":790,"end_character":72},"in_reply_to":"f79e34fc_060bc72a","updated":"2023-09-29 19:18:17.000000000","message":"thanks for the dedup patch! the if Namespace code branch started with a small piece of code, then grew bigger and bigger to the point that it had not much difference from ShardRange branch anymore.","commit_id":"11803c122b27001d9365fe3dc6686ce271bfa372"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":true,"context_lines":[{"line_number":611,"context_line":""},{"line_number":612,"context_line":"        :param record: object entry record"},{"line_number":613,"context_line":"        :param record_type: Either shard, auto, \u0027\u0027 or None. This value"},{"line_number":614,"context_line":"                             determines how it\u0027ll build the response dict."},{"line_number":615,"context_line":"        :param shard_record_format: Either \u0027namespace\u0027, \u0027full\u0027 or None."},{"line_number":616,"context_line":"        :returns: modified record"},{"line_number":617,"context_line":"        \"\"\""}],"source_content_type":"text/x-python","patch_set":27,"id":"949d2de9_095c9645","line":614,"updated":"2023-10-02 22:23:29.000000000","message":"this seems like we\u0027re doubling up the req.params parsing\n\nin the shard-range path if the param was auto it gets converted to shard\n\nso auto, \u0027\u0027 and None are all different ways to spell \"the recoreds are objects\"\n\n```\nThere should be one-- and preferably only one --obvious way to do it.\n```","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"535964202b2c8c0c3553c5a70c9c446fd66091fa","unresolved":false,"context_lines":[{"line_number":611,"context_line":""},{"line_number":612,"context_line":"        :param record: object entry record"},{"line_number":613,"context_line":"        :param record_type: Either shard, auto, \u0027\u0027 or None. This value"},{"line_number":614,"context_line":"                             determines how it\u0027ll build the response dict."},{"line_number":615,"context_line":"        :param shard_record_format: Either \u0027namespace\u0027, \u0027full\u0027 or None."},{"line_number":616,"context_line":"        :returns: modified record"},{"line_number":617,"context_line":"        \"\"\""}],"source_content_type":"text/x-python","patch_set":27,"id":"b10d46c9_443ab117","line":614,"in_reply_to":"18830d1b_269bcf50","updated":"2023-11-03 04:25:44.000000000","message":"The follow-up refactoring patch looks good to me, I got it squashed in. Thanks for the help, Clay!","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8a24e200d9cb30696c694717a3c20781843d2652","unresolved":true,"context_lines":[{"line_number":611,"context_line":""},{"line_number":612,"context_line":"        :param record: object entry record"},{"line_number":613,"context_line":"        :param record_type: Either shard, auto, \u0027\u0027 or None. This value"},{"line_number":614,"context_line":"                             determines how it\u0027ll build the response dict."},{"line_number":615,"context_line":"        :param shard_record_format: Either \u0027namespace\u0027, \u0027full\u0027 or None."},{"line_number":616,"context_line":"        :returns: modified record"},{"line_number":617,"context_line":"        \"\"\""}],"source_content_type":"text/x-python","patch_set":27,"id":"e3eed1cf_f46f331f","line":614,"in_reply_to":"3ded376e_6523eeff","updated":"2023-10-31 19:21:20.000000000","message":"ok, but I think in this version this method is only called with shard_type explicitly set to \u0027object\u0027 or \u0027shard\u0027","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"bcb52432e997cd207866f8b22a7c55a1f0a34204","unresolved":true,"context_lines":[{"line_number":611,"context_line":""},{"line_number":612,"context_line":"        :param record: object entry record"},{"line_number":613,"context_line":"        :param record_type: Either shard, auto, \u0027\u0027 or None. This value"},{"line_number":614,"context_line":"                             determines how it\u0027ll build the response dict."},{"line_number":615,"context_line":"        :param shard_record_format: Either \u0027namespace\u0027, \u0027full\u0027 or None."},{"line_number":616,"context_line":"        :returns: modified record"},{"line_number":617,"context_line":"        \"\"\""}],"source_content_type":"text/x-python","patch_set":27,"id":"3ded376e_6523eeff","line":614,"in_reply_to":"949d2de9_095c9645","updated":"2023-10-31 16:11:06.000000000","message":"@Alistair is refactoring this function in a follow-up patch.","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"a57812bb8061973f9c43b94ddf758ccac9d7d357","unresolved":true,"context_lines":[{"line_number":611,"context_line":""},{"line_number":612,"context_line":"        :param record: object entry record"},{"line_number":613,"context_line":"        :param record_type: Either shard, auto, \u0027\u0027 or None. This value"},{"line_number":614,"context_line":"                             determines how it\u0027ll build the response dict."},{"line_number":615,"context_line":"        :param shard_record_format: Either \u0027namespace\u0027, \u0027full\u0027 or None."},{"line_number":616,"context_line":"        :returns: modified record"},{"line_number":617,"context_line":"        \"\"\""}],"source_content_type":"text/x-python","patch_set":27,"id":"18830d1b_269bcf50","line":614,"in_reply_to":"e3eed1cf_f46f331f","updated":"2023-11-02 20:54:39.000000000","message":"To be 100% I\u0027m complaining about the doc string making assumptions about what it\u0027s caller\u0027s pass in that\u0027s not true - and irrelevant.\n\nNo one passes in None.  And no one *should* pass it \u0027\u0027 or \u0027auto\u0027 it just over-complicates the docstring.  The implementation doesn\u0027t even really care, there\u0027s only two branches and so WHY are they even one function!?\n\nThis passes tests and is less bad:\n\n\tdiff --git a/swift/container/server.py b/swift/container/server.py\n\tindex 9384eee23..e425e5daf 100644\n\t--- a/swift/container/server.py\n\t+++ b/swift/container/server.py\n\t@@ -610,11 +610,13 @@ class ContainerController(BaseStorageServer):\n\t\t Replaces size with \u0027swift_bytes\u0027 content type parameter.\n\t \n\t\t :param record: object entry record\n\t-        :param record_type: Either shard, auto, \u0027\u0027 or None. This value\n\t-                             determines how it\u0027ll build the response dict.\n\t+        :param record_type: str, \u0027shard\u0027 or \u0027object\u0027.  This value determines\n\t+                            how it\u0027ll build the response dict.\n\t\t :param shard_record_format: Either \u0027namespace\u0027, \u0027full\u0027 or None.\n\t\t :returns: modified record\n\t\t \"\"\"\n\t+        if record_type not in (\u0027shard\u0027, \u0027object\u0027):\n\t+            raise ValueError(\u0027unknown record_type: %r\u0027 % record_type)\n\t\t if record_type \u003d\u003d \u0027shard\u0027:\n\t\t     response \u003d dict(record)\n\t\t     if shard_record_format \u003d\u003d \u0027full\u0027:\n\t@@ -739,6 +741,8 @@ class ContainerController(BaseStorageServer):\n\t\t db_state \u003d info.get(\u0027db_state\u0027)\n\t\t if record_type \u003d\u003d \u0027auto\u0027 and db_state in (SHARDING, SHARDED):\n\t\t     record_type \u003d \u0027shard\u0027\n\t+        else:\n\t+            record_type \u003d \u0027object\u0027\n\t\t if record_type \u003d\u003d \u0027shard\u0027:\n\t\t     return self.GET_shard(req, broker, container, params, info,\n\t\t\t\t\t   is_deleted, out_content_type, record_type)\n\n\nbut it\u0027s still \"stringly-typed\" and IMHO, something like this would be actually *good*:\n\nhttps://review.opendev.org/c/openstack/swift/+/899983","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"1e51ac4adf5517da06b2312f597ecdedcae202b3","unresolved":true,"context_lines":[{"line_number":617,"context_line":"        \"\"\""},{"line_number":618,"context_line":"        if record_type \u003d\u003d \u0027shard\u0027:"},{"line_number":619,"context_line":"            response \u003d dict(record)"},{"line_number":620,"context_line":"            if shard_record_format \u003d\u003d \u0027full\u0027:"},{"line_number":621,"context_line":"                created \u003d record.timestamp"},{"line_number":622,"context_line":"                response[\u0027last_modified\u0027] \u003d Timestamp(created).isoformat"},{"line_number":623,"context_line":"        else:"}],"source_content_type":"text/x-python","patch_set":27,"id":"84facb76_060b45a0","line":620,"updated":"2023-10-02 18:14:58.000000000","message":"ooh, might have been nice to have defaulted to the full case here, BUT as previously noted, I don\u0027t think \u0027last_modified\u0027 is relevant anyway","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"bcb52432e997cd207866f8b22a7c55a1f0a34204","unresolved":false,"context_lines":[{"line_number":617,"context_line":"        \"\"\""},{"line_number":618,"context_line":"        if record_type \u003d\u003d \u0027shard\u0027:"},{"line_number":619,"context_line":"            response \u003d dict(record)"},{"line_number":620,"context_line":"            if shard_record_format \u003d\u003d \u0027full\u0027:"},{"line_number":621,"context_line":"                created \u003d record.timestamp"},{"line_number":622,"context_line":"                response[\u0027last_modified\u0027] \u003d Timestamp(created).isoformat"},{"line_number":623,"context_line":"        else:"}],"source_content_type":"text/x-python","patch_set":27,"id":"de9721d7_96f03536","line":620,"in_reply_to":"84facb76_060b45a0","updated":"2023-10-31 16:11:06.000000000","message":"Al already started follow-up patch to refactor this function.","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"4fbcf228954a81390f569f70a531c5bff3f910ed","unresolved":true,"context_lines":[{"line_number":750,"context_line":"            record_type \u003d \u0027shard\u0027"},{"line_number":751,"context_line":"        if record_type \u003d\u003d \u0027shard\u0027:"},{"line_number":752,"context_line":"            shard_format \u003d req.headers.get("},{"line_number":753,"context_line":"                \u0027x-backend-record-shard-format\u0027, \u0027full\u0027).lower()"},{"line_number":754,"context_line":"            # For record type of \u0027shard\u0027, user can specify an additional header"},{"line_number":755,"context_line":"            # to ask for list of Namespaces instead of full ShardRanges."},{"line_number":756,"context_line":"            # This will allow proxy server who is going to retrieve Namespace"}],"source_content_type":"text/x-python","patch_set":27,"id":"ffe87d8f_9f324fc5","line":753,"updated":"2023-10-03 17:56:56.000000000","message":"On the off-chance that we need to do something like this *again*, and the proxy would prefer, in order:\n\n- some fancy new format,\n- namespaces, or finally\n- shard ranges\n\nis there anything we can/should do _today_ so we won\u0027t end up adding yet another header later? I kind of wish we had done something along the lines of an [`Accept` header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept) originally, so *this* change could start sending something like `X-Backend-Record-Type: namespace;q\u003d1, shard;q\u003d.9, object;q\u003d.8`","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"7f7419d81f176f1e13b447e561bb43ebf264f128","unresolved":false,"context_lines":[{"line_number":750,"context_line":"            record_type \u003d \u0027shard\u0027"},{"line_number":751,"context_line":"        if record_type \u003d\u003d \u0027shard\u0027:"},{"line_number":752,"context_line":"            shard_format \u003d req.headers.get("},{"line_number":753,"context_line":"                \u0027x-backend-record-shard-format\u0027, \u0027full\u0027).lower()"},{"line_number":754,"context_line":"            # For record type of \u0027shard\u0027, user can specify an additional header"},{"line_number":755,"context_line":"            # to ask for list of Namespaces instead of full ShardRanges."},{"line_number":756,"context_line":"            # This will allow proxy server who is going to retrieve Namespace"}],"source_content_type":"text/x-python","patch_set":27,"id":"c68ada50_1bc4cd1f","line":753,"in_reply_to":"bb9a6026_42b01148","updated":"2024-01-09 18:59:48.000000000","message":"Had this discussed and decided not to do it. The new header also allow us to add new format in future, for example, we can add \u0027x-backend-record-shard-format \u003d binary-tree\u0027 if we decide to break listing of shard ranges into blobs.","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"b15761c0e0e2d7e69f93b97e5e7b7847078fdb4a","unresolved":true,"context_lines":[{"line_number":750,"context_line":"            record_type \u003d \u0027shard\u0027"},{"line_number":751,"context_line":"        if record_type \u003d\u003d \u0027shard\u0027:"},{"line_number":752,"context_line":"            shard_format \u003d req.headers.get("},{"line_number":753,"context_line":"                \u0027x-backend-record-shard-format\u0027, \u0027full\u0027).lower()"},{"line_number":754,"context_line":"            # For record type of \u0027shard\u0027, user can specify an additional header"},{"line_number":755,"context_line":"            # to ask for list of Namespaces instead of full ShardRanges."},{"line_number":756,"context_line":"            # This will allow proxy server who is going to retrieve Namespace"}],"source_content_type":"text/x-python","patch_set":27,"id":"bb9a6026_42b01148","line":753,"in_reply_to":"ffe87d8f_9f324fc5","updated":"2024-01-09 15:23:00.000000000","message":"do we have the appetite/resource to do this?","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"1e51ac4adf5517da06b2312f597ecdedcae202b3","unresolved":true,"context_lines":[{"line_number":765,"context_line":"            else:"},{"line_number":766,"context_line":"                shard_format \u003d \u0027full\u0027"},{"line_number":767,"context_line":"                override_deleted \u003d info and config_true_value("},{"line_number":768,"context_line":"                    req.headers.get(\u0027x-backend-override-deleted\u0027, False))"},{"line_number":769,"context_line":""},{"line_number":770,"context_line":"            resp_headers \u003d gen_resp_headers("},{"line_number":771,"context_line":"                info, is_deleted\u003dis_deleted and not override_deleted)"}],"source_content_type":"text/x-python","patch_set":27,"id":"b2e1daba_60dc6f93","line":768,"updated":"2023-10-02 18:14:58.000000000","message":"this is the key section w.r.t. upgrade: we\u0027ll only return namespaces if the proxy has explicitly set x-backend-record-shard-format\u003dnamespace AND there\u0027s no constraints on the listing, otherwise format\u003dfull which is legacy response","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":true,"context_lines":[{"line_number":765,"context_line":"            else:"},{"line_number":766,"context_line":"                shard_format \u003d \u0027full\u0027"},{"line_number":767,"context_line":"                override_deleted \u003d info and config_true_value("},{"line_number":768,"context_line":"                    req.headers.get(\u0027x-backend-override-deleted\u0027, False))"},{"line_number":769,"context_line":""},{"line_number":770,"context_line":"            resp_headers \u003d gen_resp_headers("},{"line_number":771,"context_line":"                info, is_deleted\u003dis_deleted and not override_deleted)"}],"source_content_type":"text/x-python","patch_set":27,"id":"be77f8cb_c9208acd","line":768,"in_reply_to":"b2e1daba_60dc6f93","updated":"2023-10-02 22:23:29.000000000","message":"seems a little strange to ignore the extra constrains and fall back to record-format\u003dfull\n\nif we ever wanted get namespace to learn how to handle includes, or marker/end_marker I would expect an explicit error from backend servers would be easier to deal with","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"bcb52432e997cd207866f8b22a7c55a1f0a34204","unresolved":false,"context_lines":[{"line_number":765,"context_line":"            else:"},{"line_number":766,"context_line":"                shard_format \u003d \u0027full\u0027"},{"line_number":767,"context_line":"                override_deleted \u003d info and config_true_value("},{"line_number":768,"context_line":"                    req.headers.get(\u0027x-backend-override-deleted\u0027, False))"},{"line_number":769,"context_line":""},{"line_number":770,"context_line":"            resp_headers \u003d gen_resp_headers("},{"line_number":771,"context_line":"                info, is_deleted\u003dis_deleted and not override_deleted)"}],"source_content_type":"text/x-python","patch_set":27,"id":"db548de1_90bc9cfc","line":768,"in_reply_to":"be77f8cb_c9208acd","updated":"2023-10-31 16:11:06.000000000","message":"this was discussed offline before.","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":true,"context_lines":[{"line_number":784,"context_line":"                reverse \u003d False"},{"line_number":785,"context_line":"            fill_gaps \u003d include_own \u003d False"},{"line_number":786,"context_line":"            if states:"},{"line_number":787,"context_line":"                states \u003d list_from_csv(states)"},{"line_number":788,"context_line":"                fill_gaps \u003d any((\u0027listing\u0027 in states, \u0027updating\u0027 in states))"},{"line_number":789,"context_line":"                # The \u0027auditing\u0027 state alias is used by the sharder during"},{"line_number":790,"context_line":"                # shard audit; if the shard is shrinking then it needs to get"}],"source_content_type":"text/x-python","patch_set":27,"id":"e89d49a7_b3fb1b7f","line":787,"updated":"2023-10-02 22:23:29.000000000","message":"it seems like this already normalizes any single value or None to an empty list; do we really need to support states\u003d10","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8a24e200d9cb30696c694717a3c20781843d2652","unresolved":false,"context_lines":[{"line_number":784,"context_line":"                reverse \u003d False"},{"line_number":785,"context_line":"            fill_gaps \u003d include_own \u003d False"},{"line_number":786,"context_line":"            if states:"},{"line_number":787,"context_line":"                states \u003d list_from_csv(states)"},{"line_number":788,"context_line":"                fill_gaps \u003d any((\u0027listing\u0027 in states, \u0027updating\u0027 in states))"},{"line_number":789,"context_line":"                # The \u0027auditing\u0027 state alias is used by the sharder during"},{"line_number":790,"context_line":"                # shard audit; if the shard is shrinking then it needs to get"}],"source_content_type":"text/x-python","patch_set":27,"id":"85a10c06_89323227","line":787,"in_reply_to":"308b76dc_ce80e751","updated":"2023-10-31 19:21:20.000000000","message":"i meant do we have to support states\u003d[10] AND states\u003d10","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"bcb52432e997cd207866f8b22a7c55a1f0a34204","unresolved":true,"context_lines":[{"line_number":784,"context_line":"                reverse \u003d False"},{"line_number":785,"context_line":"            fill_gaps \u003d include_own \u003d False"},{"line_number":786,"context_line":"            if states:"},{"line_number":787,"context_line":"                states \u003d list_from_csv(states)"},{"line_number":788,"context_line":"                fill_gaps \u003d any((\u0027listing\u0027 in states, \u0027updating\u0027 in states))"},{"line_number":789,"context_line":"                # The \u0027auditing\u0027 state alias is used by the sharder during"},{"line_number":790,"context_line":"                # shard audit; if the shard is shrinking then it needs to get"}],"source_content_type":"text/x-python","patch_set":27,"id":"308b76dc_ce80e751","line":787,"in_reply_to":"e89d49a7_b3fb1b7f","updated":"2023-10-31 16:11:06.000000000","message":"Good question, for states\u003d10 which is \u0027Found\u0027, get_shard_range supports it, so get_namespace tries to be compatible. I don\u0027t remember I ever saw a container DB with state \u0027Found\u0027 in prod, but we will see one if a very early stage container DB get stuck in some way?","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":true,"context_lines":[{"line_number":793,"context_line":"                # For \u0027namespace\u0027 record type, container server only supports"},{"line_number":794,"context_line":"                # listing of states using \u0027updating\u0027 or \u0027listing\u0027, \u0027auditing\u0027"},{"line_number":795,"context_line":"                # is not supported, for which caller should set"},{"line_number":796,"context_line":"                # \u0027x-backend-record-shard-format\u0027 to \u0027full\u0027 instead."},{"line_number":797,"context_line":"                include_own \u003d \u0027auditing\u0027 in states"},{"line_number":798,"context_line":"                try:"},{"line_number":799,"context_line":"                    states \u003d broker.resolve_shard_range_states(states)"}],"source_content_type":"text/x-python","patch_set":27,"id":"1637755e_ec78eeea","line":796,"updated":"2023-10-02 22:23:29.000000000","message":"according to this comment it seems like fill_gaps is always True?","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"ab133fd538b104017d417ae79e5e983aa083a419","unresolved":false,"context_lines":[{"line_number":793,"context_line":"                # For \u0027namespace\u0027 record type, container server only supports"},{"line_number":794,"context_line":"                # listing of states using \u0027updating\u0027 or \u0027listing\u0027, \u0027auditing\u0027"},{"line_number":795,"context_line":"                # is not supported, for which caller should set"},{"line_number":796,"context_line":"                # \u0027x-backend-record-shard-format\u0027 to \u0027full\u0027 instead."},{"line_number":797,"context_line":"                include_own \u003d \u0027auditing\u0027 in states"},{"line_number":798,"context_line":"                try:"},{"line_number":799,"context_line":"                    states \u003d broker.resolve_shard_range_states(states)"}],"source_content_type":"text/x-python","patch_set":27,"id":"734f6037_0aa99724","line":796,"in_reply_to":"1637755e_ec78eeea","updated":"2023-10-04 04:32:17.000000000","message":"\"fill_gaps\" is added per suggestions from Matt and Alistair, since we have to support all other methods (include, marker, end_mark and etc), this patch has it only for preparation.","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":true,"context_lines":[{"line_number":796,"context_line":"                # \u0027x-backend-record-shard-format\u0027 to \u0027full\u0027 instead."},{"line_number":797,"context_line":"                include_own \u003d \u0027auditing\u0027 in states"},{"line_number":798,"context_line":"                try:"},{"line_number":799,"context_line":"                    states \u003d broker.resolve_shard_range_states(states)"},{"line_number":800,"context_line":"                except ValueError:"},{"line_number":801,"context_line":"                    return HTTPBadRequest(request\u003dreq, body\u003d\u0027Bad state\u0027)"},{"line_number":802,"context_line":"            # This is also not supported for \u0027namespace\u0027 record type"}],"source_content_type":"text/x-python","patch_set":27,"id":"f4b56e3b_f9a88eb7","line":799,"updated":"2023-10-02 22:23:29.000000000","message":"OMG this method will take the emtpy list returned from list_from_csv and turn it back into None 😭","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bff0a54e68a76090810057a6fac976d6d5e3e745","unresolved":false,"context_lines":[{"line_number":796,"context_line":"                # \u0027x-backend-record-shard-format\u0027 to \u0027full\u0027 instead."},{"line_number":797,"context_line":"                include_own \u003d \u0027auditing\u0027 in states"},{"line_number":798,"context_line":"                try:"},{"line_number":799,"context_line":"                    states \u003d broker.resolve_shard_range_states(states)"},{"line_number":800,"context_line":"                except ValueError:"},{"line_number":801,"context_line":"                    return HTTPBadRequest(request\u003dreq, body\u003d\u0027Bad state\u0027)"},{"line_number":802,"context_line":"            # This is also not supported for \u0027namespace\u0027 record type"}],"source_content_type":"text/x-python","patch_set":27,"id":"8ba0796c_c85bcc5e","line":799,"in_reply_to":"f4b56e3b_f9a88eb7","updated":"2024-01-04 16:20:07.000000000","message":"Done","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":true,"context_lines":[{"line_number":801,"context_line":"                    return HTTPBadRequest(request\u003dreq, body\u003d\u0027Bad state\u0027)"},{"line_number":802,"context_line":"            # This is also not supported for \u0027namespace\u0027 record type"},{"line_number":803,"context_line":"            include_deleted \u003d config_true_value("},{"line_number":804,"context_line":"                req.headers.get(\u0027x-backend-include-deleted\u0027, False))"},{"line_number":805,"context_line":""},{"line_number":806,"context_line":"            if shard_format \u003d\u003d \u0027namespace\u0027:"},{"line_number":807,"context_line":"                container_list \u003d broker.get_namespaces(states, fill_gaps)"}],"source_content_type":"text/x-python","patch_set":27,"id":"f15ccc5e_a9bb03ff","line":804,"updated":"2023-10-02 22:23:29.000000000","message":"there\u0027s almost 50 lines of params parsing for shard_ranges and we\u0027re now sneaking/interleaving support for namespaces record-format (whose api supports a different subset of params)\n\nI think it might look better as:\n\n```\nformat \u003d headers.get(\u0027format\u0027, \u0027full\u0027)\nif format \u003d\u003d \u0027namespace\u0027:\n    params \u003d self._parse_namespace_params(...)\n    container_list \u003d broker.get_namespace(**params)\nelse:\n    # everyone can deal with full!\n    params \u003d self._parse_shard_range_params(...)\n    container_list \u003d broker.get_shard_ranges(**params)\n\n```\n\nI thougth about this more and returning format\u003dfull is a good fallback","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8a24e200d9cb30696c694717a3c20781843d2652","unresolved":false,"context_lines":[{"line_number":801,"context_line":"                    return HTTPBadRequest(request\u003dreq, body\u003d\u0027Bad state\u0027)"},{"line_number":802,"context_line":"            # This is also not supported for \u0027namespace\u0027 record type"},{"line_number":803,"context_line":"            include_deleted \u003d config_true_value("},{"line_number":804,"context_line":"                req.headers.get(\u0027x-backend-include-deleted\u0027, False))"},{"line_number":805,"context_line":""},{"line_number":806,"context_line":"            if shard_format \u003d\u003d \u0027namespace\u0027:"},{"line_number":807,"context_line":"                container_list \u003d broker.get_namespaces(states, fill_gaps)"}],"source_content_type":"text/x-python","patch_set":27,"id":"75eeef68_e7dbf246","line":804,"in_reply_to":"53cd547f_bb900b51","updated":"2023-10-31 19:21:20.000000000","message":"since we sometimes fallback to format\u003dfull when certain params are present it makes sense to combine the param parsing:\n\n\tformat, params \u003d self._parse_get_shard_params(...)\n\tif format \u003d\u003d \u0027namespace\u0027:\n\t    container_list \u003d broker.get_namespaces(**params)\n\telse:\n\t    container_list \u003d broker.get_shard_ranges(**params)\n\n... and that\u0027s more or less what GET_shards does","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"ab133fd538b104017d417ae79e5e983aa083a419","unresolved":true,"context_lines":[{"line_number":801,"context_line":"                    return HTTPBadRequest(request\u003dreq, body\u003d\u0027Bad state\u0027)"},{"line_number":802,"context_line":"            # This is also not supported for \u0027namespace\u0027 record type"},{"line_number":803,"context_line":"            include_deleted \u003d config_true_value("},{"line_number":804,"context_line":"                req.headers.get(\u0027x-backend-include-deleted\u0027, False))"},{"line_number":805,"context_line":""},{"line_number":806,"context_line":"            if shard_format \u003d\u003d \u0027namespace\u0027:"},{"line_number":807,"context_line":"                container_list \u003d broker.get_namespaces(states, fill_gaps)"}],"source_content_type":"text/x-python","patch_set":27,"id":"53cd547f_bb900b51","line":804,"in_reply_to":"f15ccc5e_a9bb03ff","updated":"2023-10-04 04:32:17.000000000","message":"yeah, the style you are suggesting is exactly this patch was using until #25. \u0027namespace\u0027 code branch was small to start with, but grew bigger to look like \u0027ShardRange\u0027 branch more and more, then Alistair and I decided to merge those two branches.","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"1e51ac4adf5517da06b2312f597ecdedcae202b3","unresolved":true,"context_lines":[{"line_number":804,"context_line":"                req.headers.get(\u0027x-backend-include-deleted\u0027, False))"},{"line_number":805,"context_line":""},{"line_number":806,"context_line":"            if shard_format \u003d\u003d \u0027namespace\u0027:"},{"line_number":807,"context_line":"                container_list \u003d broker.get_namespaces(states, fill_gaps)"},{"line_number":808,"context_line":"            else:"},{"line_number":809,"context_line":"                container_list \u003d broker.get_shard_ranges("},{"line_number":810,"context_line":"                    marker, end_marker, includes, reverse, states\u003dstates,"}],"source_content_type":"text/x-python","patch_set":27,"id":"8fd88928_c8bc45b1","line":807,"updated":"2023-10-02 18:14:58.000000000","message":"new path","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"bcb52432e997cd207866f8b22a7c55a1f0a34204","unresolved":false,"context_lines":[{"line_number":804,"context_line":"                req.headers.get(\u0027x-backend-include-deleted\u0027, False))"},{"line_number":805,"context_line":""},{"line_number":806,"context_line":"            if shard_format \u003d\u003d \u0027namespace\u0027:"},{"line_number":807,"context_line":"                container_list \u003d broker.get_namespaces(states, fill_gaps)"},{"line_number":808,"context_line":"            else:"},{"line_number":809,"context_line":"                container_list \u003d broker.get_shard_ranges("},{"line_number":810,"context_line":"                    marker, end_marker, includes, reverse, states\u003dstates,"}],"source_content_type":"text/x-python","patch_set":27,"id":"8c67ea7a_a29ac020","line":807,"in_reply_to":"8fd88928_c8bc45b1","updated":"2023-10-31 16:11:06.000000000","message":"Ack","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"1e51ac4adf5517da06b2312f597ecdedcae202b3","unresolved":true,"context_lines":[{"line_number":809,"context_line":"                container_list \u003d broker.get_shard_ranges("},{"line_number":810,"context_line":"                    marker, end_marker, includes, reverse, states\u003dstates,"},{"line_number":811,"context_line":"                    include_deleted\u003dinclude_deleted, fill_gaps\u003dfill_gaps,"},{"line_number":812,"context_line":"                    include_own\u003dinclude_own)"},{"line_number":813,"context_line":"        else:"},{"line_number":814,"context_line":"            shard_format \u003d None"},{"line_number":815,"context_line":"            requested_policy_index \u003d self.get_and_validate_policy_index(req)"}],"source_content_type":"text/x-python","patch_set":27,"id":"a01f3d3f_75f63a07","line":812,"updated":"2023-10-02 18:14:58.000000000","message":"default is legacy response path","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"bcb52432e997cd207866f8b22a7c55a1f0a34204","unresolved":false,"context_lines":[{"line_number":809,"context_line":"                container_list \u003d broker.get_shard_ranges("},{"line_number":810,"context_line":"                    marker, end_marker, includes, reverse, states\u003dstates,"},{"line_number":811,"context_line":"                    include_deleted\u003dinclude_deleted, fill_gaps\u003dfill_gaps,"},{"line_number":812,"context_line":"                    include_own\u003dinclude_own)"},{"line_number":813,"context_line":"        else:"},{"line_number":814,"context_line":"            shard_format \u003d None"},{"line_number":815,"context_line":"            requested_policy_index \u003d self.get_and_validate_policy_index(req)"}],"source_content_type":"text/x-python","patch_set":27,"id":"0894105b_3dba8355","line":812,"in_reply_to":"a01f3d3f_75f63a07","updated":"2023-10-31 16:11:06.000000000","message":"Ack","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":true,"context_lines":[{"line_number":843,"context_line":""},{"line_number":844,"context_line":"        listing \u003d [self.update_data_record("},{"line_number":845,"context_line":"            record, record_type, shard_record_format)"},{"line_number":846,"context_line":"            for record in container_list]"},{"line_number":847,"context_line":"        if out_content_type.endswith(\u0027/xml\u0027):"},{"line_number":848,"context_line":"            body \u003d listing_formats.container_to_xml(listing, container)"},{"line_number":849,"context_line":"        elif out_content_type.endswith(\u0027/json\u0027):"}],"source_content_type":"text/x-python","patch_set":27,"id":"9530dba4_48cd2270","line":846,"updated":"2023-10-02 22:23:29.000000000","message":"so the broker path is pretty clearly separate; but the container-server path intermixes the object/shard-range/namespace GET paths quite a bit","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"ab133fd538b104017d417ae79e5e983aa083a419","unresolved":false,"context_lines":[{"line_number":843,"context_line":""},{"line_number":844,"context_line":"        listing \u003d [self.update_data_record("},{"line_number":845,"context_line":"            record, record_type, shard_record_format)"},{"line_number":846,"context_line":"            for record in container_list]"},{"line_number":847,"context_line":"        if out_content_type.endswith(\u0027/xml\u0027):"},{"line_number":848,"context_line":"            body \u003d listing_formats.container_to_xml(listing, container)"},{"line_number":849,"context_line":"        elif out_content_type.endswith(\u0027/json\u0027):"}],"source_content_type":"text/x-python","patch_set":27,"id":"af24384f_16035a43","line":846,"in_reply_to":"9530dba4_48cd2270","updated":"2023-10-04 04:32:17.000000000","message":"Ack","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8a24e200d9cb30696c694717a3c20781843d2652","unresolved":true,"context_lines":[{"line_number":630,"context_line":"                \u0027bytes\u0027: size, \u0027hash\u0027: etag, \u0027name\u0027: name_,"},{"line_number":631,"context_line":"                \u0027content_type\u0027: content_type}"},{"line_number":632,"context_line":"            override_bytes_from_content_type(response, logger\u003dself.logger)"},{"line_number":633,"context_line":"            response[\u0027last_modified\u0027] \u003d Timestamp(created).isoformat"},{"line_number":634,"context_line":"        return response"},{"line_number":635,"context_line":""},{"line_number":636,"context_line":"    @public"}],"source_content_type":"text/x-python","patch_set":29,"id":"75af33c5_c45f3fe2","line":633,"updated":"2023-10-31 19:21:20.000000000","message":"these methods seem pretty disjoint\n\nin the shard case we convert the record to a dict and return it mostly un-modified except for some special handling in full wrt last_modified timestamp\n\nfor the object case we unpack as tuple and re-write into a dict who\u0027s keys are different from the database column names i\u0027m guessing...\n\nSo why are we forcing them ito one method?","commit_id":"95dcd77bf6b77f83ab3471d40b0f658ee5c5014f"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"a57812bb8061973f9c43b94ddf758ccac9d7d357","unresolved":false,"context_lines":[{"line_number":630,"context_line":"                \u0027bytes\u0027: size, \u0027hash\u0027: etag, \u0027name\u0027: name_,"},{"line_number":631,"context_line":"                \u0027content_type\u0027: content_type}"},{"line_number":632,"context_line":"            override_bytes_from_content_type(response, logger\u003dself.logger)"},{"line_number":633,"context_line":"            response[\u0027last_modified\u0027] \u003d Timestamp(created).isoformat"},{"line_number":634,"context_line":"        return response"},{"line_number":635,"context_line":""},{"line_number":636,"context_line":"    @public"}],"source_content_type":"text/x-python","patch_set":29,"id":"94e536bb_dcade212","line":633,"in_reply_to":"75af33c5_c45f3fe2","updated":"2023-11-02 20:54:39.000000000","message":"Ack","commit_id":"95dcd77bf6b77f83ab3471d40b0f658ee5c5014f"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8a24e200d9cb30696c694717a3c20781843d2652","unresolved":true,"context_lines":[{"line_number":744,"context_line":"                                  is_deleted, out_content_type, record_type)"},{"line_number":745,"context_line":"        else:"},{"line_number":746,"context_line":"            return self.GET_object(req, broker, container, params, info,"},{"line_number":747,"context_line":"                                   is_deleted, out_content_type, record_type)"},{"line_number":748,"context_line":""},{"line_number":749,"context_line":"    def GET_shard(self, req, broker, container, params, info,"},{"line_number":750,"context_line":"                  is_deleted, out_content_type, record_type):"}],"source_content_type":"text/x-python","patch_set":29,"id":"fb380a33_d151e3f5","line":747,"updated":"2023-10-31 19:21:20.000000000","message":"I think breaking the GET path into separate handlers is a good thing.  KUDOS.","commit_id":"95dcd77bf6b77f83ab3471d40b0f658ee5c5014f"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"b263eb52a246427f54c4caf4fda57755aa96fa73","unresolved":false,"context_lines":[{"line_number":744,"context_line":"                                  is_deleted, out_content_type, record_type)"},{"line_number":745,"context_line":"        else:"},{"line_number":746,"context_line":"            return self.GET_object(req, broker, container, params, info,"},{"line_number":747,"context_line":"                                   is_deleted, out_content_type, record_type)"},{"line_number":748,"context_line":""},{"line_number":749,"context_line":"    def GET_shard(self, req, broker, container, params, info,"},{"line_number":750,"context_line":"                  is_deleted, out_content_type, record_type):"}],"source_content_type":"text/x-python","patch_set":29,"id":"ab03c6e1_f1ebfcd9","line":747,"in_reply_to":"fb380a33_d151e3f5","updated":"2023-11-02 00:49:07.000000000","message":"Ack","commit_id":"95dcd77bf6b77f83ab3471d40b0f658ee5c5014f"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8a24e200d9cb30696c694717a3c20781843d2652","unresolved":true,"context_lines":[{"line_number":853,"context_line":"        prefix \u003d params.get(\u0027prefix\u0027)"},{"line_number":854,"context_line":"        delimiter \u003d params.get(\u0027delimiter\u0027)"},{"line_number":855,"context_line":"        limit \u003d params[\u0027limit\u0027]"},{"line_number":856,"context_line":"        shard_format \u003d None"},{"line_number":857,"context_line":"        requested_policy_index \u003d self.get_and_validate_policy_index(req)"},{"line_number":858,"context_line":"        resp_headers \u003d gen_resp_headers(info, is_deleted\u003dis_deleted)"},{"line_number":859,"context_line":"        if is_deleted:"}],"source_content_type":"text/x-python","patch_set":29,"id":"0dbafd59_4563cfdb","line":856,"updated":"2023-10-31 19:21:20.000000000","message":"probably not needed since create_listing already has a default kwarg for shard_format\u003dNone","commit_id":"95dcd77bf6b77f83ab3471d40b0f658ee5c5014f"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"b263eb52a246427f54c4caf4fda57755aa96fa73","unresolved":false,"context_lines":[{"line_number":853,"context_line":"        prefix \u003d params.get(\u0027prefix\u0027)"},{"line_number":854,"context_line":"        delimiter \u003d params.get(\u0027delimiter\u0027)"},{"line_number":855,"context_line":"        limit \u003d params[\u0027limit\u0027]"},{"line_number":856,"context_line":"        shard_format \u003d None"},{"line_number":857,"context_line":"        requested_policy_index \u003d self.get_and_validate_policy_index(req)"},{"line_number":858,"context_line":"        resp_headers \u003d gen_resp_headers(info, is_deleted\u003dis_deleted)"},{"line_number":859,"context_line":"        if is_deleted:"}],"source_content_type":"text/x-python","patch_set":29,"id":"ad3d4bcf_69fccb31","line":856,"in_reply_to":"0dbafd59_4563cfdb","updated":"2023-11-02 00:49:07.000000000","message":"Ack","commit_id":"95dcd77bf6b77f83ab3471d40b0f658ee5c5014f"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"c911d8629cb67aa7956424dc2df2e5efe5e2c5d4","unresolved":true,"context_lines":[{"line_number":635,"context_line":"        response \u003d {"},{"line_number":636,"context_line":"            \u0027bytes\u0027: size, \u0027hash\u0027: etag, \u0027name\u0027: name_,"},{"line_number":637,"context_line":"            \u0027content_type\u0027: content_type}"},{"line_number":638,"context_line":"        override_bytes_from_content_type(response, logger\u003dself.logger)"},{"line_number":639,"context_line":"        response[\u0027last_modified\u0027] \u003d Timestamp(created).isoformat"},{"line_number":640,"context_line":"        return response"},{"line_number":641,"context_line":""}],"source_content_type":"text/x-python","patch_set":34,"id":"5f2965bc_e061e0ab","line":638,"updated":"2023-11-03 10:56:20.000000000","message":"ok! this has pretty much reverted to how it was before sharding","commit_id":"d5a82e6874a0c73b9d55c5bb5b1c42fb8a90e230"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"18e816986da0f7d7dbb1575c0927744a62acb048","unresolved":false,"context_lines":[{"line_number":635,"context_line":"        response \u003d {"},{"line_number":636,"context_line":"            \u0027bytes\u0027: size, \u0027hash\u0027: etag, \u0027name\u0027: name_,"},{"line_number":637,"context_line":"            \u0027content_type\u0027: content_type}"},{"line_number":638,"context_line":"        override_bytes_from_content_type(response, logger\u003dself.logger)"},{"line_number":639,"context_line":"        response[\u0027last_modified\u0027] \u003d Timestamp(created).isoformat"},{"line_number":640,"context_line":"        return response"},{"line_number":641,"context_line":""}],"source_content_type":"text/x-python","patch_set":34,"id":"a9da8e96_0bfc069f","line":638,"in_reply_to":"5f2965bc_e061e0ab","updated":"2023-11-10 19:48:08.000000000","message":"Ack","commit_id":"d5a82e6874a0c73b9d55c5bb5b1c42fb8a90e230"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"c911d8629cb67aa7956424dc2df2e5efe5e2c5d4","unresolved":true,"context_lines":[{"line_number":702,"context_line":"          ``listing`` will cause the listing to include all shard ranges in a"},{"line_number":703,"context_line":"          state suitable for contributing to an object listing; ``updating``"},{"line_number":704,"context_line":"          will cause the listing to include all shard ranges in a state"},{"line_number":705,"context_line":"          suitable to accept an object update."},{"line_number":706,"context_line":""},{"line_number":707,"context_line":"          If either of these aliases is used then the shard range listing will"},{"line_number":708,"context_line":"          if necessary be extended with a synthesised \u0027filler\u0027 range in order"}],"source_content_type":"text/x-python","patch_set":34,"id":"de892999_75d5a797","line":705,"updated":"2023-11-03 10:56:20.000000000","message":"this is missing \u0027auditing\u0027","commit_id":"d5a82e6874a0c73b9d55c5bb5b1c42fb8a90e230"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"842bbe0a38788868d7d93cd1a743428f8631947a","unresolved":false,"context_lines":[{"line_number":702,"context_line":"          ``listing`` will cause the listing to include all shard ranges in a"},{"line_number":703,"context_line":"          state suitable for contributing to an object listing; ``updating``"},{"line_number":704,"context_line":"          will cause the listing to include all shard ranges in a state"},{"line_number":705,"context_line":"          suitable to accept an object update."},{"line_number":706,"context_line":""},{"line_number":707,"context_line":"          If either of these aliases is used then the shard range listing will"},{"line_number":708,"context_line":"          if necessary be extended with a synthesised \u0027filler\u0027 range in order"}],"source_content_type":"text/x-python","patch_set":34,"id":"22b7a2ac_4d185aa7","line":705,"in_reply_to":"de892999_75d5a797","updated":"2023-12-11 19:00:06.000000000","message":"Done","commit_id":"d5a82e6874a0c73b9d55c5bb5b1c42fb8a90e230"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"abdaf1ad1905cacd3520e5dd5c60706bbac6553e","unresolved":false,"context_lines":[{"line_number":702,"context_line":"          ``listing`` will cause the listing to include all shard ranges in a"},{"line_number":703,"context_line":"          state suitable for contributing to an object listing; ``updating``"},{"line_number":704,"context_line":"          will cause the listing to include all shard ranges in a state"},{"line_number":705,"context_line":"          suitable to accept an object update."},{"line_number":706,"context_line":""},{"line_number":707,"context_line":"          If either of these aliases is used then the shard range listing will"},{"line_number":708,"context_line":"          if necessary be extended with a synthesised \u0027filler\u0027 range in order"}],"source_content_type":"text/x-python","patch_set":34,"id":"368501a8_b57e972d","line":705,"in_reply_to":"de892999_75d5a797","updated":"2023-11-06 05:04:07.000000000","message":"Done","commit_id":"d5a82e6874a0c73b9d55c5bb5b1c42fb8a90e230"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"c911d8629cb67aa7956424dc2df2e5efe5e2c5d4","unresolved":true,"context_lines":[{"line_number":815,"context_line":"            # For \u0027namespace\u0027 record type, container server only supports"},{"line_number":816,"context_line":"            # listing of states using \u0027updating\u0027 or \u0027listing\u0027, \u0027auditing\u0027"},{"line_number":817,"context_line":"            # is not supported, for which caller should set"},{"line_number":818,"context_line":"            # \u0027x-backend-record-shard-format\u0027 to \u0027full\u0027 instead."},{"line_number":819,"context_line":"            include_own \u003d \u0027auditing\u0027 in states"},{"line_number":820,"context_line":"            try:"},{"line_number":821,"context_line":"                states \u003d broker.resolve_shard_range_states(states)"}],"source_content_type":"text/x-python","patch_set":34,"id":"cb7c7555_588ac912","line":818,"updated":"2023-11-03 10:56:20.000000000","message":":( namespaces *are* returned if states\u003d\u003d\u0027auditing\u0027, but they won\u0027t include own shard range","commit_id":"d5a82e6874a0c73b9d55c5bb5b1c42fb8a90e230"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"18e816986da0f7d7dbb1575c0927744a62acb048","unresolved":false,"context_lines":[{"line_number":815,"context_line":"            # For \u0027namespace\u0027 record type, container server only supports"},{"line_number":816,"context_line":"            # listing of states using \u0027updating\u0027 or \u0027listing\u0027, \u0027auditing\u0027"},{"line_number":817,"context_line":"            # is not supported, for which caller should set"},{"line_number":818,"context_line":"            # \u0027x-backend-record-shard-format\u0027 to \u0027full\u0027 instead."},{"line_number":819,"context_line":"            include_own \u003d \u0027auditing\u0027 in states"},{"line_number":820,"context_line":"            try:"},{"line_number":821,"context_line":"                states \u003d broker.resolve_shard_range_states(states)"}],"source_content_type":"text/x-python","patch_set":34,"id":"e5998101_35c93c46","line":818,"in_reply_to":"83cdc454_10744538","updated":"2023-11-10 19:48:08.000000000","message":"Ack","commit_id":"d5a82e6874a0c73b9d55c5bb5b1c42fb8a90e230"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"abdaf1ad1905cacd3520e5dd5c60706bbac6553e","unresolved":true,"context_lines":[{"line_number":815,"context_line":"            # For \u0027namespace\u0027 record type, container server only supports"},{"line_number":816,"context_line":"            # listing of states using \u0027updating\u0027 or \u0027listing\u0027, \u0027auditing\u0027"},{"line_number":817,"context_line":"            # is not supported, for which caller should set"},{"line_number":818,"context_line":"            # \u0027x-backend-record-shard-format\u0027 to \u0027full\u0027 instead."},{"line_number":819,"context_line":"            include_own \u003d \u0027auditing\u0027 in states"},{"line_number":820,"context_line":"            try:"},{"line_number":821,"context_line":"                states \u003d broker.resolve_shard_range_states(states)"}],"source_content_type":"text/x-python","patch_set":34,"id":"83cdc454_10744538","line":818,"in_reply_to":"cb7c7555_588ac912","updated":"2023-11-06 05:04:07.000000000","message":"I added comments in GET docstring.","commit_id":"d5a82e6874a0c73b9d55c5bb5b1c42fb8a90e230"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"c911d8629cb67aa7956424dc2df2e5efe5e2c5d4","unresolved":true,"context_lines":[{"line_number":823,"context_line":"                return HTTPBadRequest(request\u003dreq, body\u003d\u0027Bad state\u0027)"},{"line_number":824,"context_line":"        # This is also not supported for \u0027namespace\u0027 record type"},{"line_number":825,"context_line":"        include_deleted \u003d config_true_value("},{"line_number":826,"context_line":"            req.headers.get(\u0027x-backend-include-deleted\u0027, False))"},{"line_number":827,"context_line":""},{"line_number":828,"context_line":"        if shard_format \u003d\u003d \u0027namespace\u0027:"},{"line_number":829,"context_line":"            shard_format_full \u003d False"}],"source_content_type":"text/x-python","patch_set":34,"id":"7a7f4de1_237a46b9","line":826,"updated":"2023-11-03 10:56:20.000000000","message":"I can\u0027t imagine that we will *ever* support this for namespaces because the data structure has no way to indicate that the namespace is deleted. I therefore think that there is a case for returning 400 if this header is used with x-backend-record-shard-format\u003dnamespace.","commit_id":"d5a82e6874a0c73b9d55c5bb5b1c42fb8a90e230"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"abdaf1ad1905cacd3520e5dd5c60706bbac6553e","unresolved":false,"context_lines":[{"line_number":823,"context_line":"                return HTTPBadRequest(request\u003dreq, body\u003d\u0027Bad state\u0027)"},{"line_number":824,"context_line":"        # This is also not supported for \u0027namespace\u0027 record type"},{"line_number":825,"context_line":"        include_deleted \u003d config_true_value("},{"line_number":826,"context_line":"            req.headers.get(\u0027x-backend-include-deleted\u0027, False))"},{"line_number":827,"context_line":""},{"line_number":828,"context_line":"        if shard_format \u003d\u003d \u0027namespace\u0027:"},{"line_number":829,"context_line":"            shard_format_full \u003d False"}],"source_content_type":"text/x-python","patch_set":34,"id":"f4055827_db04afad","line":826,"in_reply_to":"7a7f4de1_237a46b9","updated":"2023-11-06 05:04:07.000000000","message":"added returning of 400 and related test case.","commit_id":"d5a82e6874a0c73b9d55c5bb5b1c42fb8a90e230"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"c911d8629cb67aa7956424dc2df2e5efe5e2c5d4","unresolved":true,"context_lines":[{"line_number":882,"context_line":"        return self.create_resp(req, out_content_type, info, resp_headers,"},{"line_number":883,"context_line":"                                broker.metadata, container, listing)"},{"line_number":884,"context_line":""},{"line_number":885,"context_line":"    def create_resp(self, req, out_content_type, info, resp_headers,"},{"line_number":886,"context_line":"                    metadata, container, listing):"},{"line_number":887,"context_line":"        for key, (value, _timestamp) in metadata.items():"},{"line_number":888,"context_line":"            if value and (key.lower() in self.save_headers or"}],"source_content_type":"text/x-python","patch_set":34,"id":"6a1522c2_f501267e","line":885,"updated":"2023-11-03 10:56:20.000000000","message":"can we use a less ambiguous method name? This method specifically creates a listing response (so _create_listing_response or _create_GET_response)\n\nA different proposed patch [1] is adding _create_ok_resp (note, that one is underscored, so let\u0027s also be consistent). When I look at the module/class outline it\u0027s really helpful to be able to easily identify the method of interest.\n\n[1] https://review.opendev.org/c/openstack/swift/+/898700/comment/674d0eab_d9f8959a/","commit_id":"d5a82e6874a0c73b9d55c5bb5b1c42fb8a90e230"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"abdaf1ad1905cacd3520e5dd5c60706bbac6553e","unresolved":false,"context_lines":[{"line_number":882,"context_line":"        return self.create_resp(req, out_content_type, info, resp_headers,"},{"line_number":883,"context_line":"                                broker.metadata, container, listing)"},{"line_number":884,"context_line":""},{"line_number":885,"context_line":"    def create_resp(self, req, out_content_type, info, resp_headers,"},{"line_number":886,"context_line":"                    metadata, container, listing):"},{"line_number":887,"context_line":"        for key, (value, _timestamp) in metadata.items():"},{"line_number":888,"context_line":"            if value and (key.lower() in self.save_headers or"}],"source_content_type":"text/x-python","patch_set":34,"id":"069720ec_486c8158","line":885,"in_reply_to":"6a1522c2_f501267e","updated":"2023-11-06 05:04:07.000000000","message":"Done","commit_id":"d5a82e6874a0c73b9d55c5bb5b1c42fb8a90e230"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"e367c6989582ab6ef04e286014b46f4c1023eb77","unresolved":true,"context_lines":[{"line_number":605,"context_line":"        Return the shard_range database record as a dict, the keys will depend"},{"line_number":606,"context_line":"        on the database fields provided in the record."},{"line_number":607,"context_line":""},{"line_number":608,"context_line":"        :param record: shard entry record"},{"line_number":609,"context_line":"        :param shard_record_full: boolean, when true the timestamp field is"},{"line_number":610,"context_line":"                                  added as \"last_modified\" in iso format."},{"line_number":611,"context_line":"        :returns: modified record"}],"source_content_type":"text/x-python","patch_set":38,"id":"0b746891_e572f166","line":608,"updated":"2023-11-13 23:38:06.000000000","message":"Might be worth noting that these may be *either* `ShardRange`s *or* `Namespace`s.","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"6641867462956b530bfcb3af9268e1a8f3f322eb","unresolved":false,"context_lines":[{"line_number":605,"context_line":"        Return the shard_range database record as a dict, the keys will depend"},{"line_number":606,"context_line":"        on the database fields provided in the record."},{"line_number":607,"context_line":""},{"line_number":608,"context_line":"        :param record: shard entry record"},{"line_number":609,"context_line":"        :param shard_record_full: boolean, when true the timestamp field is"},{"line_number":610,"context_line":"                                  added as \"last_modified\" in iso format."},{"line_number":611,"context_line":"        :returns: modified record"}],"source_content_type":"text/x-python","patch_set":38,"id":"40ac2a07_e1fc61f0","line":608,"in_reply_to":"0b746891_e572f166","updated":"2023-11-14 01:59:11.000000000","message":"Ack","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"e367c6989582ab6ef04e286014b46f4c1023eb77","unresolved":true,"context_lines":[{"line_number":607,"context_line":""},{"line_number":608,"context_line":"        :param record: shard entry record"},{"line_number":609,"context_line":"        :param shard_record_full: boolean, when true the timestamp field is"},{"line_number":610,"context_line":"                                  added as \"last_modified\" in iso format."},{"line_number":611,"context_line":"        :returns: modified record"},{"line_number":612,"context_line":"        \"\"\""},{"line_number":613,"context_line":"        response \u003d dict(record)"}],"source_content_type":"text/x-python","patch_set":38,"id":"f56c6f40_a4eabff5","line":610,"updated":"2023-11-13 23:38:06.000000000","message":"This should be exactly equivalent to `isinstance(record, ShardRange)`, yeah? Presumably, though, carrying the bool around is cheaper than repeated `isinstance` checks...","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"3f4812df4982d0832906bdbd476dc32a24c0a01f","unresolved":false,"context_lines":[{"line_number":607,"context_line":""},{"line_number":608,"context_line":"        :param record: shard entry record"},{"line_number":609,"context_line":"        :param shard_record_full: boolean, when true the timestamp field is"},{"line_number":610,"context_line":"                                  added as \"last_modified\" in iso format."},{"line_number":611,"context_line":"        :returns: modified record"},{"line_number":612,"context_line":"        \"\"\""},{"line_number":613,"context_line":"        response \u003d dict(record)"}],"source_content_type":"text/x-python","patch_set":38,"id":"c7d84226_7e94148d","line":610,"in_reply_to":"5ce69b61_3db6f1d2","updated":"2023-11-14 18:57:34.000000000","message":"\u003e Good to know that carrying and checking the bool is cheaper, will continue to use it.\n\nWell, that\u0027s my _assumption_, anyway. I haven\u0027t done any profiling on it, but intuitively it would make sense.","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"6641867462956b530bfcb3af9268e1a8f3f322eb","unresolved":false,"context_lines":[{"line_number":607,"context_line":""},{"line_number":608,"context_line":"        :param record: shard entry record"},{"line_number":609,"context_line":"        :param shard_record_full: boolean, when true the timestamp field is"},{"line_number":610,"context_line":"                                  added as \"last_modified\" in iso format."},{"line_number":611,"context_line":"        :returns: modified record"},{"line_number":612,"context_line":"        \"\"\""},{"line_number":613,"context_line":"        response \u003d dict(record)"}],"source_content_type":"text/x-python","patch_set":38,"id":"5ce69b61_3db6f1d2","line":610,"in_reply_to":"f56c6f40_a4eabff5","updated":"2023-11-14 01:59:11.000000000","message":"yes, it\u0027s equivalent to ``isinstance(record, ShardRange)``. Good to know that carrying and checking the bool is cheaper, will continue to use it.","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"e367c6989582ab6ef04e286014b46f4c1023eb77","unresolved":true,"context_lines":[{"line_number":608,"context_line":"        :param record: shard entry record"},{"line_number":609,"context_line":"        :param shard_record_full: boolean, when true the timestamp field is"},{"line_number":610,"context_line":"                                  added as \"last_modified\" in iso format."},{"line_number":611,"context_line":"        :returns: modified record"},{"line_number":612,"context_line":"        \"\"\""},{"line_number":613,"context_line":"        response \u003d dict(record)"},{"line_number":614,"context_line":"        if shard_record_full:"}],"source_content_type":"text/x-python","patch_set":38,"id":"b10919c6_ed2bcb97","line":611,"range":{"start_line":611,"start_character":18,"end_line":611,"end_character":33},"updated":"2023-11-13 23:38:06.000000000","message":"nit: I\u0027d be partial towards something like \"dict suitable for listing responses\"","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"6641867462956b530bfcb3af9268e1a8f3f322eb","unresolved":false,"context_lines":[{"line_number":608,"context_line":"        :param record: shard entry record"},{"line_number":609,"context_line":"        :param shard_record_full: boolean, when true the timestamp field is"},{"line_number":610,"context_line":"                                  added as \"last_modified\" in iso format."},{"line_number":611,"context_line":"        :returns: modified record"},{"line_number":612,"context_line":"        \"\"\""},{"line_number":613,"context_line":"        response \u003d dict(record)"},{"line_number":614,"context_line":"        if shard_record_full:"}],"source_content_type":"text/x-python","patch_set":38,"id":"3460f799_df11d731","line":611,"range":{"start_line":611,"start_character":18,"end_line":611,"end_character":33},"in_reply_to":"b10919c6_ed2bcb97","updated":"2023-11-14 01:59:11.000000000","message":"Ack","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"18e816986da0f7d7dbb1575c0927744a62acb048","unresolved":true,"context_lines":[{"line_number":716,"context_line":"          container. This alias will not cause a \u0027filler\u0027 range to be added,"},{"line_number":717,"context_line":"          but will cause the container\u0027s own shard range to be included in the"},{"line_number":718,"context_line":"          listing. For now, ``auditing`` is only supported when"},{"line_number":719,"context_line":"          \u0027X-Backend-Record-Shard-Format\u0027 is \u0027full\u0027."},{"line_number":720,"context_line":""},{"line_number":721,"context_line":"        * Shard range listings can be simplified to include only Namespace"},{"line_number":722,"context_line":"          only attributes (name, lower and upper) if the caller send the header"}],"source_content_type":"text/x-python","patch_set":38,"id":"b23cb41f_c91f07a3","line":719,"updated":"2023-11-10 19:48:08.000000000","message":"I think it\u0027s quite nice to list auditing in the GET dosctring on the same level of billing as the listing and updating states.\n\nI *hope* the container-sharder is already explicitly `record-type: shard \u0026states\u003dauditing`; which should default to returning shard-fomat: full\n\nSince I expect the sharder will continue to want `shard-format: full` - I\u0027m not that worried we\u0027d accidently ask for `shard-format: namespace \u0026states\u003dauditing` but if someone thinks such a use-case is useful eventually it would be nice if either:\n\na) they got an error\nb) they got the same shard-ranges as shard-format: full\n\nIt sounds like this doc-string says you get namespaces, but it\u0027s not the expected list of namespaces - and that\u0027s not a great behavior to codify even as a caveat.  Is there a good reason not to do either A or B now that we\u0027ve identified the gap?","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"6641867462956b530bfcb3af9268e1a8f3f322eb","unresolved":false,"context_lines":[{"line_number":716,"context_line":"          container. This alias will not cause a \u0027filler\u0027 range to be added,"},{"line_number":717,"context_line":"          but will cause the container\u0027s own shard range to be included in the"},{"line_number":718,"context_line":"          listing. For now, ``auditing`` is only supported when"},{"line_number":719,"context_line":"          \u0027X-Backend-Record-Shard-Format\u0027 is \u0027full\u0027."},{"line_number":720,"context_line":""},{"line_number":721,"context_line":"        * Shard range listings can be simplified to include only Namespace"},{"line_number":722,"context_line":"          only attributes (name, lower and upper) if the caller send the header"}],"source_content_type":"text/x-python","patch_set":38,"id":"d4cfe56f_edc94d2d","line":719,"in_reply_to":"b23cb41f_c91f07a3","updated":"2023-11-14 01:59:11.000000000","message":"I have added changes to return 400 on namespace GET requests with \u0027auditing\u0027 state, will push up the new patch after current release is done.","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"e367c6989582ab6ef04e286014b46f4c1023eb77","unresolved":false,"context_lines":[{"line_number":779,"context_line":"        states \u003d params.get(\u0027states\u0027)"},{"line_number":780,"context_line":"        includes \u003d params.get(\u0027includes\u0027)"},{"line_number":781,"context_line":"        shard_format \u003d req.headers.get("},{"line_number":782,"context_line":"            \u0027x-backend-record-shard-format\u0027, \u0027full\u0027).lower()"},{"line_number":783,"context_line":"        include_deleted \u003d config_true_value("},{"line_number":784,"context_line":"            req.headers.get(\u0027x-backend-include-deleted\u0027, False))"},{"line_number":785,"context_line":"        if include_deleted and shard_format \u003d\u003d \u0027namespace\u0027:"}],"source_content_type":"text/x-python","patch_set":38,"id":"5eed1509_43f91e7d","line":782,"range":{"start_line":782,"start_character":46,"end_line":782,"end_character":50},"updated":"2023-11-13 23:38:06.000000000","message":"Right; gotta default to the expensive one for rolling upgrades and have clients opt-in to the better/faster format.","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"18e816986da0f7d7dbb1575c0927744a62acb048","unresolved":true,"context_lines":[{"line_number":785,"context_line":"        if include_deleted and shard_format \u003d\u003d \u0027namespace\u0027:"},{"line_number":786,"context_line":"            # Namespace doesn\u0027t support \u0027deleted\u0027 attribute."},{"line_number":787,"context_line":"            return HTTPBadRequest(request\u003dreq,"},{"line_number":788,"context_line":"                                  body\u003d\u0027No include_deleted for namespace GET\u0027)"},{"line_number":789,"context_line":""},{"line_number":790,"context_line":"        # For record type of \u0027shard\u0027, user can specify an additional header"},{"line_number":791,"context_line":"        # to ask for list of Namespaces instead of full ShardRanges."}],"source_content_type":"text/x-python","patch_set":38,"id":"cab0ed25_5398ec26","line":788,"updated":"2023-11-10 19:48:08.000000000","message":"NICE!","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"3f4812df4982d0832906bdbd476dc32a24c0a01f","unresolved":true,"context_lines":[{"line_number":785,"context_line":"        if include_deleted and shard_format \u003d\u003d \u0027namespace\u0027:"},{"line_number":786,"context_line":"            # Namespace doesn\u0027t support \u0027deleted\u0027 attribute."},{"line_number":787,"context_line":"            return HTTPBadRequest(request\u003dreq,"},{"line_number":788,"context_line":"                                  body\u003d\u0027No include_deleted for namespace GET\u0027)"},{"line_number":789,"context_line":""},{"line_number":790,"context_line":"        # For record type of \u0027shard\u0027, user can specify an additional header"},{"line_number":791,"context_line":"        # to ask for list of Namespaces instead of full ShardRanges."}],"source_content_type":"text/x-python","patch_set":38,"id":"f2c98d25_81a3ee2a","line":788,"in_reply_to":"5a34e447_9373ca9f","updated":"2023-11-14 18:57:34.000000000","message":"OK, I can buy that argument. I\u0027ll keep an eye out for the rolling-upgrade case when I get further out the chain.","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"6641867462956b530bfcb3af9268e1a8f3f322eb","unresolved":true,"context_lines":[{"line_number":785,"context_line":"        if include_deleted and shard_format \u003d\u003d \u0027namespace\u0027:"},{"line_number":786,"context_line":"            # Namespace doesn\u0027t support \u0027deleted\u0027 attribute."},{"line_number":787,"context_line":"            return HTTPBadRequest(request\u003dreq,"},{"line_number":788,"context_line":"                                  body\u003d\u0027No include_deleted for namespace GET\u0027)"},{"line_number":789,"context_line":""},{"line_number":790,"context_line":"        # For record type of \u0027shard\u0027, user can specify an additional header"},{"line_number":791,"context_line":"        # to ask for list of Namespaces instead of full ShardRanges."}],"source_content_type":"text/x-python","patch_set":38,"id":"5a34e447_9373ca9f","line":788,"in_reply_to":"7ce28f50_b822742b","updated":"2023-11-14 01:59:11.000000000","message":"In the following patch https://review.opendev.org/c/openstack/swift/+/900350/, container server will support mark/end_marker/include/reserve for Namespace as well. Since we have no plan to support ``include_deleted`` or ``auditing in states`` for container Namespace GET in the end, I feel it makes sense to have these two cases to return 400.\n\nNote: just added 400 return for the 2nd case, will upload it after current release is done.","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"e367c6989582ab6ef04e286014b46f4c1023eb77","unresolved":true,"context_lines":[{"line_number":785,"context_line":"        if include_deleted and shard_format \u003d\u003d \u0027namespace\u0027:"},{"line_number":786,"context_line":"            # Namespace doesn\u0027t support \u0027deleted\u0027 attribute."},{"line_number":787,"context_line":"            return HTTPBadRequest(request\u003dreq,"},{"line_number":788,"context_line":"                                  body\u003d\u0027No include_deleted for namespace GET\u0027)"},{"line_number":789,"context_line":""},{"line_number":790,"context_line":"        # For record type of \u0027shard\u0027, user can specify an additional header"},{"line_number":791,"context_line":"        # to ask for list of Namespaces instead of full ShardRanges."}],"source_content_type":"text/x-python","patch_set":38,"id":"7ce28f50_b822742b","line":788,"in_reply_to":"cab0ed25_5398ec26","updated":"2023-11-13 23:38:06.000000000","message":"But why do we 400 for `include_deleted`, then quietly switch to returning `ShardRange`s for `marker`, `end_marker`, etc.?","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"8d8195542db644352565bd617030845015b81ec1","unresolved":false,"context_lines":[{"line_number":785,"context_line":"        if include_deleted and shard_format \u003d\u003d \u0027namespace\u0027:"},{"line_number":786,"context_line":"            # Namespace doesn\u0027t support \u0027deleted\u0027 attribute."},{"line_number":787,"context_line":"            return HTTPBadRequest(request\u003dreq,"},{"line_number":788,"context_line":"                                  body\u003d\u0027No include_deleted for namespace GET\u0027)"},{"line_number":789,"context_line":""},{"line_number":790,"context_line":"        # For record type of \u0027shard\u0027, user can specify an additional header"},{"line_number":791,"context_line":"        # to ask for list of Namespaces instead of full ShardRanges."}],"source_content_type":"text/x-python","patch_set":38,"id":"e7ff76b1_6824e1e8","line":788,"in_reply_to":"f2c98d25_81a3ee2a","updated":"2023-11-14 21:25:35.000000000","message":"I am in the process of modifying those Namespace test cases for marker/end_marker/include, since they are being supported in the following patch. Probably will squash the following patch into this current patch, in order for them to land upstream.","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"3add8e7afddeff703c713a8d45f5e76623f5bef4","unresolved":true,"context_lines":[{"line_number":851,"context_line":"            except ValueError:"},{"line_number":852,"context_line":"                return HTTPBadRequest(request\u003dreq, body\u003d\u0027Bad state\u0027)"},{"line_number":853,"context_line":""},{"line_number":854,"context_line":"        if shard_format \u003d\u003d \u0027namespace\u0027:"},{"line_number":855,"context_line":"            # Namespace GET supports a subset of functions of Shard Range GET,"},{"line_number":856,"context_line":"            # for example, it doesn\u0027t support \u0027x-backend-include-deleted\u0027"},{"line_number":857,"context_line":"            # header, nor \u0027auditing\u0027 state query parameter."}],"source_content_type":"text/x-python","patch_set":43,"id":"6d9da5b3_7f8a9220","line":854,"updated":"2023-11-22 23:08:43.000000000","message":"I\u0027m starting to think that the @timing decorator on the method isn\u0027t the best way to get the telemetry we want.  We\u0027re going to end up refactoring code just to measure the timing difference from proxy getting namespaces and shard ranges\n\n... I guess once the proxy starts asking for record-format namespace we\u0027ll just see GET_shards timing go down.  And since the auditor uses the replication port on a different server we probably be pretty confident our GET_shards timing on the cluster facing network is *just* namespaces.","commit_id":"0799b998006292a8f2abc970cd335d462892d9df"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"74eaf120af694826c670d061515def674745fb63","unresolved":true,"context_lines":[{"line_number":800,"context_line":"        # to talk to older version of container servers who don\u0027t support"},{"line_number":801,"context_line":"        # Namespace yet during upgrade."},{"line_number":802,"context_line":"        shard_format \u003d req.headers.get("},{"line_number":803,"context_line":"            \u0027x-backend-record-shard-format\u0027, \u0027full\u0027).lower()"},{"line_number":804,"context_line":"        override_deleted \u003d info and config_true_value("},{"line_number":805,"context_line":"            req.headers.get(\u0027x-backend-override-deleted\u0027, False))"},{"line_number":806,"context_line":""}],"source_content_type":"text/x-python","patch_set":52,"id":"39c7254c_1943f095","line":803,"updated":"2024-01-05 15:04:39.000000000","message":"I think we should revert to explicitly defaulting the string to \u0027full\u0027 if not \u0027namespace\u0027, because we return whatever string we\u0027re sent in the response header.\n\nSo if request has \u0027x-backend-record-shard-format\u0027 \u003d \u0027nonsense\u0027 we should return \u0027x-backend-record-shard-format\u0027 \u003d \u0027full\u0027\n\nAlso, more of a nit, these new lines could be moved to line 812, or better, the override_deleted handling could move to the start of method, since it\u0027s now more like an early-return guard.\n\nSee https://review.opendev.org/c/openstack/swift/+/904840","commit_id":"b272276accd44edfa6aa557c0bd19e4e2e999dd4"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"dced6e75bce319a699b4e3a3353f1af5d40459bf","unresolved":false,"context_lines":[{"line_number":800,"context_line":"        # to talk to older version of container servers who don\u0027t support"},{"line_number":801,"context_line":"        # Namespace yet during upgrade."},{"line_number":802,"context_line":"        shard_format \u003d req.headers.get("},{"line_number":803,"context_line":"            \u0027x-backend-record-shard-format\u0027, \u0027full\u0027).lower()"},{"line_number":804,"context_line":"        override_deleted \u003d info and config_true_value("},{"line_number":805,"context_line":"            req.headers.get(\u0027x-backend-override-deleted\u0027, False))"},{"line_number":806,"context_line":""}],"source_content_type":"text/x-python","patch_set":52,"id":"7c6882c0_951ec020","line":803,"in_reply_to":"39c7254c_1943f095","updated":"2024-01-06 01:13:39.000000000","message":"Thanks, Al! I got it squashed in.","commit_id":"b272276accd44edfa6aa557c0bd19e4e2e999dd4"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"74e9ae2e078e75459ac35188586733c97d906994","unresolved":true,"context_lines":[{"line_number":626,"context_line":"        if shard_record_full:"},{"line_number":627,"context_line":"            created \u003d record.timestamp"},{"line_number":628,"context_line":"            response[\u0027last_modified\u0027] \u003d Timestamp(created).isoformat"},{"line_number":629,"context_line":"        return response"},{"line_number":630,"context_line":""},{"line_number":631,"context_line":"    def update_object_record(self, record):"},{"line_number":632,"context_line":"        \"\"\""}],"source_content_type":"text/x-python","patch_set":57,"id":"f0ca2d7e_eb682694","line":629,"updated":"2024-01-10 14:27:08.000000000","message":"nit: I\u0027m not sure this method is necessary: for namespaces it is trivial cast to dict, for shard ranges it\u0027s still simple enough to inline.\n\n```\ndiff --git a/swift/container/server.py b/swift/container/server.py\nindex a27df6e3a..25ea4d85f 100644\n--- a/swift/container/server.py\n+++ b/swift/container/server.py\n@@ -845,18 +845,18 @@ class ContainerController(BaseStorageServer):\n             if include_own:\n                 return HTTPBadRequest(\n                     request\u003dreq, body\u003d\u0027No auditing state for namespace GET\u0027)\n-            shard_format_full \u003d False\n-            container_list \u003d broker.get_namespaces(\n+            namespaces \u003d broker.get_namespaces(\n                 marker, end_marker, includes, reverse, states, fill_gaps)\n+            listing \u003d [dict(ns) for ns in namespaces]\n         else:\n             resp_headers[\u0027X-Backend-Record-Shard-Format\u0027] \u003d \u0027full\u0027\n-            shard_format_full \u003d True\n-            container_list \u003d broker.get_shard_ranges(\n+            shard_ranges \u003d broker.get_shard_ranges(\n                 marker, end_marker, includes, reverse, states\u003dstates,\n                 include_deleted\u003dinclude_deleted, fill_gaps\u003dfill_gaps,\n                 include_own\u003dinclude_own)\n-        listing \u003d [self.update_shard_record(record, shard_format_full)\n-                   for record in container_list]\n+            listing \u003d [\n+                dict(sr, last_modified\u003dTimestamp(sr.timestamp).isoformat)\n+                for sr in shard_ranges]\n         return self._create_GET_response(req, out_content_type, info,\n                                          resp_headers, broker.metadata,\n                                          container, listing)```","commit_id":"7be1763455236ebf184d6be483798bad5e2acc9d"}],"test/unit/common/test_utils.py":[{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"bf33de1f5f106065568e0985e20cc76fff8f97e0","unresolved":true,"context_lines":[{"line_number":2386,"context_line":"        self.assertEqual({\u0027x\u0027, 1},"},{"line_number":2387,"context_line":"                         utils.transform_to_set([\u0027x\u0027, \u0027x\u0027, 1]))"},{"line_number":2388,"context_line":"        self.assertEqual({\u0027x\u0027, \u0027y\u0027},"},{"line_number":2389,"context_line":"                         utils.transform_to_set((\u0027x\u0027, \u0027y\u0027)))"},{"line_number":2390,"context_line":""},{"line_number":2391,"context_line":"    def test_config_true_value(self):"},{"line_number":2392,"context_line":"        orig_trues \u003d utils.TRUE_VALUES"}],"source_content_type":"text/x-python","patch_set":12,"id":"e3c0462a_2ee1c9ff","line":2389,"updated":"2023-08-31 06:46:04.000000000","message":"We probably should also add a None check. The function has a if None branch so we should be covering that too:\n\n    self.assertIsNone(utils.transform_to_set(None))\n\nAnd empty:\n\n    self.assertEqual({}, utils.transform_to_set([]))\n  \nOh and maybe a list and tuple that have repeating values, just to cover every edge case:\n\n    self.assertEqual({1, 2}, utils.transform_to_set([1, 2, 1, 2]))","commit_id":"4638670d9e4dc6850b5447e2d5756be79b4c1aff"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"fd158f484a16e6ddbe812130fa865b4ebe5e7a8b","unresolved":false,"context_lines":[{"line_number":2386,"context_line":"        self.assertEqual({\u0027x\u0027, 1},"},{"line_number":2387,"context_line":"                         utils.transform_to_set([\u0027x\u0027, \u0027x\u0027, 1]))"},{"line_number":2388,"context_line":"        self.assertEqual({\u0027x\u0027, \u0027y\u0027},"},{"line_number":2389,"context_line":"                         utils.transform_to_set((\u0027x\u0027, \u0027y\u0027)))"},{"line_number":2390,"context_line":""},{"line_number":2391,"context_line":"    def test_config_true_value(self):"},{"line_number":2392,"context_line":"        orig_trues \u003d utils.TRUE_VALUES"}],"source_content_type":"text/x-python","patch_set":12,"id":"91fabc8c_1ca7ac53","line":2389,"in_reply_to":"e3c0462a_2ee1c9ff","updated":"2023-08-31 21:06:25.000000000","message":"Done","commit_id":"4638670d9e4dc6850b5447e2d5756be79b4c1aff"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":true,"context_lines":[{"line_number":2374,"context_line":"            self.assertEqual(v, v.lower())"},{"line_number":2375,"context_line":""},{"line_number":2376,"context_line":"    def test_transform_to_set(self):"},{"line_number":2377,"context_line":"        self.assertIsNone(utils.transform_to_set(None))"},{"line_number":2378,"context_line":"        self.assertEqual(set(), utils.transform_to_set([]))"},{"line_number":2379,"context_line":""},{"line_number":2380,"context_line":"        self.assertEqual({1}, utils.transform_to_set(1))"}],"source_content_type":"text/x-python","patch_set":27,"id":"5e2637ae_2477d76c","line":2377,"updated":"2023-10-02 22:23:29.000000000","message":"this is the only one I don\u0027t like","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"bcb52432e997cd207866f8b22a7c55a1f0a34204","unresolved":false,"context_lines":[{"line_number":2374,"context_line":"            self.assertEqual(v, v.lower())"},{"line_number":2375,"context_line":""},{"line_number":2376,"context_line":"    def test_transform_to_set(self):"},{"line_number":2377,"context_line":"        self.assertIsNone(utils.transform_to_set(None))"},{"line_number":2378,"context_line":"        self.assertEqual(set(), utils.transform_to_set([]))"},{"line_number":2379,"context_line":""},{"line_number":2380,"context_line":"        self.assertEqual({1}, utils.transform_to_set(1))"}],"source_content_type":"text/x-python","patch_set":27,"id":"7ba2c918_20d5a5a5","line":2377,"in_reply_to":"5e2637ae_2477d76c","updated":"2023-10-31 16:11:06.000000000","message":"Done","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":true,"context_lines":[{"line_number":2385,"context_line":"        self.assertEqual({1, 2}, utils.transform_to_set({2, 1}))"},{"line_number":2386,"context_line":"        self.assertEqual({1, 2}, utils.transform_to_set([1, 2, 1, 2]))"},{"line_number":2387,"context_line":""},{"line_number":2388,"context_line":"        self.assertEqual({\u0027x\u0027}, utils.transform_to_set(\u0027x\u0027))"},{"line_number":2389,"context_line":"        self.assertEqual({\u0027x\u0027}, utils.transform_to_set([\u0027x\u0027]))"},{"line_number":2390,"context_line":"        self.assertEqual({\u0027x\u0027, 1},"},{"line_number":2391,"context_line":"                         utils.transform_to_set([\u0027x\u0027, \u0027x\u0027, 1]))"}],"source_content_type":"text/x-python","patch_set":27,"id":"f237df7e_91be4525","line":2388,"updated":"2023-10-02 22:23:29.000000000","message":"but that\u0027s the same as default set constructor would do - try `transform_to_set(\u0027xyz\u0027)`","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"bcb52432e997cd207866f8b22a7c55a1f0a34204","unresolved":false,"context_lines":[{"line_number":2385,"context_line":"        self.assertEqual({1, 2}, utils.transform_to_set({2, 1}))"},{"line_number":2386,"context_line":"        self.assertEqual({1, 2}, utils.transform_to_set([1, 2, 1, 2]))"},{"line_number":2387,"context_line":""},{"line_number":2388,"context_line":"        self.assertEqual({\u0027x\u0027}, utils.transform_to_set(\u0027x\u0027))"},{"line_number":2389,"context_line":"        self.assertEqual({\u0027x\u0027}, utils.transform_to_set([\u0027x\u0027]))"},{"line_number":2390,"context_line":"        self.assertEqual({\u0027x\u0027, 1},"},{"line_number":2391,"context_line":"                         utils.transform_to_set([\u0027x\u0027, \u0027x\u0027, 1]))"}],"source_content_type":"text/x-python","patch_set":27,"id":"4d1d1566_80213e99","line":2388,"in_reply_to":"f237df7e_91be4525","updated":"2023-10-31 16:11:06.000000000","message":"Done","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8a24e200d9cb30696c694717a3c20781843d2652","unresolved":true,"context_lines":[{"line_number":2375,"context_line":""},{"line_number":2376,"context_line":"    def test_transform_to_set(self):"},{"line_number":2377,"context_line":"        self.assertEqual(set(), utils.transform_to_set(None))"},{"line_number":2378,"context_line":"        self.assertEqual(set(), utils.transform_to_set([]))"},{"line_number":2379,"context_line":""},{"line_number":2380,"context_line":"        self.assertEqual({1}, utils.transform_to_set(1))"},{"line_number":2381,"context_line":"        self.assertEqual({1}, utils.transform_to_set([1]))"}],"source_content_type":"text/x-python","patch_set":29,"id":"27254a2d_7470cad5","line":2378,"updated":"2023-10-31 19:21:20.000000000","message":"these are good conditions to cover","commit_id":"95dcd77bf6b77f83ab3471d40b0f658ee5c5014f"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"b263eb52a246427f54c4caf4fda57755aa96fa73","unresolved":false,"context_lines":[{"line_number":2375,"context_line":""},{"line_number":2376,"context_line":"    def test_transform_to_set(self):"},{"line_number":2377,"context_line":"        self.assertEqual(set(), utils.transform_to_set(None))"},{"line_number":2378,"context_line":"        self.assertEqual(set(), utils.transform_to_set([]))"},{"line_number":2379,"context_line":""},{"line_number":2380,"context_line":"        self.assertEqual({1}, utils.transform_to_set(1))"},{"line_number":2381,"context_line":"        self.assertEqual({1}, utils.transform_to_set([1]))"}],"source_content_type":"text/x-python","patch_set":29,"id":"2e5b7ffd_173e4471","line":2378,"in_reply_to":"27254a2d_7470cad5","updated":"2023-11-02 00:49:07.000000000","message":"Ack","commit_id":"95dcd77bf6b77f83ab3471d40b0f658ee5c5014f"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8a24e200d9cb30696c694717a3c20781843d2652","unresolved":true,"context_lines":[{"line_number":2391,"context_line":"                         utils.transform_to_set([\u0027x\u0027, \u0027x\u0027, 1]))"},{"line_number":2392,"context_line":"        self.assertEqual({\u0027x\u0027, \u0027y\u0027},"},{"line_number":2393,"context_line":"                         utils.transform_to_set((\u0027x\u0027, \u0027y\u0027)))"},{"line_number":2394,"context_line":"        self.assertEqual({\u0027xyz\u0027}, utils.transform_to_set(\u0027xyz\u0027))"},{"line_number":2395,"context_line":""},{"line_number":2396,"context_line":"    def test_config_true_value(self):"},{"line_number":2397,"context_line":"        orig_trues \u003d utils.TRUE_VALUES"}],"source_content_type":"text/x-python","patch_set":29,"id":"26b722e8_90c7e724","line":2394,"updated":"2023-10-31 19:21:20.000000000","message":"the behavior I think we\u0027d most likely break is:\n\n    self.assertEqual({\u0027\u0027}, utils.transform_to_set(\u0027\u0027))\n    self.assertEqual({0}, utils.transform_to_set(0))","commit_id":"95dcd77bf6b77f83ab3471d40b0f658ee5c5014f"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"b263eb52a246427f54c4caf4fda57755aa96fa73","unresolved":false,"context_lines":[{"line_number":2391,"context_line":"                         utils.transform_to_set([\u0027x\u0027, \u0027x\u0027, 1]))"},{"line_number":2392,"context_line":"        self.assertEqual({\u0027x\u0027, \u0027y\u0027},"},{"line_number":2393,"context_line":"                         utils.transform_to_set((\u0027x\u0027, \u0027y\u0027)))"},{"line_number":2394,"context_line":"        self.assertEqual({\u0027xyz\u0027}, utils.transform_to_set(\u0027xyz\u0027))"},{"line_number":2395,"context_line":""},{"line_number":2396,"context_line":"    def test_config_true_value(self):"},{"line_number":2397,"context_line":"        orig_trues \u003d utils.TRUE_VALUES"}],"source_content_type":"text/x-python","patch_set":29,"id":"f068f29e_49975aeb","line":2394,"in_reply_to":"26b722e8_90c7e724","updated":"2023-11-02 00:49:07.000000000","message":"Done","commit_id":"95dcd77bf6b77f83ab3471d40b0f658ee5c5014f"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"e367c6989582ab6ef04e286014b46f4c1023eb77","unresolved":true,"context_lines":[{"line_number":8482,"context_line":""},{"line_number":8483,"context_line":"    def test_sort_key_order(self):"},{"line_number":8484,"context_line":"        self.assertEqual(utils.ShardRange.sort_key_order("},{"line_number":8485,"context_line":"            \"test\", 10, 20, \"active\"), (20, \"active\", 10, \"test\"))"},{"line_number":8486,"context_line":""},{"line_number":8487,"context_line":"    def test_sort_key(self):"},{"line_number":8488,"context_line":"        orig_shard_ranges \u003d ["}],"source_content_type":"text/x-python","patch_set":38,"id":"e046384b_1f496f7b","line":8485,"updated":"2023-11-13 23:38:06.000000000","message":"I don\u0027t love how disconnected this is from the types we actually want to be passing in. Something like\n```\n        self.assertEqual(\n            utils.ShardRange.sort_key_order(\"a/c\", \u0027lower\u0027, \u0027upper\u0027,\n                                            utils.ShardRange.ACTIVE),\n            (\u0027upper\u0027, utils.ShardRange.ACTIVE, \u0027lower\u0027, \"a/c\"))\n```\nseems like an improvement. Even then, though, that\u0027s not how the closure inside `backend.py` actually *uses* the thing; better still would be\n```\n        self.assertEqual(\n            utils.ShardRange.sort_key_order(\n                name\u003d\"a/c\",\n                lower\u003d\u0027lower\u0027,\n                upper\u003d\u0027upper\u0027,\n                state\u003dutils.ShardRange.ACTIVE),\n            (\u0027upper\u0027, utils.ShardRange.ACTIVE, \u0027lower\u0027, \"a/c\"))\n```","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"6641867462956b530bfcb3af9268e1a8f3f322eb","unresolved":false,"context_lines":[{"line_number":8482,"context_line":""},{"line_number":8483,"context_line":"    def test_sort_key_order(self):"},{"line_number":8484,"context_line":"        self.assertEqual(utils.ShardRange.sort_key_order("},{"line_number":8485,"context_line":"            \"test\", 10, 20, \"active\"), (20, \"active\", 10, \"test\"))"},{"line_number":8486,"context_line":""},{"line_number":8487,"context_line":"    def test_sort_key(self):"},{"line_number":8488,"context_line":"        orig_shard_ranges \u003d ["}],"source_content_type":"text/x-python","patch_set":38,"id":"53163b47_b88a1885","line":8485,"in_reply_to":"e046384b_1f496f7b","updated":"2023-11-14 01:59:11.000000000","message":"Ack","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"}],"test/unit/container/test_backend.py":[{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"4e664f0e0766cb7f496d3aadc2f927ce29554976","unresolved":true,"context_lines":[{"line_number":4675,"context_line":"        self.assertEqual(shard_ranges[0], ShardRange(*actual[0]))"},{"line_number":4676,"context_line":""},{"line_number":4677,"context_line":"    @with_tempdir"},{"line_number":4678,"context_line":"    def test_get_all_namespaces(self, tempdir):"},{"line_number":4679,"context_line":"        db_path \u003d os.path.join(tempdir, \u0027container.db\u0027)"},{"line_number":4680,"context_line":"        broker \u003d ContainerBroker(db_path, account\u003d\u0027a\u0027, container\u003d\u0027c\u0027)"},{"line_number":4681,"context_line":"        broker.initialize(next(self.ts).internal, 0)"}],"source_content_type":"text/x-python","patch_set":8,"id":"56d38f0c_1a4a613f","line":4678,"updated":"2023-08-23 05:23:12.000000000","message":"need more test cases.","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"a00c00e13e7707065cd8681010ccf6fb7dae2274","unresolved":false,"context_lines":[{"line_number":4675,"context_line":"        self.assertEqual(shard_ranges[0], ShardRange(*actual[0]))"},{"line_number":4676,"context_line":""},{"line_number":4677,"context_line":"    @with_tempdir"},{"line_number":4678,"context_line":"    def test_get_all_namespaces(self, tempdir):"},{"line_number":4679,"context_line":"        db_path \u003d os.path.join(tempdir, \u0027container.db\u0027)"},{"line_number":4680,"context_line":"        broker \u003d ContainerBroker(db_path, account\u003d\u0027a\u0027, container\u003d\u0027c\u0027)"},{"line_number":4681,"context_line":"        broker.initialize(next(self.ts).internal, 0)"}],"source_content_type":"text/x-python","patch_set":8,"id":"fdfd73bf_66cfcd0b","line":4678,"in_reply_to":"56d38f0c_1a4a613f","updated":"2023-09-04 00:45:02.000000000","message":"Done","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"d7efa6e02d77ec38b860851868e2da2de3d45bb8","unresolved":true,"context_lines":[{"line_number":4718,"context_line":"                    ShardRange.SHARDING])"},{"line_number":4719,"context_line":"        expected_ns \u003d namespaces[:3] + \\"},{"line_number":4720,"context_line":"            namespaces[4:5] + [Namespace(\u0027a/c\u0027, \u0027h\u0027, \u0027z\u0027)]"},{"line_number":4721,"context_line":"        self.assertEqual(expected_ns, actual_ns)"},{"line_number":4722,"context_line":""},{"line_number":4723,"context_line":"        orig_execute \u003d GreenDBConnection.execute"},{"line_number":4724,"context_line":"        mock_call_args \u003d []"}],"source_content_type":"text/x-python","patch_set":8,"id":"8389fa48_db9c69cd","line":4721,"updated":"2023-08-24 12:22:29.000000000","message":"as more tests are added, it would be good to always cross-check with get_shard_ranges - that will prevent any future regression if, for example, the sort order is changed in one method but not the other","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"dba5cdc4d8314b4dc7c21ec88d42d98beac2cfbf","unresolved":false,"context_lines":[{"line_number":4718,"context_line":"                    ShardRange.SHARDING])"},{"line_number":4719,"context_line":"        expected_ns \u003d namespaces[:3] + \\"},{"line_number":4720,"context_line":"            namespaces[4:5] + [Namespace(\u0027a/c\u0027, \u0027h\u0027, \u0027z\u0027)]"},{"line_number":4721,"context_line":"        self.assertEqual(expected_ns, actual_ns)"},{"line_number":4722,"context_line":""},{"line_number":4723,"context_line":"        orig_execute \u003d GreenDBConnection.execute"},{"line_number":4724,"context_line":"        mock_call_args \u003d []"}],"source_content_type":"text/x-python","patch_set":8,"id":"6aa234f9_12653b60","line":4721,"in_reply_to":"8389fa48_db9c69cd","updated":"2023-08-30 02:56:43.000000000","message":"Ack","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"d7efa6e02d77ec38b860851868e2da2de3d45bb8","unresolved":true,"context_lines":[{"line_number":4734,"context_line":"                        ShardRange.SHARDING])"},{"line_number":4735,"context_line":"        self.assertEqual(expected_ns, actual_ns)"},{"line_number":4736,"context_line":"        self.assertEqual(2, len(mock_call_args))"},{"line_number":4737,"context_line":"        # verify that includes keyword plumbs through to an SQL condition"},{"line_number":4738,"context_line":"        self.assertIn("},{"line_number":4739,"context_line":"            \"WHERE deleted \u003d 0 AND name !\u003d ? AND state in (?,?,?)\","},{"line_number":4740,"context_line":"            mock_call_args[0][1])"}],"source_content_type":"text/x-python","patch_set":8,"id":"8e8f0db4_e525cb70","line":4737,"updated":"2023-08-24 12:22:29.000000000","message":"stale comment","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"dba5cdc4d8314b4dc7c21ec88d42d98beac2cfbf","unresolved":false,"context_lines":[{"line_number":4734,"context_line":"                        ShardRange.SHARDING])"},{"line_number":4735,"context_line":"        self.assertEqual(expected_ns, actual_ns)"},{"line_number":4736,"context_line":"        self.assertEqual(2, len(mock_call_args))"},{"line_number":4737,"context_line":"        # verify that includes keyword plumbs through to an SQL condition"},{"line_number":4738,"context_line":"        self.assertIn("},{"line_number":4739,"context_line":"            \"WHERE deleted \u003d 0 AND name !\u003d ? AND state in (?,?,?)\","},{"line_number":4740,"context_line":"            mock_call_args[0][1])"}],"source_content_type":"text/x-python","patch_set":8,"id":"13b4b547_f7efffb0","line":4737,"in_reply_to":"8e8f0db4_e525cb70","updated":"2023-08-30 02:56:43.000000000","message":"Ack","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"d7efa6e02d77ec38b860851868e2da2de3d45bb8","unresolved":true,"context_lines":[{"line_number":4739,"context_line":"            \"WHERE deleted \u003d 0 AND name !\u003d ? AND state in (?,?,?)\","},{"line_number":4740,"context_line":"            mock_call_args[0][1])"},{"line_number":4741,"context_line":"        self.assertEqual([\u0027a/c\u0027, ShardRange.CREATED, ShardRange.ACTIVE,"},{"line_number":4742,"context_line":"                         ShardRange.SHARDING], mock_call_args[0][2])"},{"line_number":4743,"context_line":""},{"line_number":4744,"context_line":"    @with_tempdir"},{"line_number":4745,"context_line":"    def test_get_own_shard_range(self, tempdir):"}],"source_content_type":"text/x-python","patch_set":8,"id":"05923d69_4a2c993a","line":4742,"updated":"2023-08-24 12:22:29.000000000","message":"I\u0027m not sure this assertion is necessary - vs. changing the request states and verifying that the only the expected namespaces are returned","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"dba5cdc4d8314b4dc7c21ec88d42d98beac2cfbf","unresolved":false,"context_lines":[{"line_number":4739,"context_line":"            \"WHERE deleted \u003d 0 AND name !\u003d ? AND state in (?,?,?)\","},{"line_number":4740,"context_line":"            mock_call_args[0][1])"},{"line_number":4741,"context_line":"        self.assertEqual([\u0027a/c\u0027, ShardRange.CREATED, ShardRange.ACTIVE,"},{"line_number":4742,"context_line":"                         ShardRange.SHARDING], mock_call_args[0][2])"},{"line_number":4743,"context_line":""},{"line_number":4744,"context_line":"    @with_tempdir"},{"line_number":4745,"context_line":"    def test_get_own_shard_range(self, tempdir):"}],"source_content_type":"text/x-python","patch_set":8,"id":"65ec72c4_5c73c66b","line":4742,"in_reply_to":"05923d69_4a2c993a","updated":"2023-08-30 02:56:43.000000000","message":"Ack","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":true,"context_lines":[{"line_number":4788,"context_line":"            filler \u003d [Namespace(\u0027a/c\u0027, filler_lower, \u0027z\u0027)]"},{"line_number":4789,"context_line":"            self.assertEqual(expected_ns + filler, actual)"},{"line_number":4790,"context_line":""},{"line_number":4791,"context_line":"        do_test(ShardRange.CREATED,"},{"line_number":4792,"context_line":"                [Namespace(\u0027a/c0\u0027, \u0027a\u0027, \u0027c\u0027), Namespace(\u0027a/c1\u0027, \u0027c\u0027, \u0027d\u0027)])"},{"line_number":4793,"context_line":"        do_test([ShardRange.CREATED],"},{"line_number":4794,"context_line":"                [Namespace(\u0027a/c0\u0027, \u0027a\u0027, \u0027c\u0027), Namespace(\u0027a/c1\u0027, \u0027c\u0027, \u0027d\u0027)])"}],"source_content_type":"text/x-python","patch_set":27,"id":"27b8a577_07819fed","line":4791,"updated":"2023-10-02 22:23:29.000000000","message":"grrr.... just pass a list!","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"bcb52432e997cd207866f8b22a7c55a1f0a34204","unresolved":false,"context_lines":[{"line_number":4788,"context_line":"            filler \u003d [Namespace(\u0027a/c\u0027, filler_lower, \u0027z\u0027)]"},{"line_number":4789,"context_line":"            self.assertEqual(expected_ns + filler, actual)"},{"line_number":4790,"context_line":""},{"line_number":4791,"context_line":"        do_test(ShardRange.CREATED,"},{"line_number":4792,"context_line":"                [Namespace(\u0027a/c0\u0027, \u0027a\u0027, \u0027c\u0027), Namespace(\u0027a/c1\u0027, \u0027c\u0027, \u0027d\u0027)])"},{"line_number":4793,"context_line":"        do_test([ShardRange.CREATED],"},{"line_number":4794,"context_line":"                [Namespace(\u0027a/c0\u0027, \u0027a\u0027, \u0027c\u0027), Namespace(\u0027a/c1\u0027, \u0027c\u0027, \u0027d\u0027)])"}],"source_content_type":"text/x-python","patch_set":27,"id":"16a46a0e_84313643","line":4791,"in_reply_to":"27b8a577_07819fed","updated":"2023-10-31 16:11:06.000000000","message":"I want to also test \u0027fill_gaps\u003dTrue\u0027 here, so that\u0027ll be easier to read and check the results if I pass the expected Namespace list.","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8a24e200d9cb30696c694717a3c20781843d2652","unresolved":true,"context_lines":[{"line_number":4788,"context_line":"            filler \u003d [Namespace(\u0027a/c\u0027, filler_lower, \u0027z\u0027)]"},{"line_number":4789,"context_line":"            self.assertEqual(expected_ns + filler, actual)"},{"line_number":4790,"context_line":""},{"line_number":4791,"context_line":"        do_test(ShardRange.CREATED,"},{"line_number":4792,"context_line":"                [Namespace(\u0027a/c0\u0027, \u0027a\u0027, \u0027c\u0027), Namespace(\u0027a/c1\u0027, \u0027c\u0027, \u0027d\u0027)])"},{"line_number":4793,"context_line":"        do_test([ShardRange.CREATED],"},{"line_number":4794,"context_line":"                [Namespace(\u0027a/c0\u0027, \u0027a\u0027, \u0027c\u0027), Namespace(\u0027a/c1\u0027, \u0027c\u0027, \u0027d\u0027)])"}],"source_content_type":"text/x-python","patch_set":29,"id":"a090e832_2b6ce1b5","line":4791,"updated":"2023-10-31 19:21:20.000000000","message":"as best I can tell this is the only place we call `get_namespaces(states\u003d\u003cint\u003e)` instead of a list of states","commit_id":"95dcd77bf6b77f83ab3471d40b0f658ee5c5014f"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"a57812bb8061973f9c43b94ddf758ccac9d7d357","unresolved":false,"context_lines":[{"line_number":4788,"context_line":"            filler \u003d [Namespace(\u0027a/c\u0027, filler_lower, \u0027z\u0027)]"},{"line_number":4789,"context_line":"            self.assertEqual(expected_ns + filler, actual)"},{"line_number":4790,"context_line":""},{"line_number":4791,"context_line":"        do_test(ShardRange.CREATED,"},{"line_number":4792,"context_line":"                [Namespace(\u0027a/c0\u0027, \u0027a\u0027, \u0027c\u0027), Namespace(\u0027a/c1\u0027, \u0027c\u0027, \u0027d\u0027)])"},{"line_number":4793,"context_line":"        do_test([ShardRange.CREATED],"},{"line_number":4794,"context_line":"                [Namespace(\u0027a/c0\u0027, \u0027a\u0027, \u0027c\u0027), Namespace(\u0027a/c1\u0027, \u0027c\u0027, \u0027d\u0027)])"}],"source_content_type":"text/x-python","patch_set":29,"id":"96571dc1_6d822987","line":4791,"in_reply_to":"a090e832_2b6ce1b5","updated":"2023-11-02 20:54:39.000000000","message":"Ack","commit_id":"95dcd77bf6b77f83ab3471d40b0f658ee5c5014f"}],"test/unit/container/test_server.py":[{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"d7efa6e02d77ec38b860851868e2da2de3d45bb8","unresolved":true,"context_lines":[{"line_number":3480,"context_line":"                                \u0027states\u0027: \u0027auditing\u0027})"},{"line_number":3481,"context_line":"        self.assertEqual(expected, json.loads(resp.body))"},{"line_number":3482,"context_line":""},{"line_number":3483,"context_line":"    def test_GET_namespaces(self):"},{"line_number":3484,"context_line":"        # make a shard container"},{"line_number":3485,"context_line":"        ts_iter \u003d make_timestamp_iter()"},{"line_number":3486,"context_line":"        shard_ranges \u003d []"}],"source_content_type":"text/x-python","patch_set":8,"id":"f05a0c0f_1034e213","line":3483,"updated":"2023-08-24 12:22:29.000000000","message":"nit: test_GET_namespace_list?","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"dba5cdc4d8314b4dc7c21ec88d42d98beac2cfbf","unresolved":false,"context_lines":[{"line_number":3480,"context_line":"                                \u0027states\u0027: \u0027auditing\u0027})"},{"line_number":3481,"context_line":"        self.assertEqual(expected, json.loads(resp.body))"},{"line_number":3482,"context_line":""},{"line_number":3483,"context_line":"    def test_GET_namespaces(self):"},{"line_number":3484,"context_line":"        # make a shard container"},{"line_number":3485,"context_line":"        ts_iter \u003d make_timestamp_iter()"},{"line_number":3486,"context_line":"        shard_ranges \u003d []"}],"source_content_type":"text/x-python","patch_set":8,"id":"09ba1ea5_f1daf349","line":3483,"in_reply_to":"f05a0c0f_1034e213","updated":"2023-08-30 02:56:43.000000000","message":"not related anymore.","commit_id":"0689fb11a16d80f908d78ded65350d7a2248bcb0"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":true,"context_lines":[{"line_number":3571,"context_line":"            expected_sr \u003d [{\u0027name\u0027: sr[\u0027name\u0027], \u0027lower\u0027: sr[\u0027lower\u0027],"},{"line_number":3572,"context_line":"                            \u0027upper\u0027: sr[\u0027upper\u0027]}"},{"line_number":3573,"context_line":"                           for sr in json.loads(resp.body)]"},{"line_number":3574,"context_line":"            self.assertEqual(expected_ns, expected_sr)"},{"line_number":3575,"context_line":""},{"line_number":3576,"context_line":"        # root\u0027s shard ranges for listing"},{"line_number":3577,"context_line":"        root_path \u003d container_path \u003d \u0027a/c\u0027"}],"source_content_type":"text/x-python","patch_set":27,"id":"036fa54a_c79c367c","line":3574,"updated":"2023-10-02 22:23:29.000000000","message":"this assert is pretty nice!","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"bcb52432e997cd207866f8b22a7c55a1f0a34204","unresolved":false,"context_lines":[{"line_number":3571,"context_line":"            expected_sr \u003d [{\u0027name\u0027: sr[\u0027name\u0027], \u0027lower\u0027: sr[\u0027lower\u0027],"},{"line_number":3572,"context_line":"                            \u0027upper\u0027: sr[\u0027upper\u0027]}"},{"line_number":3573,"context_line":"                           for sr in json.loads(resp.body)]"},{"line_number":3574,"context_line":"            self.assertEqual(expected_ns, expected_sr)"},{"line_number":3575,"context_line":""},{"line_number":3576,"context_line":"        # root\u0027s shard ranges for listing"},{"line_number":3577,"context_line":"        root_path \u003d container_path \u003d \u0027a/c\u0027"}],"source_content_type":"text/x-python","patch_set":27,"id":"9721d065_d313304f","line":3574,"in_reply_to":"036fa54a_c79c367c","updated":"2023-10-31 16:11:06.000000000","message":"Ack","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":true,"context_lines":[{"line_number":3739,"context_line":"            self.assertEqual(\u0027shard\u0027, resp.headers[\u0027X-Backend-Record-Type\u0027])"},{"line_number":3740,"context_line":"            self.assertIn(\u0027X-Backend-Record-Shard-Format\u0027, resp.headers)"},{"line_number":3741,"context_line":"            self.assertEqual("},{"line_number":3742,"context_line":"                \u0027full\u0027, resp.headers[\u0027X-Backend-Record-Shard-Format\u0027])"},{"line_number":3743,"context_line":""},{"line_number":3744,"context_line":"        check_shard_GET(shard_ranges[:3], \u0027a/c\u0027,"},{"line_number":3745,"context_line":"                        params\u003d\u0027\u0026states\u003dlisting\u0026end_marker\u003dpickle\u0027)"}],"source_content_type":"text/x-python","patch_set":27,"id":"09c9ad45_3fecab4e","line":3742,"updated":"2023-10-02 22:23:29.000000000","message":"so we asked for format\u003dnamespace but got format\u003dfull\n\n... maybe that\u0027s in the spirit of ignoring params we don\u0027t understand.  On upgrade to a swift that supports extra params for namespace we can probably handle full shard range repsonses pretty well.","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"ab133fd538b104017d417ae79e5e983aa083a419","unresolved":false,"context_lines":[{"line_number":3739,"context_line":"            self.assertEqual(\u0027shard\u0027, resp.headers[\u0027X-Backend-Record-Type\u0027])"},{"line_number":3740,"context_line":"            self.assertIn(\u0027X-Backend-Record-Shard-Format\u0027, resp.headers)"},{"line_number":3741,"context_line":"            self.assertEqual("},{"line_number":3742,"context_line":"                \u0027full\u0027, resp.headers[\u0027X-Backend-Record-Shard-Format\u0027])"},{"line_number":3743,"context_line":""},{"line_number":3744,"context_line":"        check_shard_GET(shard_ranges[:3], \u0027a/c\u0027,"},{"line_number":3745,"context_line":"                        params\u003d\u0027\u0026states\u003dlisting\u0026end_marker\u003dpickle\u0027)"}],"source_content_type":"text/x-python","patch_set":27,"id":"f339cd76_c02b8b76","line":3742,"in_reply_to":"09c9ad45_3fecab4e","updated":"2023-10-04 04:32:17.000000000","message":"Ack","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"283d9d3768e33901be4ee23f036d2d13b298b605","unresolved":true,"context_lines":[{"line_number":3747,"context_line":"            reversed(shard_ranges[:3]), \u0027a/c\u0027,"},{"line_number":3748,"context_line":"            params\u003d\u0027\u0026states\u003dlisting\u0026reverse\u003dtrue\u0026marker\u003dpickle\u0027)"},{"line_number":3749,"context_line":"        check_shard_GET(shard_ranges[1:4], \u0027a/c\u0027,"},{"line_number":3750,"context_line":"                        params\u003d\u0027\u0026states\u003dupdating\u0026end_marker\u003dtreacle\u0027)"},{"line_number":3751,"context_line":"        check_shard_GET("},{"line_number":3752,"context_line":"            reversed(shard_ranges[1:4]), \u0027a/c\u0027,"},{"line_number":3753,"context_line":"            params\u003d\u0027\u0026states\u003dupdating\u0026reverse\u003dtrue\u0026marker\u003dtreacle\u0027)"}],"source_content_type":"text/x-python","patch_set":27,"id":"771213bf_90a48e14","line":3750,"updated":"2023-10-02 22:23:29.000000000","message":"i have never had this stuff - \"black molasses\" - is it good?","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"bcb52432e997cd207866f8b22a7c55a1f0a34204","unresolved":false,"context_lines":[{"line_number":3747,"context_line":"            reversed(shard_ranges[:3]), \u0027a/c\u0027,"},{"line_number":3748,"context_line":"            params\u003d\u0027\u0026states\u003dlisting\u0026reverse\u003dtrue\u0026marker\u003dpickle\u0027)"},{"line_number":3749,"context_line":"        check_shard_GET(shard_ranges[1:4], \u0027a/c\u0027,"},{"line_number":3750,"context_line":"                        params\u003d\u0027\u0026states\u003dupdating\u0026end_marker\u003dtreacle\u0027)"},{"line_number":3751,"context_line":"        check_shard_GET("},{"line_number":3752,"context_line":"            reversed(shard_ranges[1:4]), \u0027a/c\u0027,"},{"line_number":3753,"context_line":"            params\u003d\u0027\u0026states\u003dupdating\u0026reverse\u003dtrue\u0026marker\u003dtreacle\u0027)"}],"source_content_type":"text/x-python","patch_set":27,"id":"59b224a4_2f2189df","line":3750,"in_reply_to":"771213bf_90a48e14","updated":"2023-10-31 16:11:06.000000000","message":"Ack","commit_id":"4bfbf4ae0835c433c2f7a24eb493407dd2a51783"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8a24e200d9cb30696c694717a3c20781843d2652","unresolved":true,"context_lines":[{"line_number":3482,"context_line":""},{"line_number":3483,"context_line":"    def test_GET_namespaces(self):"},{"line_number":3484,"context_line":"        # make a shard container"},{"line_number":3485,"context_line":"        ts_iter \u003d make_timestamp_iter()"},{"line_number":3486,"context_line":"        shard_ranges \u003d []"},{"line_number":3487,"context_line":"        lower \u003d \u0027\u0027"},{"line_number":3488,"context_line":"        for state in sorted(ShardRange.STATES.keys()):"}],"source_content_type":"text/x-python","patch_set":29,"id":"6b40b823_6266bc5f","line":3485,"updated":"2023-10-31 19:21:20.000000000","message":"i think it\u0027s a common convention in newer tests to have one of these as self.ts","commit_id":"95dcd77bf6b77f83ab3471d40b0f658ee5c5014f"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bff0a54e68a76090810057a6fac976d6d5e3e745","unresolved":false,"context_lines":[{"line_number":3482,"context_line":""},{"line_number":3483,"context_line":"    def test_GET_namespaces(self):"},{"line_number":3484,"context_line":"        # make a shard container"},{"line_number":3485,"context_line":"        ts_iter \u003d make_timestamp_iter()"},{"line_number":3486,"context_line":"        shard_ranges \u003d []"},{"line_number":3487,"context_line":"        lower \u003d \u0027\u0027"},{"line_number":3488,"context_line":"        for state in sorted(ShardRange.STATES.keys()):"}],"source_content_type":"text/x-python","patch_set":29,"id":"ba13256a_880c2d71","line":3485,"in_reply_to":"6b40b823_6266bc5f","updated":"2024-01-04 16:20:07.000000000","message":"Done","commit_id":"95dcd77bf6b77f83ab3471d40b0f658ee5c5014f"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"8a24e200d9cb30696c694717a3c20781843d2652","unresolved":true,"context_lines":[{"line_number":3586,"context_line":"        params \u003d \u0027\u0026states\u003dlisting\u0027"},{"line_number":3587,"context_line":"        do_test_unsharded(root_path, container_path, params, expected_states)"},{"line_number":3588,"context_line":""},{"line_number":3589,"context_line":"        # root\u0027s shard ranges for updating"},{"line_number":3590,"context_line":"        params \u003d \u0027\u0026states\u003dupdating\u0027"},{"line_number":3591,"context_line":"        expected_states \u003d ["},{"line_number":3592,"context_line":"            ShardRange.CREATED, ShardRange.CLEAVED, ShardRange.ACTIVE,"}],"source_content_type":"text/x-python","patch_set":29,"id":"e1c0e89d_16aa05fc","line":3589,"updated":"2023-10-31 19:21:20.000000000","message":"I find tests that use closures and hit multiple tests scenarios are difficult to maintain and reason about because the state is constantly mutating.\n\nHad you intended for this scenario to cover container_path \u003d\u003d \u0027a/c\u0027 (i.e. testing a root) or container_path \u003d\u003d \u0027.shards_a/c\u0027 (i.e. testing a subshard similar to the case below?).\n\nWhat is actually being tested?","commit_id":"95dcd77bf6b77f83ab3471d40b0f658ee5c5014f"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"a57812bb8061973f9c43b94ddf758ccac9d7d357","unresolved":false,"context_lines":[{"line_number":3586,"context_line":"        params \u003d \u0027\u0026states\u003dlisting\u0027"},{"line_number":3587,"context_line":"        do_test_unsharded(root_path, container_path, params, expected_states)"},{"line_number":3588,"context_line":""},{"line_number":3589,"context_line":"        # root\u0027s shard ranges for updating"},{"line_number":3590,"context_line":"        params \u003d \u0027\u0026states\u003dupdating\u0027"},{"line_number":3591,"context_line":"        expected_states \u003d ["},{"line_number":3592,"context_line":"            ShardRange.CREATED, ShardRange.CLEAVED, ShardRange.ACTIVE,"}],"source_content_type":"text/x-python","patch_set":29,"id":"13ecfe68_cfd12ad6","line":3589,"in_reply_to":"b5144020_87dbbdd0","updated":"2023-11-02 20:54:39.000000000","message":"Ack","commit_id":"95dcd77bf6b77f83ab3471d40b0f658ee5c5014f"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"b263eb52a246427f54c4caf4fda57755aa96fa73","unresolved":true,"context_lines":[{"line_number":3586,"context_line":"        params \u003d \u0027\u0026states\u003dlisting\u0027"},{"line_number":3587,"context_line":"        do_test_unsharded(root_path, container_path, params, expected_states)"},{"line_number":3588,"context_line":""},{"line_number":3589,"context_line":"        # root\u0027s shard ranges for updating"},{"line_number":3590,"context_line":"        params \u003d \u0027\u0026states\u003dupdating\u0027"},{"line_number":3591,"context_line":"        expected_states \u003d ["},{"line_number":3592,"context_line":"            ShardRange.CREATED, ShardRange.CLEAVED, ShardRange.ACTIVE,"}],"source_content_type":"text/x-python","patch_set":29,"id":"b5144020_87dbbdd0","line":3589,"in_reply_to":"e1c0e89d_16aa05fc","updated":"2023-11-02 00:49:07.000000000","message":"probably previous test cases were not easy to read, after the test refactoring, the two cases you mentioned now have their own function names: ``test_unsharded_root_get_namespaces_state_updating`` and ``test_unsharded_subshard_get_namespaces_state_updating``","commit_id":"95dcd77bf6b77f83ab3471d40b0f658ee5c5014f"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"a57812bb8061973f9c43b94ddf758ccac9d7d357","unresolved":false,"context_lines":[{"line_number":3478,"context_line":"                    for sr in shard_ranges[1:2] + [own_sr]]"},{"line_number":3479,"context_line":"        resp \u003d do_test(ts_now, {\u0027marker\u0027: \u0027f\u0027, \u0027end_marker\u0027: \u0027t\u0027,"},{"line_number":3480,"context_line":"                                \u0027states\u0027: \u0027auditing\u0027})"},{"line_number":3481,"context_line":"        self.assertEqual(expected, json.loads(resp.body))"},{"line_number":3482,"context_line":""},{"line_number":3483,"context_line":"    def _do_get_namespaces_unsharded(self, root_path, path,"},{"line_number":3484,"context_line":"                                     params, expected_states):"}],"source_content_type":"text/x-python","patch_set":32,"id":"c1b3fb68_2b197c59","line":3481,"updated":"2023-11-02 20:54:39.000000000","message":"heh, I hadn\u0027t previously noticed the bunched up tests came from an existing prolific convention in this module - that\u0027s too bad, and good on you for doing better!","commit_id":"5380c3ba8763ed30da5f0837c5b2efc45e6a8003"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"a57812bb8061973f9c43b94ddf758ccac9d7d357","unresolved":true,"context_lines":[{"line_number":3622,"context_line":"        for state in sorted(ShardRange.STATES.keys()):"},{"line_number":3623,"context_line":"            upper \u003d str(state)"},{"line_number":3624,"context_line":"            shard_ranges.append("},{"line_number":3625,"context_line":"                ShardRange(\u0027.shards_a/c_%s\u0027 % upper, next(self.ts),"},{"line_number":3626,"context_line":"                           lower, upper, state * 100, state * 1000,"},{"line_number":3627,"context_line":"                           meta_timestamp\u003dnext(self.ts),"},{"line_number":3628,"context_line":"                           state\u003dstate, state_timestamp\u003dnext(self.ts)))"}],"source_content_type":"text/x-python","patch_set":32,"id":"9523562d_38358e82","line":3625,"updated":"2023-11-02 20:54:39.000000000","message":"it would be nice to consistently use self.ts instead of sometimes the ts_iter alias.","commit_id":"5380c3ba8763ed30da5f0837c5b2efc45e6a8003"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"535964202b2c8c0c3553c5a70c9c446fd66091fa","unresolved":false,"context_lines":[{"line_number":3622,"context_line":"        for state in sorted(ShardRange.STATES.keys()):"},{"line_number":3623,"context_line":"            upper \u003d str(state)"},{"line_number":3624,"context_line":"            shard_ranges.append("},{"line_number":3625,"context_line":"                ShardRange(\u0027.shards_a/c_%s\u0027 % upper, next(self.ts),"},{"line_number":3626,"context_line":"                           lower, upper, state * 100, state * 1000,"},{"line_number":3627,"context_line":"                           meta_timestamp\u003dnext(self.ts),"},{"line_number":3628,"context_line":"                           state\u003dstate, state_timestamp\u003dnext(self.ts)))"}],"source_content_type":"text/x-python","patch_set":32,"id":"fee2086b_c22af5f5","line":3625,"in_reply_to":"9523562d_38358e82","updated":"2023-11-03 04:25:44.000000000","message":"Done","commit_id":"5380c3ba8763ed30da5f0837c5b2efc45e6a8003"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"a57812bb8061973f9c43b94ddf758ccac9d7d357","unresolved":true,"context_lines":[{"line_number":3634,"context_line":"                                     meta_timestamp\u003dnext(ts_iter),"},{"line_number":3635,"context_line":"                                     state\u003dShardRange.ACTIVE,"},{"line_number":3636,"context_line":"                                     state_timestamp\u003dnext(ts_iter),"},{"line_number":3637,"context_line":"                                     epoch\u003dnext(ts_iter))"},{"line_number":3638,"context_line":"        filler_sr \u003d own_shard_range.copy(lower\u003dexpected_sr[-1].upper)"},{"line_number":3639,"context_line":"        expected_sr.append(filler_sr)"},{"line_number":3640,"context_line":"        expected_ns \u003d [{\u0027name\u0027: sr.name, \u0027lower\u0027: sr.lower_str,"}],"source_content_type":"text/x-python","patch_set":32,"id":"1683fdae_5251582b","line":3637,"updated":"2023-11-02 20:54:39.000000000","message":"it seems like these arguments when we create this databases\u0027s osr are what make it \"sharded\"","commit_id":"5380c3ba8763ed30da5f0837c5b2efc45e6a8003"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"abdaf1ad1905cacd3520e5dd5c60706bbac6553e","unresolved":false,"context_lines":[{"line_number":3634,"context_line":"                                     meta_timestamp\u003dnext(ts_iter),"},{"line_number":3635,"context_line":"                                     state\u003dShardRange.ACTIVE,"},{"line_number":3636,"context_line":"                                     state_timestamp\u003dnext(ts_iter),"},{"line_number":3637,"context_line":"                                     epoch\u003dnext(ts_iter))"},{"line_number":3638,"context_line":"        filler_sr \u003d own_shard_range.copy(lower\u003dexpected_sr[-1].upper)"},{"line_number":3639,"context_line":"        expected_sr.append(filler_sr)"},{"line_number":3640,"context_line":"        expected_ns \u003d [{\u0027name\u0027: sr.name, \u0027lower\u0027: sr.lower_str,"}],"source_content_type":"text/x-python","patch_set":32,"id":"58542f80_3f3bfcec","line":3637,"in_reply_to":"1683fdae_5251582b","updated":"2023-11-06 05:04:07.000000000","message":"Besides setting epoch, another place is where the following code calls ``set_sharding_state`` and ``set_sharded_state``.\nSee here:\nhttps://review.opendev.org/c/openstack/swift/+/890470/35/test/unit/container/test_server.py#3663","commit_id":"5380c3ba8763ed30da5f0837c5b2efc45e6a8003"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"a57812bb8061973f9c43b94ddf758ccac9d7d357","unresolved":true,"context_lines":[{"line_number":3657,"context_line":"        self.assertEqual(202, req.get_response(self.controller).status_int)"},{"line_number":3658,"context_line":""},{"line_number":3659,"context_line":"        # set broker to sharded state so"},{"line_number":3660,"context_line":"        # X-Backend-Override-Shard-Name-Filter does have effect"},{"line_number":3661,"context_line":"        shard_broker \u003d self.controller._get_container_broker("},{"line_number":3662,"context_line":"            \u0027sda1\u0027, \u0027p\u0027, \u0027.shards_a\u0027, \u0027c\u0027)"},{"line_number":3663,"context_line":"        self.assertTrue(shard_broker.set_sharding_state())"}],"source_content_type":"text/x-python","patch_set":32,"id":"8a87256a_e1483f81","line":3660,"updated":"2023-11-02 20:54:39.000000000","message":"heh, i\u0027ve never heard of this header!  😄\n\noops, apparently I\u0027d just forgot about it:\n\nI5fc696625d69d1ee9218ee2a508a1b9be6cf9685\n\nI think \"x-backend-override-shard\" is a kind of name space, and what we \"override\" is the \"name-filter\" - where a \"filter\" ~\u003d \"query param\" and \"name\" is either shard range name (e.g. marker/end_marker) or a object name (e.g. includes)... but also \"reverse\" because... seriously we just want the complete shard listing.\n\nSo I think for namespaces this should be the expected case - it would be unexpected to get a record-format: namespaces requests that included marker/end_marker/reverse or didn\u0027t include x-backend-override-shard-name-filter - is that right?\n\nN.B. the correct answer is \u0027X-Backend-Override-Shard-Name-Filter: sharded\u0027 \n\n\tvagrant@saio:~$ curl -XGET -s \"http://127.0.0.1:6011/sdb1/54/AUTH_test/lots-of-files?format\u003djson\u0026reverse\u003dtrue\" -H \u0027x-backend-record-type: shard\u0027 -H \u0027X-Backend-Override-Shard-Name-Filter: true\u0027 -I | grep -i override\n\tvagrant@saio:~$ curl -XGET -s \"http://127.0.0.1:6011/sdb1/54/AUTH_test/lots-of-files?format\u003djson\u0026reverse\u003dtrue\" -H \u0027x-backend-record-type: shard\u0027 -H \u0027X-Backend-Override-Shard-Name-Filter: sharded\u0027 -I | grep -i override\n\tX-Backend-Override-Shard-Name-Filter: true\n\nI thought it was a boolean!\n\nhttps://github.com/NVIDIA/swift/blob/master/swift/container/server.py#L746","commit_id":"5380c3ba8763ed30da5f0837c5b2efc45e6a8003"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"18e816986da0f7d7dbb1575c0927744a62acb048","unresolved":false,"context_lines":[{"line_number":3657,"context_line":"        self.assertEqual(202, req.get_response(self.controller).status_int)"},{"line_number":3658,"context_line":""},{"line_number":3659,"context_line":"        # set broker to sharded state so"},{"line_number":3660,"context_line":"        # X-Backend-Override-Shard-Name-Filter does have effect"},{"line_number":3661,"context_line":"        shard_broker \u003d self.controller._get_container_broker("},{"line_number":3662,"context_line":"            \u0027sda1\u0027, \u0027p\u0027, \u0027.shards_a\u0027, \u0027c\u0027)"},{"line_number":3663,"context_line":"        self.assertTrue(shard_broker.set_sharding_state())"}],"source_content_type":"text/x-python","patch_set":32,"id":"851448d6_c1507c9f","line":3660,"in_reply_to":"01936ba6_95144922","updated":"2023-11-10 19:48:08.000000000","message":"\u003e If \u0027X-Backend-Override-Shard-Name-Filter\u0027 is true\n\ni believe is \"sharded\" - the return value is boolen, the input value is sharding state (or db state?)\n\n\u003e Since this namespace patch doesn\u0027t support those parameters yet, so \u0027X-Backend-Override-Shard-Name-Filter\u0027 would have no effect to get_namespaces\n\nin test-server I think a request for \"GET record-format: namespaces\" *would* be effected by \"x-backend-override-shard-name-filter\" if a) the database wasn\u0027t in a matching state or b) the params are not supported by namespace and would be ignored on state sharding\n\n\u003e After future patch adds support of those parameters, this header will impact get_namespace in same way as get_shard_ranges\n\nI expect that the proxy will continue to be the main place where we make a decorated GET request to a root db w/o knowing if we\u0027ll get back objects or name-spaces.  If namespaces continue to ignore name filters that might remove the need for x-backend-override-shard-name-filter: sharded since it seems like if there\u0027s other places we ask for filtered shard-records we could infer from the explicit record-type that the filters are for the the shard-records regardless of the record-format","commit_id":"5380c3ba8763ed30da5f0837c5b2efc45e6a8003"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"abdaf1ad1905cacd3520e5dd5c60706bbac6553e","unresolved":false,"context_lines":[{"line_number":3657,"context_line":"        self.assertEqual(202, req.get_response(self.controller).status_int)"},{"line_number":3658,"context_line":""},{"line_number":3659,"context_line":"        # set broker to sharded state so"},{"line_number":3660,"context_line":"        # X-Backend-Override-Shard-Name-Filter does have effect"},{"line_number":3661,"context_line":"        shard_broker \u003d self.controller._get_container_broker("},{"line_number":3662,"context_line":"            \u0027sda1\u0027, \u0027p\u0027, \u0027.shards_a\u0027, \u0027c\u0027)"},{"line_number":3663,"context_line":"        self.assertTrue(shard_broker.set_sharding_state())"}],"source_content_type":"text/x-python","patch_set":32,"id":"01936ba6_95144922","line":3660,"in_reply_to":"8a87256a_e1483f81","updated":"2023-11-06 05:04:07.000000000","message":"If \u0027X-Backend-Override-Shard-Name-Filter\u0027 is true, container will ignore \u0027mark\u0027/\u0027end_marker\u0027/\u0027includes\u0027/\u0027reverse\u0027 when GET shard ranges or namespaces. Since this namespace patch doesn\u0027t support those parameters yet, so \u0027X-Backend-Override-Shard-Name-Filter\u0027 would have no effect to get_namespaces. After future patch adds support of those parameters, this header will impact get_namespace in same way as get_shard_ranges.","commit_id":"5380c3ba8763ed30da5f0837c5b2efc45e6a8003"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"a57812bb8061973f9c43b94ddf758ccac9d7d357","unresolved":true,"context_lines":[{"line_number":3690,"context_line":"        # shard\u0027s namespaces for listing"},{"line_number":3691,"context_line":"        root_path \u003d \u0027a/c\u0027"},{"line_number":3692,"context_line":"        container_path \u003d \u0027.shards_a/c\u0027"},{"line_number":3693,"context_line":"        params \u003d \u0027\u0026states\u003dlisting\u0027"},{"line_number":3694,"context_line":"        expected_states \u003d ["},{"line_number":3695,"context_line":"            ShardRange.CLEAVED, ShardRange.ACTIVE, ShardRange.SHARDING,"},{"line_number":3696,"context_line":"            ShardRange.SHRINKING]"}],"source_content_type":"text/x-python","patch_set":32,"id":"a74f734c_7424d5b3","line":3693,"updated":"2023-11-02 20:54:39.000000000","message":"ok, and now I have to understand how this test is different:\n\n    def test_unsharded_subshard_get_namespaces_state_updating(self):\n        # shard\u0027s namespaces for updating\n        root_path \u003d \u0027a/c\u0027\n        container_path \u003d \u0027.shards_a/c\u0027\n        params \u003d \u0027\u0026states\u003dupdating\u0027","commit_id":"5380c3ba8763ed30da5f0837c5b2efc45e6a8003"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"18e816986da0f7d7dbb1575c0927744a62acb048","unresolved":false,"context_lines":[{"line_number":3690,"context_line":"        # shard\u0027s namespaces for listing"},{"line_number":3691,"context_line":"        root_path \u003d \u0027a/c\u0027"},{"line_number":3692,"context_line":"        container_path \u003d \u0027.shards_a/c\u0027"},{"line_number":3693,"context_line":"        params \u003d \u0027\u0026states\u003dlisting\u0027"},{"line_number":3694,"context_line":"        expected_states \u003d ["},{"line_number":3695,"context_line":"            ShardRange.CLEAVED, ShardRange.ACTIVE, ShardRange.SHARDING,"},{"line_number":3696,"context_line":"            ShardRange.SHRINKING]"}],"source_content_type":"text/x-python","patch_set":32,"id":"ecd5f3e4_9b5db94c","line":3693,"in_reply_to":"502f4364_5085bb31","updated":"2023-11-10 19:48:08.000000000","message":"yeah that was mis-copy 😊\n\nI think I have the six cases now!\n\n\tdef test_GET_namespaces_unsharded_root_state_listing(self):\n\tdef test_GET_namespaces_unsharded_subshard_state_listing(self):\n\tdef test_GET_namespaces_unsharded_root_state_updating(self):\n\tdef test_GET_namespaces_unsharded_subshard_state_updating(self):\n\tdef test_GET_namespaces_sharded_subshard_state_listing(self):\n\tdef test_GET_namespaces_sharded_subshard_state_updating(self):\n\nthanks for breaking them out!","commit_id":"5380c3ba8763ed30da5f0837c5b2efc45e6a8003"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"abdaf1ad1905cacd3520e5dd5c60706bbac6553e","unresolved":true,"context_lines":[{"line_number":3690,"context_line":"        # shard\u0027s namespaces for listing"},{"line_number":3691,"context_line":"        root_path \u003d \u0027a/c\u0027"},{"line_number":3692,"context_line":"        container_path \u003d \u0027.shards_a/c\u0027"},{"line_number":3693,"context_line":"        params \u003d \u0027\u0026states\u003dlisting\u0027"},{"line_number":3694,"context_line":"        expected_states \u003d ["},{"line_number":3695,"context_line":"            ShardRange.CLEAVED, ShardRange.ACTIVE, ShardRange.SHARDING,"},{"line_number":3696,"context_line":"            ShardRange.SHRINKING]"}],"source_content_type":"text/x-python","patch_set":32,"id":"502f4364_5085bb31","line":3693,"in_reply_to":"a74f734c_7424d5b3","updated":"2023-11-06 05:04:07.000000000","message":"one tests \u0027states\u003dlisting\u0027 and another tests \u0027states\u003dupdating\u0027?","commit_id":"5380c3ba8763ed30da5f0837c5b2efc45e6a8003"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"c911d8629cb67aa7956424dc2df2e5efe5e2c5d4","unresolved":true,"context_lines":[{"line_number":3569,"context_line":"            \u0027full\u0027, resp.headers[\u0027X-Backend-Record-Shard-Format\u0027])"},{"line_number":3570,"context_line":"        self.assertEqual(raw_sr, json.loads(resp.body))"},{"line_number":3571,"context_line":""},{"line_number":3572,"context_line":"    def test_unsharded_root_get_namespaces_state_listing(self):"},{"line_number":3573,"context_line":"        # root\u0027s namespaces for listing"},{"line_number":3574,"context_line":"        root_path \u003d container_path \u003d \u0027a/c\u0027"},{"line_number":3575,"context_line":"        params \u003d \u0027\u0026states\u003dlisting\u0027"}],"source_content_type":"text/x-python","patch_set":34,"id":"b2bbc604_a02ddbd6","line":3572,"updated":"2023-11-03 10:56:20.000000000","message":"could we name all the tests so that the name starts with \"test_GET_namespaces...\" ? - that would be consistent with the test_GET_shard_ranges... but also makes it easier to look for all related tests\n\nAlso, be consistent with upper and lower case GET in test names","commit_id":"d5a82e6874a0c73b9d55c5bb5b1c42fb8a90e230"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"abdaf1ad1905cacd3520e5dd5c60706bbac6553e","unresolved":false,"context_lines":[{"line_number":3569,"context_line":"            \u0027full\u0027, resp.headers[\u0027X-Backend-Record-Shard-Format\u0027])"},{"line_number":3570,"context_line":"        self.assertEqual(raw_sr, json.loads(resp.body))"},{"line_number":3571,"context_line":""},{"line_number":3572,"context_line":"    def test_unsharded_root_get_namespaces_state_listing(self):"},{"line_number":3573,"context_line":"        # root\u0027s namespaces for listing"},{"line_number":3574,"context_line":"        root_path \u003d container_path \u003d \u0027a/c\u0027"},{"line_number":3575,"context_line":"        params \u003d \u0027\u0026states\u003dlisting\u0027"}],"source_content_type":"text/x-python","patch_set":34,"id":"1caf37c7_6cea5045","line":3572,"in_reply_to":"b2bbc604_a02ddbd6","updated":"2023-11-06 05:04:07.000000000","message":"Ack","commit_id":"d5a82e6874a0c73b9d55c5bb5b1c42fb8a90e230"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"18e816986da0f7d7dbb1575c0927744a62acb048","unresolved":true,"context_lines":[{"line_number":3706,"context_line":"        self._do_get_namespaces_sharded("},{"line_number":3707,"context_line":"            root_path, container_path, params, expected_states)"},{"line_number":3708,"context_line":""},{"line_number":3709,"context_line":"    def test_GET_namespaces_not_supported(self):"},{"line_number":3710,"context_line":"        # make a container"},{"line_number":3711,"context_line":"        ts_iter \u003d make_timestamp_iter()"},{"line_number":3712,"context_line":"        ts_now \u003d Timestamp.now()  # used when mocking Timestamp.now()"}],"source_content_type":"text/x-python","patch_set":38,"id":"1712cb45_f3937632","line":3709,"updated":"2023-11-10 19:48:08.000000000","message":"this test covers a lot of ground\n\nagain I think individual tests that name the desired behavior would be easier to read, find and maintain.","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bff0a54e68a76090810057a6fac976d6d5e3e745","unresolved":true,"context_lines":[{"line_number":3706,"context_line":"        self._do_get_namespaces_sharded("},{"line_number":3707,"context_line":"            root_path, container_path, params, expected_states)"},{"line_number":3708,"context_line":""},{"line_number":3709,"context_line":"    def test_GET_namespaces_not_supported(self):"},{"line_number":3710,"context_line":"        # make a container"},{"line_number":3711,"context_line":"        ts_iter \u003d make_timestamp_iter()"},{"line_number":3712,"context_line":"        ts_now \u003d Timestamp.now()  # used when mocking Timestamp.now()"}],"source_content_type":"text/x-python","patch_set":38,"id":"ced1e0ee_ebb2028f","line":3709,"in_reply_to":"1712cb45_f3937632","updated":"2024-01-04 16:20:07.000000000","message":"this test is much shorter in the follow on patch","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"18e816986da0f7d7dbb1575c0927744a62acb048","unresolved":true,"context_lines":[{"line_number":3769,"context_line":"            self.assertEqual(\u0027shard\u0027, resp.headers[\u0027X-Backend-Record-Type\u0027])"},{"line_number":3770,"context_line":"            self.assertIn(\u0027X-Backend-Record-Shard-Format\u0027, resp.headers)"},{"line_number":3771,"context_line":"            self.assertEqual("},{"line_number":3772,"context_line":"                \u0027full\u0027, resp.headers[\u0027X-Backend-Record-Shard-Format\u0027])"},{"line_number":3773,"context_line":""},{"line_number":3774,"context_line":"        check_shard_GET(shard_ranges[:3], \u0027a/c\u0027,"},{"line_number":3775,"context_line":"                        params\u003d\u0027\u0026states\u003dlisting\u0026end_marker\u003dpickle\u0027)"}],"source_content_type":"text/x-python","patch_set":38,"id":"ce6999c1_c2d615ef","line":3772,"updated":"2023-11-10 19:48:08.000000000","message":"despite asking for `shard-format: namespace` the returned response is `shard-format: full` for all the conditions enumerated by `_check_shard_GET`","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"18e816986da0f7d7dbb1575c0927744a62acb048","unresolved":true,"context_lines":[{"line_number":3772,"context_line":"                \u0027full\u0027, resp.headers[\u0027X-Backend-Record-Shard-Format\u0027])"},{"line_number":3773,"context_line":""},{"line_number":3774,"context_line":"        check_shard_GET(shard_ranges[:3], \u0027a/c\u0027,"},{"line_number":3775,"context_line":"                        params\u003d\u0027\u0026states\u003dlisting\u0026end_marker\u003dpickle\u0027)"},{"line_number":3776,"context_line":"        check_shard_GET("},{"line_number":3777,"context_line":"            reversed(shard_ranges[:3]), \u0027a/c\u0027,"},{"line_number":3778,"context_line":"            params\u003d\u0027\u0026states\u003dlisting\u0026reverse\u003dtrue\u0026marker\u003dpickle\u0027)"}],"source_content_type":"text/x-python","patch_set":38,"id":"70684f35_e862f45f","line":3775,"updated":"2023-11-10 19:48:08.000000000","message":"\u0027\u0027\u003d\u003eapple, apple\u003d\u003eham, ham\u003d\u003esalami","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bff0a54e68a76090810057a6fac976d6d5e3e745","unresolved":false,"context_lines":[{"line_number":3772,"context_line":"                \u0027full\u0027, resp.headers[\u0027X-Backend-Record-Shard-Format\u0027])"},{"line_number":3773,"context_line":""},{"line_number":3774,"context_line":"        check_shard_GET(shard_ranges[:3], \u0027a/c\u0027,"},{"line_number":3775,"context_line":"                        params\u003d\u0027\u0026states\u003dlisting\u0026end_marker\u003dpickle\u0027)"},{"line_number":3776,"context_line":"        check_shard_GET("},{"line_number":3777,"context_line":"            reversed(shard_ranges[:3]), \u0027a/c\u0027,"},{"line_number":3778,"context_line":"            params\u003d\u0027\u0026states\u003dlisting\u0026reverse\u003dtrue\u0026marker\u003dpickle\u0027)"}],"source_content_type":"text/x-python","patch_set":38,"id":"2420fbb9_ee3c9705","line":3775,"in_reply_to":"70684f35_e862f45f","updated":"2024-01-04 16:20:07.000000000","message":"Acknowledged","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"18e816986da0f7d7dbb1575c0927744a62acb048","unresolved":true,"context_lines":[{"line_number":3777,"context_line":"            reversed(shard_ranges[:3]), \u0027a/c\u0027,"},{"line_number":3778,"context_line":"            params\u003d\u0027\u0026states\u003dlisting\u0026reverse\u003dtrue\u0026marker\u003dpickle\u0027)"},{"line_number":3779,"context_line":"        check_shard_GET(shard_ranges[1:4], \u0027a/c\u0027,"},{"line_number":3780,"context_line":"                        params\u003d\u0027\u0026states\u003dupdating\u0026end_marker\u003dtreacle\u0027)"},{"line_number":3781,"context_line":"        check_shard_GET("},{"line_number":3782,"context_line":"            reversed(shard_ranges[1:4]), \u0027a/c\u0027,"},{"line_number":3783,"context_line":"            params\u003d\u0027\u0026states\u003dupdating\u0026reverse\u003dtrue\u0026marker\u003dtreacle\u0027)"}],"source_content_type":"text/x-python","patch_set":38,"id":"72f1e345_f1b6a6ff","line":3780,"updated":"2023-11-10 19:48:08.000000000","message":"\u0027\u0027\u003d\u003eapple is SHRINKING, not good for updates - ends with salami\u003d\u003eyoghurt","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bff0a54e68a76090810057a6fac976d6d5e3e745","unresolved":false,"context_lines":[{"line_number":3777,"context_line":"            reversed(shard_ranges[:3]), \u0027a/c\u0027,"},{"line_number":3778,"context_line":"            params\u003d\u0027\u0026states\u003dlisting\u0026reverse\u003dtrue\u0026marker\u003dpickle\u0027)"},{"line_number":3779,"context_line":"        check_shard_GET(shard_ranges[1:4], \u0027a/c\u0027,"},{"line_number":3780,"context_line":"                        params\u003d\u0027\u0026states\u003dupdating\u0026end_marker\u003dtreacle\u0027)"},{"line_number":3781,"context_line":"        check_shard_GET("},{"line_number":3782,"context_line":"            reversed(shard_ranges[1:4]), \u0027a/c\u0027,"},{"line_number":3783,"context_line":"            params\u003d\u0027\u0026states\u003dupdating\u0026reverse\u003dtrue\u0026marker\u003dtreacle\u0027)"}],"source_content_type":"text/x-python","patch_set":38,"id":"ab244ffd_90b3907b","line":3780,"in_reply_to":"72f1e345_f1b6a6ff","updated":"2024-01-04 16:20:07.000000000","message":"Acknowledged","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"18e816986da0f7d7dbb1575c0927744a62acb048","unresolved":true,"context_lines":[{"line_number":3782,"context_line":"            reversed(shard_ranges[1:4]), \u0027a/c\u0027,"},{"line_number":3783,"context_line":"            params\u003d\u0027\u0026states\u003dupdating\u0026reverse\u003dtrue\u0026marker\u003dtreacle\u0027)"},{"line_number":3784,"context_line":"        check_shard_GET(shard_ranges[1:2], \u0027a/c\u0027, params\u003d\u0027\u0026includes\u003dcheese\u0027)"},{"line_number":3785,"context_line":"        check_shard_GET(shard_ranges[1:2], \u0027a/c\u0027, params\u003d\u0027\u0026includes\u003dham\u0027)"},{"line_number":3786,"context_line":"        check_shard_GET(reversed(shard_ranges), \u0027a/c\u0027, params\u003d\u0027\u0026reverse\u003dtrue\u0027)"},{"line_number":3787,"context_line":""},{"line_number":3788,"context_line":"        # Test namespace GET with \u0027X-Backend-Include-Deleted\u0027 header."}],"source_content_type":"text/x-python","patch_set":38,"id":"32523b97_b38fe254","line":3785,"updated":"2023-11-10 19:48:08.000000000","message":"cheese and ham are both in apple\u003d\u003eham","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bff0a54e68a76090810057a6fac976d6d5e3e745","unresolved":false,"context_lines":[{"line_number":3782,"context_line":"            reversed(shard_ranges[1:4]), \u0027a/c\u0027,"},{"line_number":3783,"context_line":"            params\u003d\u0027\u0026states\u003dupdating\u0026reverse\u003dtrue\u0026marker\u003dtreacle\u0027)"},{"line_number":3784,"context_line":"        check_shard_GET(shard_ranges[1:2], \u0027a/c\u0027, params\u003d\u0027\u0026includes\u003dcheese\u0027)"},{"line_number":3785,"context_line":"        check_shard_GET(shard_ranges[1:2], \u0027a/c\u0027, params\u003d\u0027\u0026includes\u003dham\u0027)"},{"line_number":3786,"context_line":"        check_shard_GET(reversed(shard_ranges), \u0027a/c\u0027, params\u003d\u0027\u0026reverse\u003dtrue\u0027)"},{"line_number":3787,"context_line":""},{"line_number":3788,"context_line":"        # Test namespace GET with \u0027X-Backend-Include-Deleted\u0027 header."}],"source_content_type":"text/x-python","patch_set":38,"id":"9f0c7dc5_079843f2","line":3785,"in_reply_to":"32523b97_b38fe254","updated":"2024-01-04 16:20:07.000000000","message":"Acknowledged","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"18e816986da0f7d7dbb1575c0927744a62acb048","unresolved":true,"context_lines":[{"line_number":3783,"context_line":"            params\u003d\u0027\u0026states\u003dupdating\u0026reverse\u003dtrue\u0026marker\u003dtreacle\u0027)"},{"line_number":3784,"context_line":"        check_shard_GET(shard_ranges[1:2], \u0027a/c\u0027, params\u003d\u0027\u0026includes\u003dcheese\u0027)"},{"line_number":3785,"context_line":"        check_shard_GET(shard_ranges[1:2], \u0027a/c\u0027, params\u003d\u0027\u0026includes\u003dham\u0027)"},{"line_number":3786,"context_line":"        check_shard_GET(reversed(shard_ranges), \u0027a/c\u0027, params\u003d\u0027\u0026reverse\u003dtrue\u0027)"},{"line_number":3787,"context_line":""},{"line_number":3788,"context_line":"        # Test namespace GET with \u0027X-Backend-Include-Deleted\u0027 header."},{"line_number":3789,"context_line":"        req \u003d Request.blank("}],"source_content_type":"text/x-python","patch_set":38,"id":"89460088_9142b3df","line":3786,"updated":"2023-11-10 19:48:08.000000000","message":"ah, and even reverse by itself will force `format: full` excellent.\n\nwhat about states\u003dauditing?","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bff0a54e68a76090810057a6fac976d6d5e3e745","unresolved":false,"context_lines":[{"line_number":3783,"context_line":"            params\u003d\u0027\u0026states\u003dupdating\u0026reverse\u003dtrue\u0026marker\u003dtreacle\u0027)"},{"line_number":3784,"context_line":"        check_shard_GET(shard_ranges[1:2], \u0027a/c\u0027, params\u003d\u0027\u0026includes\u003dcheese\u0027)"},{"line_number":3785,"context_line":"        check_shard_GET(shard_ranges[1:2], \u0027a/c\u0027, params\u003d\u0027\u0026includes\u003dham\u0027)"},{"line_number":3786,"context_line":"        check_shard_GET(reversed(shard_ranges), \u0027a/c\u0027, params\u003d\u0027\u0026reverse\u003dtrue\u0027)"},{"line_number":3787,"context_line":""},{"line_number":3788,"context_line":"        # Test namespace GET with \u0027X-Backend-Include-Deleted\u0027 header."},{"line_number":3789,"context_line":"        req \u003d Request.blank("}],"source_content_type":"text/x-python","patch_set":38,"id":"8f7ef1ce_d717a4e4","line":3786,"in_reply_to":"89460088_9142b3df","updated":"2024-01-04 16:20:07.000000000","message":"Acknowledged","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"18e816986da0f7d7dbb1575c0927744a62acb048","unresolved":true,"context_lines":[{"line_number":3796,"context_line":"            })"},{"line_number":3797,"context_line":"        resp \u003d req.get_response(self.controller)"},{"line_number":3798,"context_line":"        self.assertEqual(resp.status, \u0027400 Bad Request\u0027)"},{"line_number":3799,"context_line":"        self.assertEqual(resp.body, b\u0027No include_deleted for namespace GET\u0027)"},{"line_number":3800,"context_line":""},{"line_number":3801,"context_line":"    def test_GET_auto_record_type(self):"},{"line_number":3802,"context_line":"        # make a container"}],"source_content_type":"text/x-python","patch_set":38,"id":"918db2f6_2a2f6f49","line":3799,"updated":"2023-11-10 19:48:08.000000000","message":"for now this is the only argument we think should be in compatible with format: namespace","commit_id":"7b6ccbad66f76ba1622c6754ccef53c8652c5561"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"b15761c0e0e2d7e69f93b97e5e7b7847078fdb4a","unresolved":true,"context_lines":[{"line_number":3643,"context_line":"        resp \u003d req.get_response(self.controller)"},{"line_number":3644,"context_line":"        self.assertIn(\u0027X-Backend-Record-Type\u0027, resp.headers)"},{"line_number":3645,"context_line":"        self.assertEqual("},{"line_number":3646,"context_line":"            \u0027shard\u0027, resp.headers[\u0027X-Backend-Record-Type\u0027])"},{"line_number":3647,"context_line":"        raw_sr \u003d json.loads(resp.body)"},{"line_number":3648,"context_line":"        expected_sr \u003d [{\u0027name\u0027: sr[\u0027name\u0027], \u0027lower\u0027: sr[\u0027lower\u0027],"},{"line_number":3649,"context_line":"                        \u0027upper\u0027: sr[\u0027upper\u0027]}"}],"source_content_type":"text/x-python","patch_set":56,"id":"85f38ec3_fa12915a","line":3646,"updated":"2024-01-09 15:23:00.000000000","message":"we\u0027re not asserting format\u003dfull in the response","commit_id":"e3676eff802dcde5eeabbda8d5707025f40412ad"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"2224f5d5b001bc0faf7486abd899d82341056002","unresolved":false,"context_lines":[{"line_number":3643,"context_line":"        resp \u003d req.get_response(self.controller)"},{"line_number":3644,"context_line":"        self.assertIn(\u0027X-Backend-Record-Type\u0027, resp.headers)"},{"line_number":3645,"context_line":"        self.assertEqual("},{"line_number":3646,"context_line":"            \u0027shard\u0027, resp.headers[\u0027X-Backend-Record-Type\u0027])"},{"line_number":3647,"context_line":"        raw_sr \u003d json.loads(resp.body)"},{"line_number":3648,"context_line":"        expected_sr \u003d [{\u0027name\u0027: sr[\u0027name\u0027], \u0027lower\u0027: sr[\u0027lower\u0027],"},{"line_number":3649,"context_line":"                        \u0027upper\u0027: sr[\u0027upper\u0027]}"}],"source_content_type":"text/x-python","patch_set":56,"id":"c8174f2e_33f8f55e","line":3646,"in_reply_to":"6917f44c_3203a0d3","updated":"2024-01-09 20:41:09.000000000","message":"Acknowledged","commit_id":"e3676eff802dcde5eeabbda8d5707025f40412ad"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"6f847c2e4cbc4628a68dbeef23ed7a9f1c05c8d1","unresolved":true,"context_lines":[{"line_number":3643,"context_line":"        resp \u003d req.get_response(self.controller)"},{"line_number":3644,"context_line":"        self.assertIn(\u0027X-Backend-Record-Type\u0027, resp.headers)"},{"line_number":3645,"context_line":"        self.assertEqual("},{"line_number":3646,"context_line":"            \u0027shard\u0027, resp.headers[\u0027X-Backend-Record-Type\u0027])"},{"line_number":3647,"context_line":"        raw_sr \u003d json.loads(resp.body)"},{"line_number":3648,"context_line":"        expected_sr \u003d [{\u0027name\u0027: sr[\u0027name\u0027], \u0027lower\u0027: sr[\u0027lower\u0027],"},{"line_number":3649,"context_line":"                        \u0027upper\u0027: sr[\u0027upper\u0027]}"}],"source_content_type":"text/x-python","patch_set":56,"id":"6917f44c_3203a0d3","line":3646,"in_reply_to":"85f38ec3_fa12915a","updated":"2024-01-09 16:47:31.000000000","message":"done here https://review.opendev.org/c/openstack/swift/+/905124?usp\u003demail","commit_id":"e3676eff802dcde5eeabbda8d5707025f40412ad"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"b15761c0e0e2d7e69f93b97e5e7b7847078fdb4a","unresolved":true,"context_lines":[{"line_number":3780,"context_line":"            \u0027namespace\u0027, resp.headers[\u0027X-Backend-Record-Shard-Format\u0027])"},{"line_number":3781,"context_line":"        self.assertIn(\u0027X-Backend-Override-Shard-Name-Filter\u0027, resp.headers)"},{"line_number":3782,"context_line":"        self.assertTrue("},{"line_number":3783,"context_line":"            resp.headers[\u0027X-Backend-Override-Shard-Name-Filter\u0027])"},{"line_number":3784,"context_line":""},{"line_number":3785,"context_line":"    def test_GET_namespaces_sharded_subshard_state_listing(self):"},{"line_number":3786,"context_line":"        # shard\u0027s namespaces for listing"}],"source_content_type":"text/x-python","patch_set":56,"id":"007cf251_7e99f67d","line":3783,"updated":"2024-01-09 15:23:00.000000000","message":"compare with shard ranges, similar to the unsharded test?","commit_id":"e3676eff802dcde5eeabbda8d5707025f40412ad"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"2224f5d5b001bc0faf7486abd899d82341056002","unresolved":false,"context_lines":[{"line_number":3780,"context_line":"            \u0027namespace\u0027, resp.headers[\u0027X-Backend-Record-Shard-Format\u0027])"},{"line_number":3781,"context_line":"        self.assertIn(\u0027X-Backend-Override-Shard-Name-Filter\u0027, resp.headers)"},{"line_number":3782,"context_line":"        self.assertTrue("},{"line_number":3783,"context_line":"            resp.headers[\u0027X-Backend-Override-Shard-Name-Filter\u0027])"},{"line_number":3784,"context_line":""},{"line_number":3785,"context_line":"    def test_GET_namespaces_sharded_subshard_state_listing(self):"},{"line_number":3786,"context_line":"        # shard\u0027s namespaces for listing"}],"source_content_type":"text/x-python","patch_set":56,"id":"fdfaa4f8_7485bbf8","line":3783,"in_reply_to":"007cf251_7e99f67d","updated":"2024-01-09 20:41:09.000000000","message":"Acknowledged","commit_id":"e3676eff802dcde5eeabbda8d5707025f40412ad"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"b15761c0e0e2d7e69f93b97e5e7b7847078fdb4a","unresolved":true,"context_lines":[{"line_number":3839,"context_line":""},{"line_number":3840,"context_line":"        # Test namespace GET with \u0027include\u0027 or \u0027marker/end_marker\u0027 or \u0027reverse\u0027"},{"line_number":3841,"context_line":"        # parameters which are not supported."},{"line_number":3842,"context_line":"        def check_namespace_GET(expected_shard_ranges, path, params\u003d\u0027\u0027):"},{"line_number":3843,"context_line":"            req \u003d Request.blank("},{"line_number":3844,"context_line":"                \u0027/sda1/p/%s?format\u003djson%s\u0027 % (path, params), method\u003d\u0027GET\u0027,"},{"line_number":3845,"context_line":"                headers\u003d{"}],"source_content_type":"text/x-python","patch_set":56,"id":"8e11c412_42f5b0d2","line":3842,"range":{"start_line":3842,"start_character":32,"end_line":3842,"end_character":54},"updated":"2024-01-09 15:23:00.000000000","message":"nit: rename to ``expected_namespaces``","commit_id":"e3676eff802dcde5eeabbda8d5707025f40412ad"},{"author":{"_account_id":34930,"name":"Jianjian Huo","email":"jhuo@nvidia.com","username":"jhuo"},"change_message_id":"2224f5d5b001bc0faf7486abd899d82341056002","unresolved":false,"context_lines":[{"line_number":3839,"context_line":""},{"line_number":3840,"context_line":"        # Test namespace GET with \u0027include\u0027 or \u0027marker/end_marker\u0027 or \u0027reverse\u0027"},{"line_number":3841,"context_line":"        # parameters which are not supported."},{"line_number":3842,"context_line":"        def check_namespace_GET(expected_shard_ranges, path, params\u003d\u0027\u0027):"},{"line_number":3843,"context_line":"            req \u003d Request.blank("},{"line_number":3844,"context_line":"                \u0027/sda1/p/%s?format\u003djson%s\u0027 % (path, params), method\u003d\u0027GET\u0027,"},{"line_number":3845,"context_line":"                headers\u003d{"}],"source_content_type":"text/x-python","patch_set":56,"id":"cfc223de_0206261b","line":3842,"range":{"start_line":3842,"start_character":32,"end_line":3842,"end_character":54},"in_reply_to":"8e11c412_42f5b0d2","updated":"2024-01-09 20:41:09.000000000","message":"Acknowledged","commit_id":"e3676eff802dcde5eeabbda8d5707025f40412ad"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"2a05535963b506bdea7d14b1837ad1324a75db19","unresolved":true,"context_lines":[{"line_number":1746,"context_line":"    def _populate_and_delete_container(self, path):"},{"line_number":1747,"context_line":"        sr \u003d self._populate_container(path)"},{"line_number":1748,"context_line":"        req \u003d Request.blank("},{"line_number":1749,"context_line":"            \u0027/sda1/p/a/c\u0027,"},{"line_number":1750,"context_line":"            environ\u003d{\u0027REQUEST_METHOD\u0027: \u0027DELETE\u0027},"},{"line_number":1751,"context_line":"            headers\u003d{\u0027X-Timestamp\u0027: next(self.ts).internal})"},{"line_number":1752,"context_line":"        resp \u003d req.get_response(self.controller)"}],"source_content_type":"text/x-python","patch_set":57,"id":"fd79b1d9_231c3734","line":1749,"range":{"start_line":1749,"start_character":12,"end_line":1749,"end_character":25},"updated":"2024-01-11 05:56:18.000000000","message":"I assume this should be path not hard coded?","commit_id":"7be1763455236ebf184d6be483798bad5e2acc9d"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"e21248dd559c344333f1b3fec3ea14b01d2e0ac0","unresolved":false,"context_lines":[{"line_number":1746,"context_line":"    def _populate_and_delete_container(self, path):"},{"line_number":1747,"context_line":"        sr \u003d self._populate_container(path)"},{"line_number":1748,"context_line":"        req \u003d Request.blank("},{"line_number":1749,"context_line":"            \u0027/sda1/p/a/c\u0027,"},{"line_number":1750,"context_line":"            environ\u003d{\u0027REQUEST_METHOD\u0027: \u0027DELETE\u0027},"},{"line_number":1751,"context_line":"            headers\u003d{\u0027X-Timestamp\u0027: next(self.ts).internal})"},{"line_number":1752,"context_line":"        resp \u003d req.get_response(self.controller)"}],"source_content_type":"text/x-python","patch_set":57,"id":"9085fd96_75362f96","line":1749,"range":{"start_line":1749,"start_character":12,"end_line":1749,"end_character":25},"in_reply_to":"fd79b1d9_231c3734","updated":"2024-01-11 10:47:57.000000000","message":"It surely should! good eyes @Matt","commit_id":"7be1763455236ebf184d6be483798bad5e2acc9d"}]}