)]}'
{"/COMMIT_MSG":[{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"a5dfbac26b9f07b7ced82f3e120f49028d403e51","unresolved":true,"context_lines":[{"line_number":7,"context_line":"WIP proxy: Add systags abstraction for etag override"},{"line_number":8,"context_line":""},{"line_number":9,"context_line":"TODO:"},{"line_number":10,"context_line":"- add compute-md5-hash-on-demand to the proxy app"},{"line_number":11,"context_line":"- more obj controller unit tests"},{"line_number":12,"context_line":"- check for unnecessary modifications (particularly in tests)"},{"line_number":13,"context_line":""}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"dbf876bc_31222851","line":10,"range":{"start_line":10,"start_character":2,"end_line":10,"end_character":49},"updated":"2025-02-28 17:58:15.000000000","message":"this isn\u0027t strictly necessary on master - we don\u0027t need it yet, maybe for s3 checksum patch - so this could well be a follow on patch","commit_id":"89a2126d51d1ef73170b6c00e68dcc06bc95d4ce"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"a5dfbac26b9f07b7ced82f3e120f49028d403e51","unresolved":true,"context_lines":[{"line_number":24,"context_line":"to the request environ which provides an abstraction from the"},{"line_number":25,"context_line":"override-etag header and footer. Middlewares simply update the systags"},{"line_number":26,"context_line":"map with override etag and other parameter values, and the proxy app"},{"line_number":27,"context_line":"alone is responsible for generating the correct header or footer."},{"line_number":28,"context_line":""},{"line_number":29,"context_line":"Change-Id: I218bb74f1fc673982773f64907b928790fca17d0"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"3892a112_3e860992","line":27,"updated":"2025-02-28 17:58:15.000000000","message":"The response path abstraction is still not so great - middlewares still have to parse the response sysmeta header.","commit_id":"89a2126d51d1ef73170b6c00e68dcc06bc95d4ce"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":24,"context_line":"to the request environ which provides an abstraction from the"},{"line_number":25,"context_line":"override-etag header and footer. Middlewares simply update the systags"},{"line_number":26,"context_line":"map with override etag and other parameter values, and the proxy app"},{"line_number":27,"context_line":"alone is responsible for generating the correct header or footer."},{"line_number":28,"context_line":""},{"line_number":29,"context_line":"Change-Id: I218bb74f1fc673982773f64907b928790fca17d0"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"fbb47a01_ebb959b1","line":27,"in_reply_to":"3892a112_3e860992","updated":"2025-03-11 21:27:38.000000000","message":"ok, so havning a consistent interface for getting systags *out* of the override sysmeta might be nice; and would probably be different from the interface for getting systags out of the container listing hash key....","commit_id":"89a2126d51d1ef73170b6c00e68dcc06bc95d4ce"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":21,"context_line":""},{"line_number":22,"context_line":"This patch addresses those shortcomings by introducing a systags map"},{"line_number":23,"context_line":"to the request environ which provides an abstraction from the"},{"line_number":24,"context_line":"override-etag header and footer. Middlewares simply update the systags"},{"line_number":25,"context_line":"map with override etag and other parameter values, and the proxy app"},{"line_number":26,"context_line":"alone is responsible for generating the correct header or footer."},{"line_number":27,"context_line":""}],"source_content_type":"text/x-gerrit-commit-message","patch_set":5,"id":"82e43dfa_3170cd72","line":24,"updated":"2025-03-11 21:27:38.000000000","message":"I would assume the situation where the update-etag header is still sent via headers vs. footers can still be significant to some middlewares that may want to add systags (after they\u0027ve read the client body?)\n\nI would assume this interface wouldn\u0027t try to force an unencrypted/replicated object to send mime-doc-footers \"just\" because some lazy systags showed up AFTER they\u0027d already send the whole object?","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"}],"/PATCHSET_LEVEL":[{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bb78dfd45a5b210ff2159d324401c9d613469614","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":5,"id":"57ddc0c6_9fb06ae8","updated":"2025-03-19 11:44:11.000000000","message":"@clay.gerrard@gmail.com Wow thanks for your review! You will tell from my replies that there are aspects of this that I am not yet totally comfortable with, and you\u0027re feedback is really helpful to push my thinking forwards.\n\nFYI I think this is going to become a slow-ish burn topic while we get the earlier features of s3api chunked transfer and checksums done.","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":5,"id":"89e9181f_b53a9094","updated":"2025-03-11 21:27:38.000000000","message":"ok, as suggested I didn\u0027t get into tests.  I only made one pass and didn\u0027t check it out - so please assume all of my comments represent ignorance rather than any kind of considered insight or perspective.\n\nI\u0027d like to learn better\n\n1) why do mw need to know if they should use init/inhert/get systags\n2) should mw not be able to see systags added after them in the pipeline\n3) why do we want a systag callback instead of \"just\" adding systags in the footers callback\n4) is the reason we don\u0027t want to add systags in the footers callback related to the finalize_systags call in the obj.controller\n\nI\u0027d like to consider more\n\nsystags seems to inherit the special handling of update-etag (but NOT update-size), if so - is that the correct level of responsibility, or should systags be for *just* \"extra k/v pairs that go in the listing\" and the technique/interface for updating the etag (when/if needed: e.g. slo/s3api/crypto) is *completely* orthogonal from \"add random k/v pairs to the listing\" (which then as an implementation detail happens to interact with the update-etag subsystem way down in the object controller)","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bb78dfd45a5b210ff2159d324401c9d613469614","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":5,"id":"b0f8d850_da21737b","in_reply_to":"89e9181f_b53a9094","updated":"2025-03-19 11:44:11.000000000","message":"\u003e 1) why do mw need to know if they should use init/inhert/get systags\n\ninit_systags is used by the encrypter to replace the systags. The encrypter is a special case: it consumes all the existing systags and encrypts them into a single key/val pair (etag). It needs to prevent the proxy app then also processing the existing systags. I considered having the encrypter pop systags from the existing dict, but that interferes with an earlier mw that wants to reuse the systags for another (sub)request. I considered the encrypter wrapping the existing (inherit) and setting values to None, with None implying \u0027don\u0027t include in the override etag header\u0027, but I didn\u0027t like the reliance on the None-implies-ignore. I considered having encrypter go further an actually construct the override etag header so that the proxy app would do nothing, but I was trying to encapsulate the header construction in the proxy app.\n\n\u0027inherit_systags\u0027 (a.k.a. wrap_systags?) is used when copying systags into a subrequest i.e. when there is a fork in the request handling path. The subrequest may have systags added that we do NOT want polluting the original request. That may not currently be the case (?), but I\u0027m thinking of bugs where we\u0027ve re-used an environ for multiple requests and needed to weed out environ keys that inappropriately bleed from one request to another. It seems like good hygiene to prevent a subrequest polluting the original request\u0027s systags. (Thinking about this again, I guess the good hygiene still depends on ordering :/ ... if the original request is completed before the subrequest then the subrequest is going to get the original systags \u0027polluted\u0027 by whatever came from later mw handling the original request).\n\n\u003e 2) should mw not be able to see systags added after them in the pipeline\n\nThey can until \u0027inherit_systags\u0027 is used to wrap the existing, which IIRC only happens in some cases when a subrequest is created.\n\n\u003e 3) why do we want a systag callback instead of \"just\" adding systags in the\n\u003e footers callback\n\nI\u0027m trying to make the systags interface abstract w.r.t. how they get to the container server, so a mw that wants to set a systag shouldn\u0027t need to know about footer callbacks.\n\n\"You want a systag - set a key:value pair in this dict. You\u0027re not sure of it\u0027s value yet? Set a key:callback pair in the dict and we\u0027ll get back to you.\"\n\nvs.\n\n\"You want a systag - set a key:value pair in this dict. You\u0027re not sure of it\u0027s value yet? Keep a hold on this dict, install a footers callback, in your footers callback update this dict.\" (So we could have a \u0027systags_callback\u0027 which would make this sound a little more palatable, but why not register the need for a callback using the key:value pair?)\n\nIt\u0027s also handy when a mw WILL be providing the etag value and needs to signal that to the proxy app. The mw sets a etag:callback pair and the proxy app knows that it doesn\u0027t have to calculate the etag vs the mw waits for a footers callback to set etag:value, and the proxy has no idea that the mw is going to do that so also calculates the etag.  (So a workaround would be for mw to set etag:placeholder, install a footers_callback and then fix the etag value in the callback).\n\n\u003e 4) is the reason we don\u0027t want to add systags in the footers callback related\n\u003e to the finalize_systags call in the obj.controller\n\nmw can update systags in a footers_callback, and the updates will get included because footers_callbacks are called *before* the proxy finalizes systags. (that claim needs test coverage!)\n\n\u003e systags seems to inherit the special handling of update-etag (but NOT \n\u003e update-size), if so - is that the correct level of responsibility, or should\n\u003e systags be for just \"extra k/v pairs that go in the listing\" and the\n\u003e technique/interface for updating the etag (when/if needed: e.g.\n\u003e slo/s3api/crypto) is completely orthogonal from \"add random k/v pairs to the\n\u003e listing\" (which then as an implementation detail happens to interact with the\n\u003e update-etag subsystem way down in the object controller)\n\nThis is such a perceptive question!\n\nI ended up including etag as a systag key because it provided a mechanism (via a key:callback pair as discussed above) for mw to signal to the proxy that mw will provide the etag value, so the proxy doesn\u0027t need to calculate one.\n\nI could have retained the mechanism of mw adding the override-etag header, unadorned, and just moved all the other k:v pairs to the systags dict. But mw would need to give the header a placeholder value to prevent the proxy calculating the etag. And my goal was for mw to not know about the mechanics of how the container listing gets overridden.\n\nSo I ended up thinking about etag as being just one of an arbitrary set of k:v pairs that mw can add to the container listing item, except etag is a special case that will always be given a default value in the listing if not set by mw in systags.\n\nAnd of course, the other special case is \u0027size\u0027...and in principle there is no reason why \u0027size\u0027 should not also be supported in systags, with the proxy app needing to map it to the override-size header. I just constrained this patch to the override-etag header.\n\nSo I can imagine a doc saying something like \"By default container listing item dicts have keys \u0027etag\u0027, \u0027size\u0027, \u0027content-type\u0027 etc with default values set by the proxy app. A mw can vary or add to those by setting key:value pairs in the systags dict.\"\n\n(Note: there\u0027s an implication in that statement of further work which is, early in the listing response path, to have the override-etag unpacked into key:value pairs *in a single place* rather then each middleware having to extract the params it cares about. SO by the time mw gets a listing response the dicts are already populated with all the unpacked systags).","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"}],"swift/common/middleware/copy.py":[{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"a5dfbac26b9f07b7ced82f3e120f49028d403e51","unresolved":true,"context_lines":[{"line_number":414,"context_line":"            #  - DLO: etag in DLO response is not hash of actual content"},{"line_number":415,"context_line":"            sink_req.headers[\u0027Etag\u0027] \u003d source_resp.etag"},{"line_number":416,"context_line":"            resp_systags \u003d parse_etag_override_header(source_resp.headers)"},{"line_number":417,"context_line":"            underlay_systags(sink_req.environ, resp_systags)"},{"line_number":418,"context_line":"        else:"},{"line_number":419,"context_line":"            # since we\u0027re not copying the source etag, make sure that any"},{"line_number":420,"context_line":"            # container update override values are not copied."}],"source_content_type":"text/x-python","patch_set":1,"id":"26cd98f7_5a44b134","line":417,"updated":"2025-02-28 17:58:15.000000000","message":"ok, so we do now have to be mindful of headers and systags when shuttling state from responses to requests\n\nBTW the use of underlay here is to perserve the legacy precedence of put req headers over get response headers","commit_id":"89a2126d51d1ef73170b6c00e68dcc06bc95d4ce"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":415,"context_line":"            #  - DLO: etag in DLO response is not hash of actual content"},{"line_number":416,"context_line":"            sink_req.headers[\u0027Etag\u0027] \u003d source_resp.etag"},{"line_number":417,"context_line":"            resp_systags \u003d parse_override_etag_header(source_resp.headers)"},{"line_number":418,"context_line":"            inherit_systags(sink_req.environ, resp_systags)"},{"line_number":419,"context_line":"        else:"},{"line_number":420,"context_line":"            # since we\u0027re not copying the source etag, make sure that any"},{"line_number":421,"context_line":"            # container update override values are not copied."}],"source_content_type":"text/x-python","patch_set":5,"id":"a865ee41_709ad55b","line":418,"updated":"2025-03-11 21:27:38.000000000","message":"I might assume this isn\u0027t strictly *required* - but perhaps we\u0027ve tried to \"cut off\" the ability for mw to send the override-etag-header and so it\u0027s required.","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bb78dfd45a5b210ff2159d324401c9d613469614","unresolved":true,"context_lines":[{"line_number":415,"context_line":"            #  - DLO: etag in DLO response is not hash of actual content"},{"line_number":416,"context_line":"            sink_req.headers[\u0027Etag\u0027] \u003d source_resp.etag"},{"line_number":417,"context_line":"            resp_systags \u003d parse_override_etag_header(source_resp.headers)"},{"line_number":418,"context_line":"            inherit_systags(sink_req.environ, resp_systags)"},{"line_number":419,"context_line":"        else:"},{"line_number":420,"context_line":"            # since we\u0027re not copying the source etag, make sure that any"},{"line_number":421,"context_line":"            # container update override values are not copied."}],"source_content_type":"text/x-python","patch_set":5,"id":"86df63a3_71585018","line":418,"in_reply_to":"a865ee41_709ad55b","updated":"2025-03-19 11:44:11.000000000","message":"I certainly want to discourage mw from using the header","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"}],"swift/common/middleware/crypto/decrypter.py":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":225,"context_line":"                decrypted_etag \u003d self._decrypt_header("},{"line_number":226,"context_line":"                    SYSTAGS_HEADER_KEY, encrypted_etag, put_keys[\u0027container\u0027])"},{"line_number":227,"context_line":"                mod_hdr_pairs.append((SYSTAGS_HEADER_KEY, decrypted_etag))"},{"line_number":228,"context_line":""},{"line_number":229,"context_line":"        # Decrypt all user metadata. Encrypted user metadata values are stored"},{"line_number":230,"context_line":"        # in the x-object-transient-sysmeta-crypto-meta- namespace. Those are"},{"line_number":231,"context_line":"        # decrypted and moved back to the x-object-meta- namespace. Prior to"}],"source_content_type":"text/x-python","patch_set":5,"id":"5a876922_fa6eed16","line":228,"updated":"2025-03-11 21:27:38.000000000","message":"interesting; so the proxy couldn\u0027t parse the systags from the etag-update-override header in the general sense regardless...","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bb78dfd45a5b210ff2159d324401c9d613469614","unresolved":true,"context_lines":[{"line_number":225,"context_line":"                decrypted_etag \u003d self._decrypt_header("},{"line_number":226,"context_line":"                    SYSTAGS_HEADER_KEY, encrypted_etag, put_keys[\u0027container\u0027])"},{"line_number":227,"context_line":"                mod_hdr_pairs.append((SYSTAGS_HEADER_KEY, decrypted_etag))"},{"line_number":228,"context_line":""},{"line_number":229,"context_line":"        # Decrypt all user metadata. Encrypted user metadata values are stored"},{"line_number":230,"context_line":"        # in the x-object-transient-sysmeta-crypto-meta- namespace. Those are"},{"line_number":231,"context_line":"        # decrypted and moved back to the x-object-meta- namespace. Prior to"}],"source_content_type":"text/x-python","patch_set":5,"id":"02893add_dcde3a3e","line":228,"in_reply_to":"5a876922_fa6eed16","updated":"2025-03-19 11:44:11.000000000","message":"it\u0027s frustrating that in the response path we don\u0027t have anywhere to conveniently stash the parsed systags\n\nthe proxy could have parsed the header, but would only have found ``etag: encrypted_value``, then the decrypter would have needed to decrypt and expand to aother k:v pairs","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"}],"swift/common/middleware/crypto/encrypter.py":[{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"a5dfbac26b9f07b7ced82f3e120f49028d403e51","unresolved":true,"context_lines":[{"line_number":188,"context_line":"                systags[\u0027etag\u0027] \u003d append_crypto_meta(enc_val, crypto_meta)"},{"line_number":189,"context_line":""},{"line_number":190,"context_line":"            # encrypter always determines the final override-etag footer value"},{"line_number":191,"context_line":"            # TODO: this isn\u0027t necessary (other than to avoid test churn??)"},{"line_number":192,"context_line":"            compose_etag_override_header(footers, systags)"},{"line_number":193,"context_line":""},{"line_number":194,"context_line":"        install_footers_callback(req, footers_callback)"}],"source_content_type":"text/x-python","patch_set":1,"id":"088b177d_940508a5","line":191,"range":{"start_line":191,"start_character":20,"end_line":191,"end_character":40},"updated":"2025-02-28 17:58:15.000000000","message":"because the proxy is going to add the footer anyway","commit_id":"89a2126d51d1ef73170b6c00e68dcc06bc95d4ce"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":110,"context_line":"        #     \u0027etag\u0027 key that is a callback to systags_callback:"},{"line_number":111,"context_line":"        #         systags \u003d {\u0027etag\u0027: EncInputWrapper.systags_callback}"},{"line_number":112,"context_line":"        #"},{"line_number":113,"context_line":"        # In EncInputWrapper.systags_callback:"},{"line_number":114,"context_line":"        #   - compose an override etag value from the stashed copy of systags"},{"line_number":115,"context_line":"        #         override_etag \u003d orig_etag; x\u003dy"},{"line_number":116,"context_line":"        #   - return the encrypted override_etag"}],"source_content_type":"text/x-python","patch_set":5,"id":"2f1db85b_6aec26f6","line":113,"updated":"2025-03-11 21:27:38.000000000","message":"some how I might have hoped/expected that the interface was \"just\" the `footers_callback` which then *could* update `systags` and result in the correct value being sent in the footers.  I\u0027m sure that\u0027s insufficient for some reason I don\u0027t understand.","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bb78dfd45a5b210ff2159d324401c9d613469614","unresolved":true,"context_lines":[{"line_number":110,"context_line":"        #     \u0027etag\u0027 key that is a callback to systags_callback:"},{"line_number":111,"context_line":"        #         systags \u003d {\u0027etag\u0027: EncInputWrapper.systags_callback}"},{"line_number":112,"context_line":"        #"},{"line_number":113,"context_line":"        # In EncInputWrapper.systags_callback:"},{"line_number":114,"context_line":"        #   - compose an override etag value from the stashed copy of systags"},{"line_number":115,"context_line":"        #         override_etag \u003d orig_etag; x\u003dy"},{"line_number":116,"context_line":"        #   - return the encrypted override_etag"}],"source_content_type":"text/x-python","patch_set":5,"id":"a7f7e447_11ac57d5","line":113,"in_reply_to":"2f1db85b_6aec26f6","updated":"2025-03-19 11:44:11.000000000","message":"discussed this in my patchset-level reply.\n\nOne further comment:\n\nIf mw was to update *its* view of systags in footers_callback, then we *must* ensure that the proxy app has visibility of the mw view (i.e. the same dict must be shared by mw and proxy app).  This was my initial thinking, and I think it may have got me locked into the \u0027inherit\u0027 pattern with ChainMap.\n\nHowever, if mw returns a value to a systags callback, then the proxy app could have a *copy* of the systags w.r.t. the mw\u0027s view, and we don\u0027t need to ensure that proxy and app share the same dict. That *may* allow me to ditch the \u0027inherit\u0027 pattern and just copy systags into subrequests, which could be more inutitive.","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":183,"context_line":"        # unencrypted systags to the override-etag footer. The new systags has"},{"line_number":184,"context_line":"        # just an etag key that points to our systags callback. Other"},{"line_number":185,"context_line":"        # middlewares that have any further interest in the original systags"},{"line_number":186,"context_line":"        # should have kept a reference to them."},{"line_number":187,"context_line":"        self.orig_systags \u003d dict(finalize_systags(self.env))"},{"line_number":188,"context_line":"        init_systags(self.env, {\u0027etag\u0027: self.systags_callback})"},{"line_number":189,"context_line":""}],"source_content_type":"text/x-python","patch_set":5,"id":"64393b85_3c019f2b","line":186,"updated":"2025-03-11 21:27:38.000000000","message":"does \"original\" in this context mean \"originally set BEFORE read_body/update_footers\" - or just ... like \"set by someone else\u0027s \"inner_callback\" during updater_footers before our systag_callback is called\" 😕","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bb78dfd45a5b210ff2159d324401c9d613469614","unresolved":true,"context_lines":[{"line_number":183,"context_line":"        # unencrypted systags to the override-etag footer. The new systags has"},{"line_number":184,"context_line":"        # just an etag key that points to our systags callback. Other"},{"line_number":185,"context_line":"        # middlewares that have any further interest in the original systags"},{"line_number":186,"context_line":"        # should have kept a reference to them."},{"line_number":187,"context_line":"        self.orig_systags \u003d dict(finalize_systags(self.env))"},{"line_number":188,"context_line":"        init_systags(self.env, {\u0027etag\u0027: self.systags_callback})"},{"line_number":189,"context_line":""}],"source_content_type":"text/x-python","patch_set":5,"id":"d438757a_34618708","line":186,"in_reply_to":"64393b85_3c019f2b","updated":"2025-03-19 11:44:11.000000000","message":"By this point every mw to left has had update_footers called, then line 187 will call any systags callback, so \u0027original\u0027 is what have before the encrypter finally does its thing.","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":185,"context_line":"        # middlewares that have any further interest in the original systags"},{"line_number":186,"context_line":"        # should have kept a reference to them."},{"line_number":187,"context_line":"        self.orig_systags \u003d dict(finalize_systags(self.env))"},{"line_number":188,"context_line":"        init_systags(self.env, {\u0027etag\u0027: self.systags_callback})"},{"line_number":189,"context_line":""},{"line_number":190,"context_line":"    def systags_callback(self, key, environ):"},{"line_number":191,"context_line":"        # we need a non-empty systags etag to construct a sensible"}],"source_content_type":"text/x-python","patch_set":5,"id":"4c8dd161_8a4ec279","line":188,"updated":"2025-03-11 21:27:38.000000000","message":"so but `init_systags` won\u0027t \"replace\" them with the provided dict - it has to be a `init_or_update(env, optional_seed_or_updates\u003dNone)` kind of interface?","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bb78dfd45a5b210ff2159d324401c9d613469614","unresolved":true,"context_lines":[{"line_number":185,"context_line":"        # middlewares that have any further interest in the original systags"},{"line_number":186,"context_line":"        # should have kept a reference to them."},{"line_number":187,"context_line":"        self.orig_systags \u003d dict(finalize_systags(self.env))"},{"line_number":188,"context_line":"        init_systags(self.env, {\u0027etag\u0027: self.systags_callback})"},{"line_number":189,"context_line":""},{"line_number":190,"context_line":"    def systags_callback(self, key, environ):"},{"line_number":191,"context_line":"        # we need a non-empty systags etag to construct a sensible"}],"source_content_type":"text/x-python","patch_set":5,"id":"18bb17e8_d13c6e40","line":188,"in_reply_to":"4c8dd161_8a4ec279","updated":"2025-03-19 11:44:11.000000000","message":"No, init_systags replaces the current map in the environ with a new one. All the proxy will now see is a single etag:callback item in the systags.\n\nI don\u0027t like this aspect, but I have to stop the proxy app doing it\u0027s \u0027normal\u0027 thing of applying all the plaintext systags to an override-etag header.\n\nAn alternative would be for the encrypter to construct the override-etag header and for that to prevent the proxy constructing it at all. But I didn\u0027t like that because it leaves the door open for *any* middleware to do the same i.e. send an override-etag header that causes the proxy to not generate one. We don\u0027t want any mw to be able to do that.\n\nI guess I could set a flag in the environ meaning \u0027final value override-etag header is set and I know what I am doing\u0027 that causes the proxy app to no-op??","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"}],"swift/common/middleware/s3api/controllers/multi_upload.py":[{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"a5dfbac26b9f07b7ced82f3e120f49028d403e51","unresolved":true,"context_lines":[{"line_number":745,"context_line":"                          content_type\u003d\u0027application/xml\u0027)"},{"line_number":746,"context_line":"        headers[s3_etag_header] \u003d s3_etag"},{"line_number":747,"context_line":"        # No need to set \u0027etag\u0027 in systags because SLO will populate it"},{"line_number":748,"context_line":"        get_systags(req.environ)[\u0027s3_etag\u0027] \u003d s3_etag"},{"line_number":749,"context_line":"        too_small_message \u003d (\u0027s3api requires that each segment be at least \u0027"},{"line_number":750,"context_line":"                             \u0027%d bytes\u0027 % self.conf.min_segment_size)"},{"line_number":751,"context_line":""}],"source_content_type":"text/x-python","patch_set":1,"id":"c5dd0a73_7df7f2a7","line":748,"updated":"2025-02-28 17:58:15.000000000","message":"I feel this is a good example of the simpler abstraction","commit_id":"89a2126d51d1ef73170b6c00e68dcc06bc95d4ce"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":745,"context_line":"        headers[s3_etag_header] \u003d s3_etag"},{"line_number":746,"context_line":"        # Leave base header value blank; SLO will populate"},{"line_number":747,"context_line":"        c_etag \u003d \u0027; s3_etag\u003d%s\u0027 % s3_etag"},{"line_number":748,"context_line":"        headers[get_container_update_override_key(\u0027etag\u0027)] \u003d c_etag"},{"line_number":749,"context_line":""},{"line_number":750,"context_line":"        too_small_message \u003d (\u0027s3api requires that each segment be at least \u0027"},{"line_number":751,"context_line":"                             \u0027%d bytes\u0027 % self.conf.min_segment_size)"}],"source_content_type":"text/x-python","patch_set":5,"id":"7f5a3cd6_c0ca3987","side":"PARENT","line":748,"updated":"2025-03-11 21:27:38.000000000","message":"this sort of reads like you could already \"just add systags\" w/o pre-calculating the *real* override-update-etag header *value*?","commit_id":"2f74376e7e42cfdcd5cc19b4e2f3c455d1fa2966"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bb78dfd45a5b210ff2159d324401c9d613469614","unresolved":true,"context_lines":[{"line_number":745,"context_line":"        headers[s3_etag_header] \u003d s3_etag"},{"line_number":746,"context_line":"        # Leave base header value blank; SLO will populate"},{"line_number":747,"context_line":"        c_etag \u003d \u0027; s3_etag\u003d%s\u0027 % s3_etag"},{"line_number":748,"context_line":"        headers[get_container_update_override_key(\u0027etag\u0027)] \u003d c_etag"},{"line_number":749,"context_line":""},{"line_number":750,"context_line":"        too_small_message \u003d (\u0027s3api requires that each segment be at least \u0027"},{"line_number":751,"context_line":"                             \u0027%d bytes\u0027 % self.conf.min_segment_size)"}],"source_content_type":"text/x-python","patch_set":5,"id":"92849b6a_62dde7e1","side":"PARENT","line":748,"in_reply_to":"7f5a3cd6_c0ca3987","updated":"2025-03-19 11:44:11.000000000","message":"you could but only when you\u0027re confident some other mw *is* going to provide the etag value","commit_id":"2f74376e7e42cfdcd5cc19b4e2f3c455d1fa2966"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":239,"context_line":"                \u0027X-Object-Sysmeta-Slo-Etag\u0027: \u0027\u0027,"},{"line_number":240,"context_line":"                \u0027X-Object-Sysmeta-Slo-Size\u0027: \u0027\u0027,"},{"line_number":241,"context_line":"            })"},{"line_number":242,"context_line":"            set_override_etag_header(req.headers, \u0027\u0027)"},{"line_number":243,"context_line":"        resp \u003d req.get_response(self.app)"},{"line_number":244,"context_line":""},{"line_number":245,"context_line":"        if \u0027X-Amz-Copy-Source\u0027 in req.headers:"}],"source_content_type":"text/x-python","patch_set":5,"id":"ab2835af_93ac2631","line":242,"updated":"2025-03-11 21:27:38.000000000","message":"hrm... so I guess this is sort of the read-path interface peaking it\u0027s head out and then showing up in the write path similar to the copy mw\n\nBut suggests that mw *could* still try to set an explicit override-etag-update header?  It\u0027s not clear how that would interact with systags.","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bb78dfd45a5b210ff2159d324401c9d613469614","unresolved":true,"context_lines":[{"line_number":239,"context_line":"                \u0027X-Object-Sysmeta-Slo-Etag\u0027: \u0027\u0027,"},{"line_number":240,"context_line":"                \u0027X-Object-Sysmeta-Slo-Size\u0027: \u0027\u0027,"},{"line_number":241,"context_line":"            })"},{"line_number":242,"context_line":"            set_override_etag_header(req.headers, \u0027\u0027)"},{"line_number":243,"context_line":"        resp \u003d req.get_response(self.app)"},{"line_number":244,"context_line":""},{"line_number":245,"context_line":"        if \u0027X-Amz-Copy-Source\u0027 in req.headers:"}],"source_content_type":"text/x-python","patch_set":5,"id":"3457ad60_c2a4d167","line":242,"in_reply_to":"ab2835af_93ac2631","updated":"2025-03-19 11:44:11.000000000","message":"I\u0027ve to\u0027d and fro\u0027d on how the proxy app deals with an override-etag header set by mw. I prefer that the proxy app would just ignore it - mw cannot use that header. But I have a feeling I allowed it in the end.\n\nThere is a comment that speaks to my uncertainty in ``systags.finalize_override_etag_header``:\n\n```\n    # No other actor should be setting an override-etag header, but if they\n    # have then include, but update, any systags in the existing header.\n    # TODO: I\u0027m unsure how to handle an existing override-etag header, and lean\n    # more towards always ignoring it and replacing with systags only version.\n    # But there\u0027s a probe test that asserts an internal client can send an\n    # override-etag header, and that perhaps seems reasonable??\n    # probe.test_object_async_update.TestUpdateOverrides.test_update_during_PUT\n    # We don\u0027t know which middleware set the override-etag header, and\n    # therefore don\u0027t know what the order of precedence should be if a key\n    # overlaps with one in systags. Dealing with an existing header *and*\n    # footer is even harder to reason about. Should we just outlaw it, and\n    # always replace an existing header here?\n```","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"}],"swift/common/middleware/slo.py":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":1577,"context_line":"            val, sep, params \u003d req.headers.get("},{"line_number":1578,"context_line":"                override_header, \u0027\u0027).partition(\u0027;\u0027)"},{"line_number":1579,"context_line":"            req.headers[override_header] \u003d \u0027%s; slo_etag\u003d%s\u0027 % ("},{"line_number":1580,"context_line":"                (val or req.headers[\u0027Etag\u0027]) + sep + params, slo_etag)"},{"line_number":1581,"context_line":""},{"line_number":1582,"context_line":"            env \u003d req.environ"},{"line_number":1583,"context_line":"            if not env.get(\u0027CONTENT_TYPE\u0027):"}],"source_content_type":"text/x-python","patch_set":5,"id":"3c427499_d5ea3619","side":"PARENT","line":1580,"updated":"2025-03-11 21:27:38.000000000","message":"the partition() and set + params is definitely harder to see what\u0027s going on than\n```\nsystags[\u0027etag\u0027] \u003d ...\nsystags[\u0027slo_etag\u0027] \u003d ...\n```","commit_id":"2f74376e7e42cfdcd5cc19b4e2f3c455d1fa2966"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":1574,"context_line":"            # Ensure container listings have both etags. However, if any"},{"line_number":1575,"context_line":"            # middleware to the left of us touched the base value, trust them."},{"line_number":1576,"context_line":"            systags \u003d get_systags(req.environ)"},{"line_number":1577,"context_line":"            systags[\u0027etag\u0027] \u003d systags.get(\u0027etag\u0027) or req.headers[\u0027Etag\u0027]"},{"line_number":1578,"context_line":"            systags[\u0027slo_etag\u0027] \u003d slo_etag"},{"line_number":1579,"context_line":""},{"line_number":1580,"context_line":"            env \u003d req.environ"}],"source_content_type":"text/x-python","patch_set":5,"id":"532ccc2f_928d4990","line":1577,"updated":"2025-03-11 21:27:38.000000000","message":"`systags.get(\u0027etag\u0027)` is definitely less characters than `get_container_update_override_key(\u0027etag\u0027)`\n\nis there even *other* `container_udpater_override_key`s besides etag!?\n\n\u003e However, if any middleware to the left of us touched the base value, trust them.\n\nIt\u0027s not super obvious why SLO wants `systag[\u0027etag\u0027] or headers[\u0027etag\u0027]` - but apparently that was the pre-existing behavior.","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bb78dfd45a5b210ff2159d324401c9d613469614","unresolved":true,"context_lines":[{"line_number":1574,"context_line":"            # Ensure container listings have both etags. However, if any"},{"line_number":1575,"context_line":"            # middleware to the left of us touched the base value, trust them."},{"line_number":1576,"context_line":"            systags \u003d get_systags(req.environ)"},{"line_number":1577,"context_line":"            systags[\u0027etag\u0027] \u003d systags.get(\u0027etag\u0027) or req.headers[\u0027Etag\u0027]"},{"line_number":1578,"context_line":"            systags[\u0027slo_etag\u0027] \u003d slo_etag"},{"line_number":1579,"context_line":""},{"line_number":1580,"context_line":"            env \u003d req.environ"}],"source_content_type":"text/x-python","patch_set":5,"id":"27e358ff_e379eae5","line":1577,"in_reply_to":"532ccc2f_928d4990","updated":"2025-03-19 11:44:11.000000000","message":"\u003e is there even other container_udpater_override_keys besides etag!?\n\n``size``","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"}],"swift/common/middleware/symlink.py":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":562,"context_line":"            sep, params \u003d response_headers[override_header].partition(\u0027;\u0027)[1:]"},{"line_number":563,"context_line":"            req.headers[override_header] \u003d MD5_OF_EMPTY_STRING + sep + params"},{"line_number":564,"context_line":""},{"line_number":565,"context_line":"        # It\u0027s troublesome that there\u0027s so much leakage with SLO"},{"line_number":566,"context_line":"        if \u0027X-Object-Sysmeta-Slo-Etag\u0027 in response_headers and \\"},{"line_number":567,"context_line":"                override_header not in req.headers:"},{"line_number":568,"context_line":"            req.headers[override_header] \u003d \u0027%s; slo_etag\u003d%s\u0027 % ("}],"source_content_type":"text/x-python","patch_set":5,"id":"ab47d127_8584fb2d","side":"PARENT","line":565,"updated":"2025-03-11 21:27:38.000000000","message":"maybe worth carrying this comment forward; I\u0027m not sure the implications","commit_id":"2f74376e7e42cfdcd5cc19b4e2f3c455d1fa2966"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":634,"context_line":"                req.headers[TGT_BYTES_SYSMETA_SYMLINK_HDR],"},{"line_number":635,"context_line":"            ])"},{"line_number":636,"context_line":"        req.headers[get_container_update_override_key(\u0027etag\u0027)] \u003d \\"},{"line_number":637,"context_line":"            \u0027; \u0027.join(etag_override)"},{"line_number":638,"context_line":""},{"line_number":639,"context_line":"        return self._app_call(req.environ)"},{"line_number":640,"context_line":""}],"source_content_type":"text/x-python","patch_set":5,"id":"ea3450f4_a4d425a9","side":"PARENT","line":637,"updated":"2025-03-11 21:27:38.000000000","message":"so this is an example where we just blew away any ... except maybe we pulled it out first and then just do \n\n```\n\u0027;\u0027.join([\"some-opaque; value\u003dwith; some\u003dkeys\", \"our\u003dnew\", \"key\u003dvalue\"])\n```\n\nI wonder if that\u0027s essentially equivalent to the partition pattern from slo... except I guess we *never* have to update the etag?","commit_id":"2f74376e7e42cfdcd5cc19b4e2f3c455d1fa2966"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bb78dfd45a5b210ff2159d324401c9d613469614","unresolved":true,"context_lines":[{"line_number":634,"context_line":"                req.headers[TGT_BYTES_SYSMETA_SYMLINK_HDR],"},{"line_number":635,"context_line":"            ])"},{"line_number":636,"context_line":"        req.headers[get_container_update_override_key(\u0027etag\u0027)] \u003d \\"},{"line_number":637,"context_line":"            \u0027; \u0027.join(etag_override)"},{"line_number":638,"context_line":""},{"line_number":639,"context_line":"        return self._app_call(req.environ)"},{"line_number":640,"context_line":""}],"source_content_type":"text/x-python","patch_set":5,"id":"9a543d92_8444e810","side":"PARENT","line":637,"in_reply_to":"ea3450f4_a4d425a9","updated":"2025-03-19 11:44:11.000000000","message":"this does retain any existing value once you squint at it enough, but it\u0027s the kind of mess that I\u0027d like to get away from in mw","commit_id":"2f74376e7e42cfdcd5cc19b4e2f3c455d1fa2966"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":565,"context_line":"            elif \u0027X-Object-Sysmeta-Slo-Etag\u0027 in response_headers:"},{"line_number":566,"context_line":"                req_systags[\u0027etag\u0027] \u003d MD5_OF_EMPTY_STRING"},{"line_number":567,"context_line":"                req_systags[\u0027slo_etag\u0027] \u003d response_headers["},{"line_number":568,"context_line":"                    \u0027X-Object-Sysmeta-Slo-Etag\u0027]"},{"line_number":569,"context_line":""},{"line_number":570,"context_line":"        req.headers[TGT_BYTES_SYSMETA_SYMLINK_HDR] \u003d ("},{"line_number":571,"context_line":"            response_headers.get(\u0027x-object-sysmeta-slo-size\u0027) or"}],"source_content_type":"text/x-python","patch_set":5,"id":"9e2cd087_a9f6e175","line":568,"updated":"2025-03-11 21:27:38.000000000","message":"does `s3_mpu_etag` not need this treatment because technically you\u0027re not \"supposed\" to use the swift api to create symlinks to s3api objects?","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bb78dfd45a5b210ff2159d324401c9d613469614","unresolved":true,"context_lines":[{"line_number":565,"context_line":"            elif \u0027X-Object-Sysmeta-Slo-Etag\u0027 in response_headers:"},{"line_number":566,"context_line":"                req_systags[\u0027etag\u0027] \u003d MD5_OF_EMPTY_STRING"},{"line_number":567,"context_line":"                req_systags[\u0027slo_etag\u0027] \u003d response_headers["},{"line_number":568,"context_line":"                    \u0027X-Object-Sysmeta-Slo-Etag\u0027]"},{"line_number":569,"context_line":""},{"line_number":570,"context_line":"        req.headers[TGT_BYTES_SYSMETA_SYMLINK_HDR] \u003d ("},{"line_number":571,"context_line":"            response_headers.get(\u0027x-object-sysmeta-slo-size\u0027) or"}],"source_content_type":"text/x-python","patch_set":5,"id":"0b8e430b_cf071bdd","line":568,"in_reply_to":"9e2cd087_a9f6e175","updated":"2025-03-19 11:44:11.000000000","message":"good question. versioning *does* create symlinks to s3 mpu\u0027s.","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"}],"swift/common/middleware/versioned_writes/object_versioning.py":[{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"a5dfbac26b9f07b7ced82f3e120f49028d403e51","unresolved":true,"context_lines":[{"line_number":287,"context_line":"        copy_header_subset(source_resp, put_req,"},{"line_number":288,"context_line":"                           lambda k: k.lower() !\u003d \u0027x-timestamp\u0027)"},{"line_number":289,"context_line":"        resp_systags \u003d parse_etag_override_header(source_resp.headers)"},{"line_number":290,"context_line":"        get_systags(put_req.environ).update(resp_systags)"},{"line_number":291,"context_line":"        put_req.environ[\u0027wsgi.input\u0027] \u003d FileLikeIter(source_resp.app_iter)"},{"line_number":292,"context_line":"        slo_size \u003d put_req.headers.get(\u0027X-Object-Sysmeta-Slo-Size\u0027)"},{"line_number":293,"context_line":"        if slo_size:"}],"source_content_type":"text/x-python","patch_set":1,"id":"89b0df75_170a8c67","line":290,"updated":"2025-02-28 17:58:15.000000000","message":"unlike copy middleware, the get response header takes precedence over the put req systags (??)","commit_id":"89a2126d51d1ef73170b6c00e68dcc06bc95d4ce"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":287,"context_line":"        copy_header_subset(source_resp, put_req,"},{"line_number":288,"context_line":"                           lambda k: k.lower() !\u003d \u0027x-timestamp\u0027)"},{"line_number":289,"context_line":"        resp_systags \u003d parse_override_etag_header(source_resp.headers)"},{"line_number":290,"context_line":"        inherit_systags(put_req.environ, resp_systags)"},{"line_number":291,"context_line":"        put_req.environ[\u0027wsgi.input\u0027] \u003d FileLikeIter(source_resp.app_iter)"},{"line_number":292,"context_line":"        slo_size \u003d put_req.headers.get(\u0027X-Object-Sysmeta-Slo-Size\u0027)"},{"line_number":293,"context_line":"        if slo_size:"}],"source_content_type":"text/x-python","patch_set":5,"id":"54f15952_a17ba492","line":290,"updated":"2025-03-11 21:27:38.000000000","message":"yeah idk, seems like `inherit_systags(environ, dict)` is different from `init_systags(environ, dict)`","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bb78dfd45a5b210ff2159d324401c9d613469614","unresolved":true,"context_lines":[{"line_number":287,"context_line":"        copy_header_subset(source_resp, put_req,"},{"line_number":288,"context_line":"                           lambda k: k.lower() !\u003d \u0027x-timestamp\u0027)"},{"line_number":289,"context_line":"        resp_systags \u003d parse_override_etag_header(source_resp.headers)"},{"line_number":290,"context_line":"        inherit_systags(put_req.environ, resp_systags)"},{"line_number":291,"context_line":"        put_req.environ[\u0027wsgi.input\u0027] \u003d FileLikeIter(source_resp.app_iter)"},{"line_number":292,"context_line":"        slo_size \u003d put_req.headers.get(\u0027X-Object-Sysmeta-Slo-Size\u0027)"},{"line_number":293,"context_line":"        if slo_size:"}],"source_content_type":"text/x-python","patch_set":5,"id":"5944edc6_e1fd7f96","line":290,"in_reply_to":"54f15952_a17ba492","updated":"2025-03-19 11:44:11.000000000","message":"I want to revisit inherit and consider whether a regular copy would be sufficient, and more intuitive.\n\nhmm, also I\u0027m not sure why the original req\u0027s systags are not copied to the put_req ... oh, this is the PUT of the null object that is being *copied* from the user bucket, so shouldn\u0027t be annotated with systags from the client req.","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bb78dfd45a5b210ff2159d324401c9d613469614","unresolved":true,"context_lines":[{"line_number":312,"context_line":"            req.environ, path\u003dwsgi_quote(put_path_info), method\u003d\u0027PUT\u0027,"},{"line_number":313,"context_line":"            headers\u003d{\u0027X-Backend-Allow-Reserved-Names\u0027: \u0027true\u0027},"},{"line_number":314,"context_line":"            swift_source\u003d\u0027OV\u0027)"},{"line_number":315,"context_line":"        inherit_systags(put_req.environ, get_systags(req.environ))"},{"line_number":316,"context_line":"        # move the client request body over"},{"line_number":317,"context_line":"        # note that the WSGI environ may be *further* manipulated; hold on to"},{"line_number":318,"context_line":"        # a reference to the byte counter so we can get the bytes_read"}],"source_content_type":"text/x-python","patch_set":5,"id":"7e20fcfe_1dbbd6fc","line":315,"updated":"2025-03-19 11:44:11.000000000","message":"again, is the \u0027inherit\u0027 pattern necessary - maybe this could just be a copy??","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"}],"swift/common/systags.py":[{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"a5dfbac26b9f07b7ced82f3e120f49028d403e51","unresolved":true,"context_lines":[{"line_number":44,"context_line":"is responsible for constructing this header, and will overwrite any existing"},{"line_number":45,"context_line":"value."},{"line_number":46,"context_line":""},{"line_number":47,"context_line":"# TODO: auto-etag computation in the proxy app is not yet implemented"},{"line_number":48,"context_line":"The proxy app needs an etag value in order to assemble the override-etag"},{"line_number":49,"context_line":"header. If the systags map does not have an etag value when the proxy starts to"},{"line_number":50,"context_line":"read the request body then the proxy app will compute the md5 hash of the body"}],"source_content_type":"text/x-python","patch_set":1,"id":"32765a35_45e3266d","line":47,"range":{"start_line":47,"start_character":50,"end_line":47,"end_character":69},"updated":"2025-02-28 17:58:15.000000000","message":"this will probably be a follow on patch","commit_id":"89a2126d51d1ef73170b6c00e68dcc06bc95d4ce"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"6d1aabef7ea97f5b928f600060c68b5a4a996057","unresolved":false,"context_lines":[{"line_number":44,"context_line":"is responsible for constructing this header, and will overwrite any existing"},{"line_number":45,"context_line":"value."},{"line_number":46,"context_line":""},{"line_number":47,"context_line":"# TODO: auto-etag computation in the proxy app is not yet implemented"},{"line_number":48,"context_line":"The proxy app needs an etag value in order to assemble the override-etag"},{"line_number":49,"context_line":"header. If the systags map does not have an etag value when the proxy starts to"},{"line_number":50,"context_line":"read the request body then the proxy app will compute the md5 hash of the body"}],"source_content_type":"text/x-python","patch_set":1,"id":"e1b21caa_edccc2bf","line":47,"range":{"start_line":47,"start_character":50,"end_line":47,"end_character":69},"in_reply_to":"32765a35_45e3266d","updated":"2025-03-11 16:38:48.000000000","message":"Done","commit_id":"89a2126d51d1ef73170b6c00e68dcc06bc95d4ce"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"a5dfbac26b9f07b7ced82f3e120f49028d403e51","unresolved":true,"context_lines":[{"line_number":64,"context_line":"by deferring to etag calculation to the proxy app."},{"line_number":65,"context_line":"# end of TODO"},{"line_number":66,"context_line":""},{"line_number":67,"context_line":"Middlewares may wish to prevent subsequent updates to the systags polluting"},{"line_number":68,"context_line":"the current view of systags. For example, if a middleware will re-use the same"},{"line_number":69,"context_line":"request then it may wish to preserve the current state of systags. The"},{"line_number":70,"context_line":"``overlay_systags()`` function installs a new systags map that uses ChainMap to"},{"line_number":71,"context_line":"chain a new map with the current map. Subsequent updates to the current view of"}],"source_content_type":"text/x-python","patch_set":1,"id":"d0cdcbe7_488eb41b","line":68,"range":{"start_line":67,"start_character":0,"end_line":68,"end_character":27},"updated":"2025-02-28 17:58:15.000000000","message":"I found this hard to explain (I also found it had to think about!!)...\n\nBut this is relevant to the proposed s3 checksum patch, particularly here https://review.opendev.org/c/openstack/swift/+/909801/comment/8f1e9573_2bfab181/ :\n\ns3api wants to calculate a checksum and then add it to systags during a footers callback. That\u0027s simple.\n\nBut if the request is a version write then we want the s3api systag to go to the version and to the symlink. So s3api sets it in systags during footers callback for the version write, and it is still in systags for the symlink PUT. Except we don\u0027t want *other* systags added further down the pipeline of handling the version write to also then go to the symlink PUT.\n\nSo s3api needs to be able to update *its* view of systags and have that be visible in the version write footer, but also have its view of systags remain a subset of the proxy app. Hence the ChainMap.","commit_id":"89a2126d51d1ef73170b6c00e68dcc06bc95d4ce"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"6d1aabef7ea97f5b928f600060c68b5a4a996057","unresolved":false,"context_lines":[{"line_number":64,"context_line":"by deferring to etag calculation to the proxy app."},{"line_number":65,"context_line":"# end of TODO"},{"line_number":66,"context_line":""},{"line_number":67,"context_line":"Middlewares may wish to prevent subsequent updates to the systags polluting"},{"line_number":68,"context_line":"the current view of systags. For example, if a middleware will re-use the same"},{"line_number":69,"context_line":"request then it may wish to preserve the current state of systags. The"},{"line_number":70,"context_line":"``overlay_systags()`` function installs a new systags map that uses ChainMap to"},{"line_number":71,"context_line":"chain a new map with the current map. Subsequent updates to the current view of"}],"source_content_type":"text/x-python","patch_set":1,"id":"60037649_b3bae0a2","line":68,"range":{"start_line":67,"start_character":0,"end_line":68,"end_character":27},"in_reply_to":"d0cdcbe7_488eb41b","updated":"2025-03-11 16:38:48.000000000","message":"Done","commit_id":"89a2126d51d1ef73170b6c00e68dcc06bc95d4ce"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"a5dfbac26b9f07b7ced82f3e120f49028d403e51","unresolved":true,"context_lines":[{"line_number":70,"context_line":"``overlay_systags()`` function installs a new systags map that uses ChainMap to"},{"line_number":71,"context_line":"chain a new map with the current map. Subsequent updates to the current view of"},{"line_number":72,"context_line":"systags will only mutate the new map, but any updates that earlier middlewares"},{"line_number":73,"context_line":"make to their view of systags *will* be reflected in the current view. This"},{"line_number":74,"context_line":"happens by default when a subrequest is created using"},{"line_number":75,"context_line":"``swift.common.wsgi.make_env``."},{"line_number":76,"context_line":""},{"line_number":77,"context_line":"Significantly, this property enables middlewares to update the final view of"},{"line_number":78,"context_line":"systags in an ``update_footers`` callback without having their own view of"}],"source_content_type":"text/x-python","patch_set":1,"id":"b2a9e3cb_a3074b1e","line":75,"range":{"start_line":73,"start_character":71,"end_line":75,"end_character":31},"updated":"2025-02-28 17:58:15.000000000","message":"Might be better if I had this statement up front.","commit_id":"89a2126d51d1ef73170b6c00e68dcc06bc95d4ce"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"6d1aabef7ea97f5b928f600060c68b5a4a996057","unresolved":false,"context_lines":[{"line_number":70,"context_line":"``overlay_systags()`` function installs a new systags map that uses ChainMap to"},{"line_number":71,"context_line":"chain a new map with the current map. Subsequent updates to the current view of"},{"line_number":72,"context_line":"systags will only mutate the new map, but any updates that earlier middlewares"},{"line_number":73,"context_line":"make to their view of systags *will* be reflected in the current view. This"},{"line_number":74,"context_line":"happens by default when a subrequest is created using"},{"line_number":75,"context_line":"``swift.common.wsgi.make_env``."},{"line_number":76,"context_line":""},{"line_number":77,"context_line":"Significantly, this property enables middlewares to update the final view of"},{"line_number":78,"context_line":"systags in an ``update_footers`` callback without having their own view of"}],"source_content_type":"text/x-python","patch_set":1,"id":"7863a4d6_f4357210","line":75,"range":{"start_line":73,"start_character":71,"end_line":75,"end_character":31},"in_reply_to":"b2a9e3cb_a3074b1e","updated":"2025-03-11 16:38:48.000000000","message":"Done","commit_id":"89a2126d51d1ef73170b6c00e68dcc06bc95d4ce"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"a5dfbac26b9f07b7ced82f3e120f49028d403e51","unresolved":true,"context_lines":[{"line_number":168,"context_line":"    return systags"},{"line_number":169,"context_line":""},{"line_number":170,"context_line":""},{"line_number":171,"context_line":"def get_etag_override_header(headers, default\u003dNone):"},{"line_number":172,"context_line":"    return headers.get(SYSTAGS_HEADER_KEY, default)"},{"line_number":173,"context_line":""},{"line_number":174,"context_line":""}],"source_content_type":"text/x-python","patch_set":1,"id":"613a722d_e61624e4","line":171,"range":{"start_line":171,"start_character":4,"end_line":171,"end_character":22},"updated":"2025-02-28 17:58:15.000000000","message":"oops, I had intended to standardise on ``_override_etag_`` rather than ``_etag_override_`` because the header name ends ``override-etag``","commit_id":"89a2126d51d1ef73170b6c00e68dcc06bc95d4ce"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"6d1aabef7ea97f5b928f600060c68b5a4a996057","unresolved":false,"context_lines":[{"line_number":168,"context_line":"    return systags"},{"line_number":169,"context_line":""},{"line_number":170,"context_line":""},{"line_number":171,"context_line":"def get_etag_override_header(headers, default\u003dNone):"},{"line_number":172,"context_line":"    return headers.get(SYSTAGS_HEADER_KEY, default)"},{"line_number":173,"context_line":""},{"line_number":174,"context_line":""}],"source_content_type":"text/x-python","patch_set":1,"id":"9f8f8306_63c13c81","line":171,"range":{"start_line":171,"start_character":4,"end_line":171,"end_character":22},"in_reply_to":"613a722d_e61624e4","updated":"2025-03-11 16:38:48.000000000","message":"Done","commit_id":"89a2126d51d1ef73170b6c00e68dcc06bc95d4ce"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":15,"context_line":"\"\"\""},{"line_number":16,"context_line":"systags provides an abstraction for middlewares to modify the object metadata"},{"line_number":17,"context_line":"that is stored in the container server and returned in a container listing."},{"line_number":18,"context_line":"This includes overriding the etag value returned in a container listing."},{"line_number":19,"context_line":""},{"line_number":20,"context_line":"systags are only effective for an object PUT request."},{"line_number":21,"context_line":""}],"source_content_type":"text/x-python","patch_set":5,"id":"f0bf9267_ece2950d","line":18,"updated":"2025-03-11 21:27:38.000000000","message":"I wonder if this interface could survive a refactor of the container-db such that systags were pulled out of the etag field; or at least communicated to new-enough container servers via something more abstracted at the container-interface-level from the x-etag header and the hash key in the listing.\n\nIt\u0027s clear from the handling that the `systags[\u0027etag\u0027]` is \"special\" while `systags[\u0027my_namespace_key\u0027]` is more \"pass-through\"\n\ne.g. slo maybe has to set update-override-content-length ???","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bb78dfd45a5b210ff2159d324401c9d613469614","unresolved":true,"context_lines":[{"line_number":15,"context_line":"\"\"\""},{"line_number":16,"context_line":"systags provides an abstraction for middlewares to modify the object metadata"},{"line_number":17,"context_line":"that is stored in the container server and returned in a container listing."},{"line_number":18,"context_line":"This includes overriding the etag value returned in a container listing."},{"line_number":19,"context_line":""},{"line_number":20,"context_line":"systags are only effective for an object PUT request."},{"line_number":21,"context_line":""}],"source_content_type":"text/x-python","patch_set":5,"id":"31167083_cd03d556","line":18,"in_reply_to":"f0bf9267_ece2950d","updated":"2025-03-19 11:44:11.000000000","message":"I can imagine the container server perhaps accepting arbitrary ``x-backend-systag-\u003ckey\u003e`` headers. Ditto the object server, which would forward them.\n\nI can imagine that those headers are actually named ``x-object-sysmeta-systag-\u003ckey\u003e`` so that the systags are stored with objects as independent pieces of sysmeta.\n\nI can imagine there being a sub-namespace ``x-object-sysmeta-systag-user-\u003ckey\u003e`` to support user-defined object tags (that, unlike user meta, also appear in the listing)\n\nSo far I\u0027m inclined to think that the etag systag key is only special in the abstract systags pattern in that it gets a default value in container listings if not set by mw. Unfortunately, the etag systag is also *required* in the current proxy-\u003eobject server implementation to act as a carrier for other key:value pairs in the override-etag header.","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":90,"context_line":"                                           subrequest_environ"},{"line_number":91,"context_line":"                                           ------------------"},{"line_number":92,"context_line":""},{"line_number":93,"context_line":"mw2_systags \u003d inherit_systags(             systags \u003d\u003d mw2_systags \u003d\u003d"},{"line_number":94,"context_line":"                  subreq.environ,            ChainMap("},{"line_number":95,"context_line":"                  mw1_systags)                 {},"},{"line_number":96,"context_line":"                                               mw1_systags \u003d\u003d"}],"source_content_type":"text/x-python","patch_set":5,"id":"7ec705a7_986c78c6","line":93,"updated":"2025-03-11 21:27:38.000000000","message":"is the use of `inherit_systags` by this hypothetical mw (vs `get_systags` or `init_systags`) a significant design choice WRT the mw\u0027s requirements on the life-cycle of the systags dict?","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":106,"context_line":"                                               mw1_systags \u003d\u003d"},{"line_number":107,"context_line":"                                                   ChainMap("},{"line_number":108,"context_line":"                                                     {\u0027mw1_tag_1a\u0027: \u00271a\u0027,"},{"line_number":109,"context_line":"                                                      \u0027mw1_tag_1b\u0027: callback}"},{"line_number":110,"context_line":"                                                   )"},{"line_number":111,"context_line":"                                             )"},{"line_number":112,"context_line":""}],"source_content_type":"text/x-python","patch_set":5,"id":"e9071882_5314d46b","line":109,"updated":"2025-03-11 21:27:38.000000000","message":"I don\u0027t think if I understand if it\u0027s *important* that mw1_systags dict does not have `mw2_tag_2a` key - or just a side-effect of the implementation...","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bb78dfd45a5b210ff2159d324401c9d613469614","unresolved":true,"context_lines":[{"line_number":106,"context_line":"                                               mw1_systags \u003d\u003d"},{"line_number":107,"context_line":"                                                   ChainMap("},{"line_number":108,"context_line":"                                                     {\u0027mw1_tag_1a\u0027: \u00271a\u0027,"},{"line_number":109,"context_line":"                                                      \u0027mw1_tag_1b\u0027: callback}"},{"line_number":110,"context_line":"                                                   )"},{"line_number":111,"context_line":"                                             )"},{"line_number":112,"context_line":""}],"source_content_type":"text/x-python","patch_set":5,"id":"932f095f_ca6fe001","line":109,"in_reply_to":"e9071882_5314d46b","updated":"2025-03-19 11:44:11.000000000","message":"It\u0027s intended, not just a side-effect.\n\nA somewhat contrived concrete example: if the subrequest is creating a symlink then mw2_tag_2a might be some symlink-related state that you do not want to be applied to the original request (contrived because you\u0027d probably have completed the original request *first*).\n\nHowever, as noted elsewhere, I want to revisit the inherit pattern.","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":125,"context_line":"                                                   )"},{"line_number":126,"context_line":"                                             )"},{"line_number":127,"context_line":"\"\"\""},{"line_number":128,"context_line":"from collections import ChainMap"},{"line_number":129,"context_line":""},{"line_number":130,"context_line":"from swift.common.request_helpers import install_footers_callback"},{"line_number":131,"context_line":"from swift.common.utils import compose_header, safe_parse_header"}],"source_content_type":"text/x-python","patch_set":5,"id":"93192683_71e3f67b","line":128,"updated":"2025-03-11 21:27:38.000000000","message":"https://docs.python.org/3/library/collections.html#collections.ChainMap\n\n\u003e A ChainMap groups multiple dicts or other mappings together to create a single, updateable view. If no maps are specified, a single empty dictionary is provided so that a new chain always has at least one mapping.","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":130,"context_line":"from swift.common.request_helpers import install_footers_callback"},{"line_number":131,"context_line":"from swift.common.utils import compose_header, safe_parse_header"},{"line_number":132,"context_line":""},{"line_number":133,"context_line":"SYSTAGS_HEADER_KEY \u003d \u0027X-Object-Sysmeta-Container-Update-Override-Etag\u0027"},{"line_number":134,"context_line":""},{"line_number":135,"context_line":""},{"line_number":136,"context_line":"def get_systags(environ):"}],"source_content_type":"text/x-python","patch_set":5,"id":"7541a1da_b481936c","line":133,"updated":"2025-03-11 21:27:38.000000000","message":"ok, so there WAS a `get_update_override_header(\u0027etag\u0027)` sort of helper (presumably to line up with SLO updating the size in the listing) but here we just break that out to it\u0027s own thing.","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":146,"context_line":"def init_systags(environ, systags\u003dNone):"},{"line_number":147,"context_line":"    \"\"\""},{"line_number":148,"context_line":"    Initialise a ChainMap of systags in the environ, completely replacing any"},{"line_number":149,"context_line":"    existing systags."},{"line_number":150,"context_line":""},{"line_number":151,"context_line":"    :param environ: a request environ."},{"line_number":152,"context_line":"    :params systags: a dict that will be used to seed the ChainMap"}],"source_content_type":"text/x-python","patch_set":5,"id":"08833834_273ac1b3","line":149,"updated":"2025-03-11 21:27:38.000000000","message":"\u003e completely replacing any existing systags.\n\noic, that\u0027s actually how I read it at first but assumed that would never be the correct/desirable behavior.  Perhaps I under-estimated how sure a mw might be that\u0027s it\u0027s systags are the only/bestest systags.","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bb78dfd45a5b210ff2159d324401c9d613469614","unresolved":true,"context_lines":[{"line_number":146,"context_line":"def init_systags(environ, systags\u003dNone):"},{"line_number":147,"context_line":"    \"\"\""},{"line_number":148,"context_line":"    Initialise a ChainMap of systags in the environ, completely replacing any"},{"line_number":149,"context_line":"    existing systags."},{"line_number":150,"context_line":""},{"line_number":151,"context_line":"    :param environ: a request environ."},{"line_number":152,"context_line":"    :params systags: a dict that will be used to seed the ChainMap"}],"source_content_type":"text/x-python","patch_set":5,"id":"48febb9f_c678ec0a","line":149,"in_reply_to":"08833834_273ac1b3","updated":"2025-03-19 11:44:11.000000000","message":"The encrypter is the opinionated mw!","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":176,"context_line":"    :return: a ChainMap of systags"},{"line_number":177,"context_line":"    \"\"\""},{"line_number":178,"context_line":"    systags \u003d get_systags(environ)"},{"line_number":179,"context_line":"    systags.maps.append(other_systags)"},{"line_number":180,"context_line":"    return systags"},{"line_number":181,"context_line":""},{"line_number":182,"context_line":""}],"source_content_type":"text/x-python","patch_set":5,"id":"3a3dca51_ac207dd0","line":179,"updated":"2025-03-11 21:27:38.000000000","message":"\u003e maps: A user updateable list of mappings\n\nok, well you go then.","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":181,"context_line":""},{"line_number":182,"context_line":""},{"line_number":183,"context_line":"def get_override_etag_header(headers, default\u003dNone):"},{"line_number":184,"context_line":"    return headers.get(SYSTAGS_HEADER_KEY, default)"},{"line_number":185,"context_line":""},{"line_number":186,"context_line":""},{"line_number":187,"context_line":"def set_override_etag_header(headers, value):"}],"source_content_type":"text/x-python","patch_set":5,"id":"563b6f9d_a1d1d936","line":184,"updated":"2025-03-11 21:27:38.000000000","message":"let it not be forgotten: the coupling of systags and etag-override-udpate header shall never be broken!","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":185,"context_line":""},{"line_number":186,"context_line":""},{"line_number":187,"context_line":"def set_override_etag_header(headers, value):"},{"line_number":188,"context_line":"    # only use this if you are confident you know what you\u0027re doing"},{"line_number":189,"context_line":"    headers[SYSTAGS_HEADER_KEY] \u003d value"},{"line_number":190,"context_line":""},{"line_number":191,"context_line":""}],"source_content_type":"text/x-python","patch_set":5,"id":"ebe6f7b1_0d7d257e","line":188,"updated":"2025-03-11 21:27:38.000000000","message":"ROFL, not even a doc-string - just a comment.  This is the way.","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bb78dfd45a5b210ff2159d324401c9d613469614","unresolved":true,"context_lines":[{"line_number":185,"context_line":""},{"line_number":186,"context_line":""},{"line_number":187,"context_line":"def set_override_etag_header(headers, value):"},{"line_number":188,"context_line":"    # only use this if you are confident you know what you\u0027re doing"},{"line_number":189,"context_line":"    headers[SYSTAGS_HEADER_KEY] \u003d value"},{"line_number":190,"context_line":""},{"line_number":191,"context_line":""}],"source_content_type":"text/x-python","patch_set":5,"id":"7d6d13b1_6146d06f","line":188,"in_reply_to":"ebe6f7b1_0d7d257e","updated":"2025-03-19 11:44:11.000000000","message":"I mean, that should be the first line of any programming language documentation ;-)\n\nMore seriously, I don\u0027t like that I seem to still need this helper - I guess the proxy app, but maybe I should just have it set the key in headers and remove the temptation for others to set the header?","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":212,"context_line":"def finalize_systags(environ):"},{"line_number":213,"context_line":"    systags \u003d get_systags(environ)"},{"line_number":214,"context_line":"    for k, v in systags.items():"},{"line_number":215,"context_line":"        if callable(v):"},{"line_number":216,"context_line":"            systags[k] \u003d v(k, environ)"},{"line_number":217,"context_line":"    return systags"},{"line_number":218,"context_line":""}],"source_content_type":"text/x-python","patch_set":5,"id":"4f35baf5_f344a456","line":215,"updated":"2025-03-11 21:27:38.000000000","message":"ok, so it\u0027s just the case that some systag values can be callbacks... and presumably the order of the callback (which could do ANYTHING?  presumably even update the environ to append new systags!?) has to do with ... the order of the chain map.  which has to do with get vs init vs inherit...\n\nI think it will take me some practice to feel like I get it!","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":219,"context_line":""},{"line_number":220,"context_line":"def finalize_override_etag_header(req):"},{"line_number":221,"context_line":"    \"\"\""},{"line_number":222,"context_line":"    Compose an override-etag header and add it to the given ``headers`` dict."},{"line_number":223,"context_line":""},{"line_number":224,"context_line":"    :param req: a swob.Request"},{"line_number":225,"context_line":"    \"\"\""}],"source_content_type":"text/x-python","patch_set":5,"id":"6bcaf742_4bddce07","line":222,"updated":"2025-03-11 21:27:38.000000000","message":"\u003e to the given ``headers`` dict.\n\ndo the req.headers?","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":229,"context_line":"    # TODO: I\u0027m unsure how to handle an existing override-etag header, and lean"},{"line_number":230,"context_line":"    # more towards always ignoring it and replacing with systags only version."},{"line_number":231,"context_line":"    # But there\u0027s a probe test that asserts an internal client can send an"},{"line_number":232,"context_line":"    # override-etag header, and that perhaps seems reasonable??"},{"line_number":233,"context_line":"    # probe.test_object_async_update.TestUpdateOverrides.test_update_during_PUT"},{"line_number":234,"context_line":"    # We don\u0027t know which middleware set the override-etag header, and"},{"line_number":235,"context_line":"    # therefore don\u0027t know what the order of precedence should be if a key"}],"source_content_type":"text/x-python","patch_set":5,"id":"1762f751_8c9e97d1","line":232,"updated":"2025-03-11 21:27:38.000000000","message":"I also would prefer we disallow mw setting X-Object-Sysmeta-Container-Update-Override-Etag directly; if IC needs it maybe we can allow a `X-Backend-Override-Object-Sysmeta-Container-Update-Override-Etag` or even `X-Backend-Update-Systags` or something... maybe we can do some archeology to better understand the requirements that created the probetest and just make sure they\u0027re easy to satisfy with systags?\n\nMaybe the reconclier or container-sync has to learn how to use systags to make moving slo manifests transparent.\n\nMaybe we can let IC send the the override-etag header but only IF there\u0027s no systags in the environment?","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":247,"context_line":"        final_systags \u003d parse_override_etag_header(headers)"},{"line_number":248,"context_line":"        systags \u003d finalize_systags(req.environ)"},{"line_number":249,"context_line":"        final_systags.update(systags)"},{"line_number":250,"context_line":"        compose_override_etag_header(headers, final_systags)"},{"line_number":251,"context_line":""},{"line_number":252,"context_line":"    if is_systags_finalized(req.environ):"},{"line_number":253,"context_line":"        _finalize_override_etag_header(req.headers)"}],"source_content_type":"text/x-python","patch_set":5,"id":"95d324bf_86eea00d","line":250,"updated":"2025-03-11 21:27:38.000000000","message":"i\u0027m not so sure this needs to be a closure or what that\u0027s buying us.","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":249,"context_line":"        final_systags.update(systags)"},{"line_number":250,"context_line":"        compose_override_etag_header(headers, final_systags)"},{"line_number":251,"context_line":""},{"line_number":252,"context_line":"    if is_systags_finalized(req.environ):"},{"line_number":253,"context_line":"        _finalize_override_etag_header(req.headers)"},{"line_number":254,"context_line":"    else:"},{"line_number":255,"context_line":"        # defer systag finalisation to footers callback"}],"source_content_type":"text/x-python","patch_set":5,"id":"f7b63476_e4ef2a33","line":252,"updated":"2025-03-11 21:27:38.000000000","message":"what do we gain from letting the *lack* of a callable trigger an early finalize?","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bb78dfd45a5b210ff2159d324401c9d613469614","unresolved":true,"context_lines":[{"line_number":249,"context_line":"        final_systags.update(systags)"},{"line_number":250,"context_line":"        compose_override_etag_header(headers, final_systags)"},{"line_number":251,"context_line":""},{"line_number":252,"context_line":"    if is_systags_finalized(req.environ):"},{"line_number":253,"context_line":"        _finalize_override_etag_header(req.headers)"},{"line_number":254,"context_line":"    else:"},{"line_number":255,"context_line":"        # defer systag finalisation to footers callback"}],"source_content_type":"text/x-python","patch_set":5,"id":"22332e0c_14557c78","line":252,"in_reply_to":"f7b63476_e4ef2a33","updated":"2025-03-19 11:44:11.000000000","message":"we don\u0027t install a footers callback, and if no-one else has then we don\u0027t do a put-with-trailers","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":253,"context_line":"        _finalize_override_etag_header(req.headers)"},{"line_number":254,"context_line":"    else:"},{"line_number":255,"context_line":"        # defer systag finalisation to footers callback"},{"line_number":256,"context_line":"        install_footers_callback(req, _finalize_override_etag_header)"}],"source_content_type":"text/x-python","patch_set":5,"id":"c4ec7cc2_e1dd3569","line":256,"updated":"2025-03-11 21:27:38.000000000","message":"so \"sometimes\" `finalize_override_etag_header` will \"just\" install a footer callback?\n\nI assume this has something to do with the situation when the object-controller isn\u0027t actually doing a mime-put-with-trailers.","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bb78dfd45a5b210ff2159d324401c9d613469614","unresolved":true,"context_lines":[{"line_number":253,"context_line":"        _finalize_override_etag_header(req.headers)"},{"line_number":254,"context_line":"    else:"},{"line_number":255,"context_line":"        # defer systag finalisation to footers callback"},{"line_number":256,"context_line":"        install_footers_callback(req, _finalize_override_etag_header)"}],"source_content_type":"text/x-python","patch_set":5,"id":"7d015e39_215ec3dc","line":256,"in_reply_to":"c4ec7cc2_e1dd3569","updated":"2025-03-19 11:44:11.000000000","message":"yes. If mw sets a systag key:callback then we\u0027re going to do a put-with-trailers.","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"}],"swift/common/utils/__init__.py":[{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"a5dfbac26b9f07b7ced82f3e120f49028d403e51","unresolved":true,"context_lines":[{"line_number":2553,"context_line":"        return line"},{"line_number":2554,"context_line":""},{"line_number":2555,"context_line":""},{"line_number":2556,"context_line":"class MD5Input(InputProxy):"},{"line_number":2557,"context_line":"    \"\"\""},{"line_number":2558,"context_line":"    wsgi.input wrapper to calculate the MD5 of the input as it\u0027s read."},{"line_number":2559,"context_line":"    \"\"\""}],"source_content_type":"text/x-python","patch_set":1,"id":"01d393f0_a0bfd813","line":2556,"updated":"2025-02-28 17:58:15.000000000","message":"this isn\u0027t used yet, but will be once proxy steps in to compute the override etag","commit_id":"89a2126d51d1ef73170b6c00e68dcc06bc95d4ce"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"a5dfbac26b9f07b7ced82f3e120f49028d403e51","unresolved":true,"context_lines":[{"line_number":2948,"context_line":"def safe_parse_header(value):"},{"line_number":2949,"context_line":"    if value is None:"},{"line_number":2950,"context_line":"        return None, {}"},{"line_number":2951,"context_line":"    return cgi.parse_header(value)"},{"line_number":2952,"context_line":""},{"line_number":2953,"context_line":""},{"line_number":2954,"context_line":"def compose_header(value, params):"}],"source_content_type":"text/x-python","patch_set":1,"id":"12729a35_1c15a3eb","line":2951,"range":{"start_line":2951,"start_character":11,"end_line":2951,"end_character":14},"updated":"2025-02-28 17:58:15.000000000","message":"I know cgi is obsolete, but let\u0027s address that once everywhere","commit_id":"89a2126d51d1ef73170b6c00e68dcc06bc95d4ce"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"6d1aabef7ea97f5b928f600060c68b5a4a996057","unresolved":false,"context_lines":[{"line_number":2948,"context_line":"def safe_parse_header(value):"},{"line_number":2949,"context_line":"    if value is None:"},{"line_number":2950,"context_line":"        return None, {}"},{"line_number":2951,"context_line":"    return cgi.parse_header(value)"},{"line_number":2952,"context_line":""},{"line_number":2953,"context_line":""},{"line_number":2954,"context_line":"def compose_header(value, params):"}],"source_content_type":"text/x-python","patch_set":1,"id":"cfbc2956_2d347b29","line":2951,"range":{"start_line":2951,"start_character":11,"end_line":2951,"end_character":14},"in_reply_to":"12729a35_1c15a3eb","updated":"2025-03-11 16:38:48.000000000","message":"Done","commit_id":"89a2126d51d1ef73170b6c00e68dcc06bc95d4ce"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":3037,"context_line":"        return None"},{"line_number":3038,"context_line":""},{"line_number":3039,"context_line":"    parts \u003d [value or \u0027\u0027] + [\u0027%s\u003d%s\u0027 % (k, v)"},{"line_number":3040,"context_line":"                             for k, v in params.items() if v is not None]"},{"line_number":3041,"context_line":"    return \u0027; \u0027.join(parts)"},{"line_number":3042,"context_line":""},{"line_number":3043,"context_line":""}],"source_content_type":"text/x-python","patch_set":5,"id":"315571ba_290d8d2e","line":3040,"updated":"2025-03-11 21:27:38.000000000","message":"so this again reads like it\u0027s ok to say:\n\n```\nX-Override-Etag-Update: ; just\u003dsome; key\u003dvalue\n```\n\nand some how someone who needs to will figure out to add those k/v paris onto the \"real\" ETag?","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bb78dfd45a5b210ff2159d324401c9d613469614","unresolved":true,"context_lines":[{"line_number":3037,"context_line":"        return None"},{"line_number":3038,"context_line":""},{"line_number":3039,"context_line":"    parts \u003d [value or \u0027\u0027] + [\u0027%s\u003d%s\u0027 % (k, v)"},{"line_number":3040,"context_line":"                             for k, v in params.items() if v is not None]"},{"line_number":3041,"context_line":"    return \u0027; \u0027.join(parts)"},{"line_number":3042,"context_line":""},{"line_number":3043,"context_line":""}],"source_content_type":"text/x-python","patch_set":5,"id":"18e869e7_d5317ab4","line":3040,"in_reply_to":"315571ba_290d8d2e","updated":"2025-03-19 11:44:11.000000000","message":"hmmm, so it may be that we now only need to use compose_header in the proxy in which case it IS the someone that WILL somehow have an etag value.\n\nNeed to revisit to see if this can be stricter.\n\nI probably wrote this in an early stage where my ambition was lower: just abstract the manipulations of the header. But now the ambition is for mw to never manipulate the header.","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"}],"swift/proxy/controllers/obj.py":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"73a8ab1a85c7a7190f0c3c23b0b924bcb706db65","unresolved":true,"context_lines":[{"line_number":860,"context_line":"        self._update_content_type(req)"},{"line_number":861,"context_line":""},{"line_number":862,"context_line":"        # TODO: write some tests!"},{"line_number":863,"context_line":"        finalize_override_etag_header(req)"},{"line_number":864,"context_line":"        req.ensure_x_timestamp()"},{"line_number":865,"context_line":""},{"line_number":866,"context_line":"        # check constraints on object name and request headers"}],"source_content_type":"text/x-python","patch_set":5,"id":"fbb47e6b_29b0509e","line":863,"updated":"2025-03-11 21:27:38.000000000","message":"but like... it might NOT finalize_override_etag_header... if the systags have callbacks?  why is that one method over in the new systags module?\n\nOh... maybe this *is* the latest mw can add systags (??!!) and the implementation detail of \"actually we send those in the footers\" only shows up IF you\u0027re using encryption.  And so somehow this interface ensures that there\u0027s no updating systags after you read the body... except maybe you can override the *value* of a systag after you read the body?  We actually need that for add\u0027l checksum support, right?","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"bb78dfd45a5b210ff2159d324401c9d613469614","unresolved":true,"context_lines":[{"line_number":860,"context_line":"        self._update_content_type(req)"},{"line_number":861,"context_line":""},{"line_number":862,"context_line":"        # TODO: write some tests!"},{"line_number":863,"context_line":"        finalize_override_etag_header(req)"},{"line_number":864,"context_line":"        req.ensure_x_timestamp()"},{"line_number":865,"context_line":""},{"line_number":866,"context_line":"        # check constraints on object name and request headers"}],"source_content_type":"text/x-python","patch_set":5,"id":"0791a4a4_a6d2fac7","line":863,"in_reply_to":"fbb47e6b_29b0509e","updated":"2025-03-19 11:44:11.000000000","message":"Yes you can update a systag after the body has been read (systag callbacks are only called after the body has been read. \n\nYes the S3api checksumming is going to want to do that (I think - just for the sake of ListParts??) by setting something like ``s3api_checksum_crc32:\u003ccallback\u003e``.\n\nIt\u0027s open for further debate as to whether mw can *add*, rather than *update*, a systag key during a callback. That is where the inherit vs copy pattern is significant.\n\nThe implementation is in systags module because that\u0027s where I am trying to encapsulate the override-etag header pattern. I could just as well name the function ``arrange_for_systags_to_go_to_backend``.","commit_id":"33a23064aea68c8327db9e4892499501e53c7f36"}]}