)]}'
{"/COMMIT_MSG":[{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"c0d026024d328da2abce22d0d7ce53d4dbbc3732","unresolved":true,"context_lines":[{"line_number":6,"context_line":""},{"line_number":7,"context_line":"Make sure that events with environment variable swift.source S3 is emitted."},{"line_number":8,"context_line":""},{"line_number":9,"context_line":"All middlewares set swift.source environment variable, most of the time we do not want to track these but the S3 api is an exception."},{"line_number":10,"context_line":""},{"line_number":11,"context_line":"Closes-Bug: #1903047"},{"line_number":12,"context_line":""}],"source_content_type":"text/x-gerrit-commit-message","patch_set":3,"id":"60f1da49_a0495f7c","line":9,"range":{"start_line":9,"start_character":106,"end_line":9,"end_character":132},"updated":"2023-10-31 04:30:50.000000000","message":"I wonder if there might be others -- `SLO`, for example. Have large object transfers been chronically under-reported by ceilometermiddleware?","commit_id":"14d324d9289f77fdf68e8e8a543d4f3caaf7f734"},{"author":{"_account_id":29541,"name":"Joakim Hindersson","email":"j.hindersson@gmail.com","username":"jhindersson"},"change_message_id":"d6b499c541147f2c31d52b5c30c4ad679c7636f0","unresolved":true,"context_lines":[{"line_number":6,"context_line":""},{"line_number":7,"context_line":"Make sure that events with environment variable swift.source S3 is emitted."},{"line_number":8,"context_line":""},{"line_number":9,"context_line":"All middlewares set swift.source environment variable, most of the time we do not want to track these but the S3 api is an exception."},{"line_number":10,"context_line":""},{"line_number":11,"context_line":"Closes-Bug: #1903047"},{"line_number":12,"context_line":""}],"source_content_type":"text/x-gerrit-commit-message","patch_set":3,"id":"73740726_06c9a9f0","line":9,"range":{"start_line":9,"start_character":106,"end_line":9,"end_character":132},"in_reply_to":"60f1da49_a0495f7c","updated":"2023-11-02 11:44:13.000000000","message":"That is a good question, let me give that a try in the lab and see what results we get with different middlewares.","commit_id":"14d324d9289f77fdf68e8e8a543d4f3caaf7f734"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"c0d026024d328da2abce22d0d7ce53d4dbbc3732","unresolved":true,"context_lines":[{"line_number":10,"context_line":""},{"line_number":11,"context_line":"Closes-Bug: #1903047"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"Depends-On: https://review.opendev.org/c/openstack/ceilometermiddleware/+/718085"},{"line_number":14,"context_line":"Depends-On: https://review.opendev.org/c/openstack/swift/+/718086"},{"line_number":15,"context_line":"Change-Id: Ifa3588440ce0c3099542dfffc16b585191d160da"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":3,"id":"eeaaeb31_8b903fbf","line":13,"updated":"2023-10-31 04:30:50.000000000","message":"Oh, interesting... looks like the recommended pipeline placement is to have ceilometer incredibly far right in the pipeline, [between the second `proxy-logging` and the `proxy-server` app](https://docs.openstack.org/ceilometer/latest/install/swift/install-swift-ubuntu.html#configure-object-storage-to-use-telemetry). It makes me wonder whether this `swift.backend_path` patch was actually needed.\n\nAlternatively, maybe the install guide could use an update in light of the other patch? Have we talked with Romain about how they\u0027re using ceilometermiddleware and what their pipeline placement is like?\n\nMy biggest question now is, what are we trying to measure? Client requests, or backend requests? There are arguments both ways, especially with `s3api` in the mix -- on the one hand, the server sometimes must make multiple backend requests for a single client request *because of a S3/Swift API mismatch*, but on the other, you have API calls like the `DeleteObjects` API where the client knowingly and explicitly *is trying to do several things at once*.","commit_id":"14d324d9289f77fdf68e8e8a543d4f3caaf7f734"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"4a26fdf9673607c7b4bfca445b52d11a1bcf3cce","unresolved":true,"context_lines":[{"line_number":10,"context_line":""},{"line_number":11,"context_line":"Closes-Bug: #1903047"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"Depends-On: https://review.opendev.org/c/openstack/ceilometermiddleware/+/718085"},{"line_number":14,"context_line":"Depends-On: https://review.opendev.org/c/openstack/swift/+/718086"},{"line_number":15,"context_line":"Change-Id: Ifa3588440ce0c3099542dfffc16b585191d160da"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":3,"id":"e9be4e72_e5ff1df7","line":13,"in_reply_to":"b289cbef_63dc9f5a","updated":"2023-11-03 04:56:15.000000000","message":"Measuring client requests for billing purposes makes a lot of sense -- tenant X made N GET requests this month, downloading Y bytes, and M PUT requests, uploading Z bytes.\n\nPipeline placement becomes a question of how you want to treat things like SLOs: there\u0027s a single client request, then as the proxy reads the (fairly small) response, it realizes that it actually needs to make a number of additional requests and stream the concatenation of those additional responses back to the client.\n\nFrom the client\u0027s perspective, there was just the one request for N bytes. The best way capture _that_ is to be early in the pipeline, left of SLO. From much of Swift\u0027s perspective, though, there were M requests, and the first response was some JSON while the rest *sum* to N bytes. We can capture that reality pretty well at the end of the pipeline, provided we look at requests with a `swift.source` of SLO.\n\nMy worry is that we currently get the worst of both: by being late in the pipeline *and* ignoring requests with a `swift.source`, I\u0027m pretty sure we _only_ record the GET for the SLO manifest, and miss the segment data.\n\nI suppose the good news, though, is that PUTs look fairly sane: the segments would each be uploaded (and captured) via their own requests, then the single PUT for the manifest. If we _were_ including SLO subrequests, we\u0027d also capture the segment HEADs SLO performs during manifest creation...\n\nSorry, measuring things in Swift is hard 😕","commit_id":"14d324d9289f77fdf68e8e8a543d4f3caaf7f734"},{"author":{"_account_id":29541,"name":"Joakim Hindersson","email":"j.hindersson@gmail.com","username":"jhindersson"},"change_message_id":"d6b499c541147f2c31d52b5c30c4ad679c7636f0","unresolved":true,"context_lines":[{"line_number":10,"context_line":""},{"line_number":11,"context_line":"Closes-Bug: #1903047"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"Depends-On: https://review.opendev.org/c/openstack/ceilometermiddleware/+/718085"},{"line_number":14,"context_line":"Depends-On: https://review.opendev.org/c/openstack/swift/+/718086"},{"line_number":15,"context_line":"Change-Id: Ifa3588440ce0c3099542dfffc16b585191d160da"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":3,"id":"b289cbef_63dc9f5a","line":13,"in_reply_to":"eeaaeb31_8b903fbf","updated":"2023-11-02 11:44:13.000000000","message":"Yes, that is also one thing i found when troubleshooting this, moving the ceilometermiddleware to the end of the pipeline gives it access to the necessary environment variables after all other middlewares are done. Have not tested without the swift.backend_path patch but that looks like a reasonable theory.\n\nI have not, this patch is based purely on my own testing. It would be interesting to hear how it would affect others though.\n\nThat is a fair question, talking from our own view we want to measure client requests (throughput and requests) for billing purposes (even though different middlewares can make multiple backend calls I think the actual client requests is cleanest to use). I suppose that there are other use cases where backend requests are more relevant though that would in my opinion probably work better to measure on the actual backends than in the incoming proxy pipeline.\n\nYour response made me think twice if there might be a more clean way of solving this. One thought is by putting the ceilometermiddleware in the internal-proxy pipeline though i have to admit that my understanding of that implementation is limited and am not really sure if that is what we want to measure.","commit_id":"14d324d9289f77fdf68e8e8a543d4f3caaf7f734"}],"/PATCHSET_LEVEL":[{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"c0d026024d328da2abce22d0d7ce53d4dbbc3732","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"2353efa3_13f25bed","updated":"2023-10-31 04:30:50.000000000","message":"Given the documented recommended pipeline placement, this seems reasonable -- and shouldn\u0027t hurt anything if you *aren\u0027t* using the recommended pipeline placement.\n\nThere\u0027s definitely some gaps in my understanding of what ceilometer is trying to measure and why -- but I think there may be more subrequests we want to record. Would it maybe make sense for us to have a `swift_source_allow_list` config option or something, with a note that the default may be subject to change?","commit_id":"14d324d9289f77fdf68e8e8a543d4f3caaf7f734"},{"author":{"_account_id":29541,"name":"Joakim Hindersson","email":"j.hindersson@gmail.com","username":"jhindersson"},"change_message_id":"c71f85094bc1f8392fea861cb2e8709f4a0b1ae9","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"216751eb_4f7ef8fa","updated":"2023-09-25 19:41:02.000000000","message":"recheck","commit_id":"14d324d9289f77fdf68e8e8a543d4f3caaf7f734"},{"author":{"_account_id":5202,"name":"Erno Kuvaja","email":"jokke@usr.fi","username":"jokke"},"change_message_id":"824a147f16d6e4c9968e442e4bae9885d1c837fd","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"2ef44bdc_dd49f30e","updated":"2023-09-28 09:58:22.000000000","message":"recheck devstack global venv","commit_id":"14d324d9289f77fdf68e8e8a543d4f3caaf7f734"},{"author":{"_account_id":29541,"name":"Joakim Hindersson","email":"j.hindersson@gmail.com","username":"jhindersson"},"change_message_id":"637b4bafb364f2df19cdfc4053eb5da9b19fd610","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"e3757428_e0661442","in_reply_to":"2353efa3_13f25bed","updated":"2023-11-02 11:57:55.000000000","message":"No, placing it anywhere else would only make it miss some information that it can only acquire after other middlewares are done processing.\n\nA config option makes a lot of sense as it gives users flexibility now and for future middlewares. I\u0027ll see what I can come up with.","commit_id":"14d324d9289f77fdf68e8e8a543d4f3caaf7f734"}]}