)]}'
{"/PATCHSET_LEVEL":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"6e359a0687e99c08a25a06babfa6f4d629c2a5a3","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"afb94b92_62887596","updated":"2026-05-13 20:29:51.000000000","message":"I needed to get this wip off my machine:\n\n988537: wip: this more reliable; is it better? | https://review.opendev.org/c/openstack/swift/+/988537\n\nI haven\u0027t really had a chance to look carefully at each diff hunk and decide if it\u0027s that final form.","commit_id":"ca839dbed05c77a0861642021e7064b49bd34bce"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"e59dc89a70a310e5e3191733722c7a6829c0294b","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"9c2098a6_a3f0950d","updated":"2026-05-13 09:16:43.000000000","message":"I wasn\u0027t able to get a diff that I was happy with and I\u0027m kind of exhausted; but I think I have an idea for a deterministic mechanism to achieve the \"both success but mixed metadata\" result.  My compressed context if you want to try to start fresh is:\n\n```\n› We’re working in OpenStack Swift, file:\n\n      test/probe/test_timestamp_collision.py\n\n    Read ../CLAUDE.md first for test-running guidance. In this environment, probe tests should be run from the host, e.g.:\n\n      vagrant ssh -c \"pytest swift/test/probe/test_timestamp_collision.py::TestECCollision::test_overlap_data_write_streams_both_succeed_one_durable_set\"\n\n    Context:\n    We’re investigating the EC same-timestamp collision probe test:\n\n      TestECCollision.test_overlap_data_write_streams_both_succeed_one_durable_set\n\n    The EC policy in this vSAIO is 4+2, with 8 disks total. For Swift EC quorum, success requires ndata + 1 \u003d 5 object-server successes, not merely ndata \u003d 4.\n\n    Starting point:\n    Commit 011f6dbe5e had a FragZipper-style helper and the test used something like a 5/1 write split with wait_for_loser\u003dFalse. The test assertion:\n\n      final durable metadata counts should be {1, 5}\n\n    is not reliable. It can end with final durable metadata shape {6}. That is not random flakiness; it is a real possible outcome.\n\n    Important observed facts:\n    - end_of_object_data only proves the proxy sent EOF/footer for a fragment. It does not prove the request has entered/finished commit.\n    - A request can send all six fragment footers and still not have durable files on disk yet.\n    - If the other request is parked immediately before its next end_of_object_data call, polling the filesystem can observe the first request’s durable files appear. So the first request is not stuck waiting for the other request to send another\n    footer.\n    - Once same-timestamp fragments are durable, the other request may write same-timestamp non-durable files “behind” those durable files. Those post-commit writes are not the same as pre-commit linkat collisions.\n    - The existing FragZipper win/loss-style debug is misleading if interpreted as object-server success/failure. “Sent second for a frag” is only a pre-commit collision if no durable exists yet. After durable exists, “sent second” can be a\n    successful write-behind.\n    - Final {6} happens when the later request gets an early foothold and then all remaining writes happen behind the committed durable set, letting it sweep the final durable files.\n    - A request-id-only schedule like [1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0] does not guarantee whether the first two footers hit the same fragment. If [1x, 0x], req0 has an observed pre-commit collision. If [1x, 0y], req0 has an early foothold\n    instead.\n\n    The reliable target shape:\n    We want a deterministic “both succeed, final durable set mixed” case.\n\n    The useful sequence is:\n\n    1. Start both same-timestamp uploads.\n    2. Alternate pre-commit fragment footers until the trailing request has one observed same-timestamp overlap/collision.\n       - This is the important rule: keep alternating until the trailing request has “tarnish.”\n       - If the first pair is [1x, 0x], we already have it.\n       - If the first pair is [1x, 0y], keep alternating until the trailing request sends a frag already sent by the leading request.\n    3. Once the trailing request has exactly one pre-commit overlap, let the leading request finish sending enough/all remaining footers and commit.\n    4. Before allowing the trailing request to send its remaining footers, wait until durable files from the leading request are actually observed on disk. This “clears the landing pad.”\n    5. Let the trailing request send its remaining footers behind the durable set and commit.\n    6. Expected result:\n       - both requests return 201\n       - final durable metadata is mixed, counts {1, 5}\n\n    Why this works:\n    - The trailing request has exactly one pre-commit overlap, so it cannot sweep all six.\n    - After durable is observed, its remaining writes can succeed behind the durable files.\n    - With EC 4+2 quorum, it gets 1 pre-commit overlap/loss plus 5 total successes, or equivalently enough successful writes to reach quorum while still leaving one durable fragment from the leading request.\n    - The final durable set should therefore be mixed rather than all one request.\n\n    Constraints:\n    - Aim for the minimum diff.\n    - Do not add broad/confusing APIs like winners_per_req/losers_per_req/wait_for_loser/confirm_loser_before_commit unless there is a very clean model.\n    - Do not patch extra proxy methods such as _get_put_responses, _get_conn_response, _determine_chunk_destinations, or commit-response collection.\n    - Prefer keeping the existing FragZipper/end_of_object_data approach, but make the coordination observation-driven where needed.\n    - It is OK for the helper to observe which fragment indexes have been sent and to wait for durable files on disk at the latest safe point before the trailing request sends its next footer.\n    - Keep comments precise: do not claim end_of_object_data means commit. Say only what the test actually observes.\n    - If the implementation cannot be made deterministic with the existing hooks, explain concretely why the missing hook must be closer to commit/object-server response handling.\n\n    Please inspect the current file before editing. The goal is either:\n    - a crisp, minimum-diff implementation of the “alternate until trailing overlap, wait for durable, then write behind” schedule, or\n    - a concrete explanation of why the current hooks cannot express that deterministically.\n```","commit_id":"ca839dbed05c77a0861642021e7064b49bd34bce"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"1cb29497001378677c993ec6cd811f6986f08678","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"17437d6e_fbc4c5f1","updated":"2026-02-27 21:21:28.000000000","message":"i\u0027m not sure these will make sense post v2 jitter ts - if so I guess we have to find a way to make them more stable/forgiving","commit_id":"ca839dbed05c77a0861642021e7064b49bd34bce"}],"test/probe/test_timestamp_collision.py":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"1cb29497001378677c993ec6cd811f6986f08678","unresolved":true,"context_lines":[{"line_number":512,"context_line":"        key_to_files \u003d self._map_metadata_keys(file_to_meta, keys)"},{"line_number":513,"context_line":"        # TODO: this assertion isn\u0027t reliable: sometimes the second request"},{"line_number":514,"context_line":"        #   manages to completely replace the first request\u0027s frags"},{"line_number":515,"context_line":"        self.assertEqual({1, 5}, set(len(v) for v in key_to_files.values()))"},{"line_number":516,"context_line":"        # but mutually exclusive w.r.t. files"},{"line_number":517,"context_line":"        self.assertFalse("},{"line_number":518,"context_line":"            key_to_files[\u0027foo\u0027].intersection(key_to_files[\u0027bar\u0027]))"}],"source_content_type":"text/x-python","patch_set":3,"id":"4d3c4b25_43ad70ae","line":515,"updated":"2026-02-27 21:21:28.000000000","message":"```\nFAILED swift/test/probe/test_timestamp_collision.py::TestECCollision::test_overlap_data_write_streams_both_succeed_one_durable_set - AssertionError: Items in the first set but not the second:\n1\n5\nItems in the second set but not the first:\n6\n```","commit_id":"ca839dbed05c77a0861642021e7064b49bd34bce"}]}