)]}'
{"/COMMIT_MSG":[{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"5e8dc53290c2acfa3c5907c39e320ffab67afe69","unresolved":true,"context_lines":[{"line_number":14,"context_line":""},{"line_number":15,"context_line":"The harness lives under docker/matrix-eavesdrop/test/ and is kept"},{"line_number":16,"context_line":"separate from the deployment so reviewers can decide how (or whether)"},{"line_number":17,"context_line":"to wire it into the gate."},{"line_number":18,"context_line":""},{"line_number":19,"context_line":"Change-Id: I9a3c0e94fd41af3538159e1bc000608372afc981"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"10d25d84_47444075","line":17,"updated":"2026-07-01 23:28:01.000000000","message":"I think the way we would typically try to approach this is to have the system-config-run job for the service use a `run_test_playbook` playbook (for example https://opendev.org/opendev/system-config/src/branch/master/zuul.d/system-config-run.yaml#L315 ) To set things up (think deploying the synapse homeserver etc). Then have testinfra test cases drive assertions against that system.\n\nIn thise case I think you want to update the system-config-run-eavesdrop job here: https://opendev.org/opendev/system-config/src/branch/master/zuul.d/system-config-run.yaml#L164-L217 Then update the testinfra test cases here: https://opendev.org/opendev/system-config/src/branch/master/testinfra/test_eavesdrop.py","commit_id":"b3d4b378aac67d332774d15d746f2d69248c6e2a"},{"author":{"_account_id":10089,"name":"Anil Shashikumar Belur","email":"abelur@linuxfoundation.org","username":"askb"},"change_message_id":"ae88e73dd457c1f7d172ea0a3b09c546a297f3c6","unresolved":true,"context_lines":[{"line_number":14,"context_line":""},{"line_number":15,"context_line":"The harness lives under docker/matrix-eavesdrop/test/ and is kept"},{"line_number":16,"context_line":"separate from the deployment so reviewers can decide how (or whether)"},{"line_number":17,"context_line":"to wire it into the gate."},{"line_number":18,"context_line":""},{"line_number":19,"context_line":"Change-Id: I9a3c0e94fd41af3538159e1bc000608372afc981"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"d6c9fff1_1b54a4ef","line":17,"in_reply_to":"10d25d84_47444075","updated":"2026-07-02 23:08:08.000000000","message":"That approach makes sense and I\u0027d rather follow the established pattern than maintain a bespoke harness. Happy to rework this to extend the existing `system-config-run-eavesdrop` job (zuul.d/system-config-run.yaml) with a `run_test_playbook` that drives a meeting end-to-end, plus assertions in testinfra/test_eavesdrop.py - modelled on the lists3 example you linked. This also lines up with your tox L35 point on the previous change, so I\u0027ll align the unit-test job and the functional test the same way.\n\nBefore I rework it, a couple of specifics so I build it the way you\u0027d want:\n\n  1. For the meeting scenario, is the preference to drive it via a small\n     playbook that talks to the deployed bot/Synapse in the gate node,\n     then assert the rendered minutes exist under the meetings dir via\n     testinfra? Or is there an existing matrix/synapse functional example\n     closer than lists3 I should mirror?\n  2. Should the standalone docker/matrix-eavesdrop/test/ harness be\n     dropped entirely in favour of the system-config-run approach, or kept\n     as a dev-only helper (not gated)?\n  3. Same question for the unit tests (tox L35): a dedicated\n     [testenv:matrix-eavesdrop] + a Zuul job with file matchers on\n     docker/matrix-eavesdrop/** - is there an example job you\u0027d point me\n     at, or should I follow one of the existing per-service test jobs?\n\nI\u0027ll hold the rework on this change until we settle the shape; the lower four changes are independent of it.","commit_id":"b3d4b378aac67d332774d15d746f2d69248c6e2a"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"fa3615975c794d8204ca32dc4e30bf39fd40b8bc","unresolved":true,"context_lines":[{"line_number":14,"context_line":""},{"line_number":15,"context_line":"The harness lives under docker/matrix-eavesdrop/test/ and is kept"},{"line_number":16,"context_line":"separate from the deployment so reviewers can decide how (or whether)"},{"line_number":17,"context_line":"to wire it into the gate."},{"line_number":18,"context_line":""},{"line_number":19,"context_line":"Change-Id: I9a3c0e94fd41af3538159e1bc000608372afc981"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"fbc6d538_40026d56","line":17,"in_reply_to":"d6c9fff1_1b54a4ef","updated":"2026-07-06 04:18:03.000000000","message":"Note I answered these out of order below.\n\n1. I am not aware of any existing matrix/synapse functional example. I think a test playbook would be a good way to deploy a test synapse server. Then I think it is up to you to decide if it is easier to drive the example meeting via that ansible test playbook or via the testinfra tooling framework. I think asserting state about the end result is probably best in testinfra\n\n3. I think that the standalone unittest framework is good to have as something tox runnable as it makes local dev easier. Basically people can make changes and run that test suite before pushing and then let Zuul run the more complicated functional tests.\n\n2. We probably don\u0027t need the scripted version of the functional test if we have the ansible and testinfra version. From experience it seems unlikely people would run those manually and instead rely on the CI system.\n\nI tried to provide examples of refactoring the tox config and adding a new tox job for running the unittests in a previous change in this stack. We can probably continue that conversation there.","commit_id":"b3d4b378aac67d332774d15d746f2d69248c6e2a"},{"author":{"_account_id":10089,"name":"Anil Shashikumar Belur","email":"abelur@linuxfoundation.org","username":"askb"},"change_message_id":"f024bfbcc6d6d238facc8f1ffcbe846fd4b50209","unresolved":false,"context_lines":[{"line_number":14,"context_line":""},{"line_number":15,"context_line":"The harness lives under docker/matrix-eavesdrop/test/ and is kept"},{"line_number":16,"context_line":"separate from the deployment so reviewers can decide how (or whether)"},{"line_number":17,"context_line":"to wire it into the gate."},{"line_number":18,"context_line":""},{"line_number":19,"context_line":"Change-Id: I9a3c0e94fd41af3538159e1bc000608372afc981"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"f7660e58_c1cdfecd","line":17,"in_reply_to":"fbc6d538_40026d56","updated":"2026-07-07 05:53:22.000000000","message":"thanks - Marked as resolved.","commit_id":"b3d4b378aac67d332774d15d746f2d69248c6e2a"}],"/PATCHSET_LEVEL":[{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"5e8dc53290c2acfa3c5907c39e320ffab67afe69","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"486a613e_6495d10e","updated":"2026-07-01 23:28:01.000000000","message":"Note I didn\u0027t dig too much into the details of the change with my initial review. Instead I tried to illustrate how we can expand the existing functional testing for irc and matrix things to run these functional tests. I think that will probably be easier for others to review as it should follow the general shape we use a bit more.","commit_id":"b3d4b378aac67d332774d15d746f2d69248c6e2a"},{"author":{"_account_id":10089,"name":"Anil Shashikumar Belur","email":"abelur@linuxfoundation.org","username":"askb"},"change_message_id":"e8fc86f307d957c8175bbc64006a8d6f918b08d7","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":6,"id":"16799223_a1b8f56a","updated":"2026-07-08 13:50:14.000000000","message":"recheck","commit_id":"c17fdc5297358a8da1a7c139103aac8e88e457bd"}],"playbooks/zuul/test-eavesdrop.yaml":[{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"28c16f38e8f0bc2ed69e5ef9ace65038aabd2416","unresolved":true,"context_lines":[{"line_number":53,"context_line":"          -e SYNAPSE_SERVER_NAME\u003d{{ synapse_server_name }}"},{"line_number":54,"context_line":"          -e SYNAPSE_REPORT_STATS\u003dno"},{"line_number":55,"context_line":"          -v {{ synapse_data_dir }}:/data"},{"line_number":56,"context_line":"          {{ synapse_image }} generate"},{"line_number":57,"context_line":"        creates: \"{{ synapse_data_dir }}/homeserver.yaml\""},{"line_number":58,"context_line":""},{"line_number":59,"context_line":"    - name: Enable shared-secret registration and relax rate limits"}],"source_content_type":"text/x-yaml","patch_set":3,"id":"efe8592c_98aed211","line":56,"updated":"2026-07-07 02:21:28.000000000","message":"Does this server by default try to join the global federated network? If not what prevents it from doing so? I think we want to prevent it from attempting to federate with anything to keep the test environment as simple and self contained as possible.","commit_id":"b997ba6c15bfc9a19a7167537e026f74ac32776a"},{"author":{"_account_id":10089,"name":"Anil Shashikumar Belur","email":"abelur@linuxfoundation.org","username":"askb"},"change_message_id":"39c2e2d0033e7c5a1b29c3d9ccb7ac84dce5fdc1","unresolved":false,"context_lines":[{"line_number":53,"context_line":"          -e SYNAPSE_SERVER_NAME\u003d{{ synapse_server_name }}"},{"line_number":54,"context_line":"          -e SYNAPSE_REPORT_STATS\u003dno"},{"line_number":55,"context_line":"          -v {{ synapse_data_dir }}:/data"},{"line_number":56,"context_line":"          {{ synapse_image }} generate"},{"line_number":57,"context_line":"        creates: \"{{ synapse_data_dir }}/homeserver.yaml\""},{"line_number":58,"context_line":""},{"line_number":59,"context_line":"    - name: Enable shared-secret registration and relax rate limits"}],"source_content_type":"text/x-yaml","patch_set":3,"id":"969fb4b4_18d5d792","line":56,"in_reply_to":"efe8592c_98aed211","updated":"2026-07-07 04:15:24.000000000","message":"Done in PS4. The homeserver.yaml blockinfile now sets `federation_domain_whitelist: []`, which disables outbound federation entirely, so the test homeserver is fully self-contained.","commit_id":"b997ba6c15bfc9a19a7167537e026f74ac32776a"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"28c16f38e8f0bc2ed69e5ef9ace65038aabd2416","unresolved":true,"context_lines":[{"line_number":162,"context_line":"            - id: \"{{ test_room_alias }}\""},{"line_number":163,"context_line":"              path: \"opendev-meeting-test\""},{"line_number":164,"context_line":"              meetings: true"},{"line_number":165,"context_line":"              meeting_name: opendev_test"},{"line_number":166,"context_line":""},{"line_number":167,"context_line":"    - name: Restart the bot so it connects to the test homeserver and joins"},{"line_number":168,"context_line":"      command:"}],"source_content_type":"text/x-yaml","patch_set":3,"id":"5164aa3f_6cf0639f","line":165,"updated":"2026-07-07 02:21:28.000000000","message":"We should have the regular deployment configure the bot to talk to this matrix server via the faked/test only ansible vars located at https://opendev.org/opendev/system-config/src/branch/master/playbooks/zuul/templates/group_vars/eavesdrop.yaml.j2. That way we exercise the deployment system works more thoroughly and don\u0027t need to rely on a one off task here to do that.\n\nNote you probably need to keep the bot restart below to have the bot try to connect after we set things up.","commit_id":"b997ba6c15bfc9a19a7167537e026f74ac32776a"},{"author":{"_account_id":10089,"name":"Anil Shashikumar Belur","email":"abelur@linuxfoundation.org","username":"askb"},"change_message_id":"39c2e2d0033e7c5a1b29c3d9ccb7ac84dce5fdc1","unresolved":false,"context_lines":[{"line_number":162,"context_line":"            - id: \"{{ test_room_alias }}\""},{"line_number":163,"context_line":"              path: \"opendev-meeting-test\""},{"line_number":164,"context_line":"              meetings: true"},{"line_number":165,"context_line":"              meeting_name: opendev_test"},{"line_number":166,"context_line":""},{"line_number":167,"context_line":"    - name: Restart the bot so it connects to the test homeserver and joins"},{"line_number":168,"context_line":"      command:"}],"source_content_type":"text/x-yaml","patch_set":3,"id":"eeb83704_63943fbd","line":165,"in_reply_to":"5164aa3f_6cf0639f","updated":"2026-07-07 04:15:24.000000000","message":"Good call — reworked in PS4. The one-off `copy` task is gone. The bot is now parametrized in the role template (playbooks/roles/matrix-eavesdrop/\ntemplates/config.yaml: homeserver/user_id via `| default(...)`, room list via\n`matrix_eavesdrop_rooms`), and the gate supplies test values in\nplaybooks/zuul/templates/group_vars/eavesdrop.yaml.j2\n(matrix_eavesdrop_homeserver: http://localhost:8008 and a single throwaway\nroom with meetings enabled). So the normal service-eavesdrop deploy configures\nthe bot; the test playbook only stands up Synapse + the room and then restarts\nthe bot so it connects (Synapse/room don\u0027t exist at deploy time). Production\ndefaults are unchanged when the override vars are absent.","commit_id":"b997ba6c15bfc9a19a7167537e026f74ac32776a"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"28c16f38e8f0bc2ed69e5ef9ace65038aabd2416","unresolved":true,"context_lines":[{"line_number":214,"context_line":"      loop:"},{"line_number":215,"context_line":"        - {who: alice, body: \"#startmeeting OpenDev Test Meeting\"}"},{"line_number":216,"context_line":"        - {who: alice, body: \"#topic Rollout plan\"}"},{"line_number":217,"context_line":"        - {who: bob, body: \"#info we are go for launch\"}"},{"line_number":218,"context_line":"        - {who: alice, body: \"#action bob to verify minutes\"}"},{"line_number":219,"context_line":"        - {who: bob, body: \"waves at everyone\", msgtype: m.emote}"},{"line_number":220,"context_line":"        - {who: alice, body: \"#agreed this works\"}"}],"source_content_type":"text/x-yaml","patch_set":3,"id":"47919c9a_a161ac9b","line":217,"updated":"2026-07-07 02:21:28.000000000","message":"Request #3 appears to fail in this loop. You can see the logs for that at https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_1b7/openstack/1b7a0d2c3c494ceb9d23edb7cc6599a0/bridge99.opendev.org/ara-report/results/458.html To get there you can follow the zuul job link -\u003e logs -\u003e bridge99 -\u003e ara -\u003e raw -\u003e then navigate ara to the failing task which gets you the link above.\n\nThe error message says bob is not in the room so we may need some additional checks above?","commit_id":"b997ba6c15bfc9a19a7167537e026f74ac32776a"},{"author":{"_account_id":10089,"name":"Anil Shashikumar Belur","email":"abelur@linuxfoundation.org","username":"askb"},"change_message_id":"39c2e2d0033e7c5a1b29c3d9ccb7ac84dce5fdc1","unresolved":false,"context_lines":[{"line_number":214,"context_line":"      loop:"},{"line_number":215,"context_line":"        - {who: alice, body: \"#startmeeting OpenDev Test Meeting\"}"},{"line_number":216,"context_line":"        - {who: alice, body: \"#topic Rollout plan\"}"},{"line_number":217,"context_line":"        - {who: bob, body: \"#info we are go for launch\"}"},{"line_number":218,"context_line":"        - {who: alice, body: \"#action bob to verify minutes\"}"},{"line_number":219,"context_line":"        - {who: bob, body: \"waves at everyone\", msgtype: m.emote}"},{"line_number":220,"context_line":"        - {who: alice, body: \"#agreed this works\"}"}],"source_content_type":"text/x-yaml","patch_set":3,"id":"3f5b6f4c_cf0bf7f2","line":217,"in_reply_to":"47919c9a_a161ac9b","updated":"2026-07-07 04:15:24.000000000","message":"Fixed in PS4. bob was invited but never joined. Since the test room is\npublic, PS4 drops the invite and adds an explicit join for bob\n(POST /join/{roomId}) before the meeting script runs, so bob is a member\nwhen he sends `#info`.","commit_id":"b997ba6c15bfc9a19a7167537e026f74ac32776a"}]}