)]}'
{"/PATCHSET_LEVEL":[{"author":{"_account_id":7118,"name":"Ian Wienand","email":"iwienand@redhat.com","username":"iwienand"},"change_message_id":"913f770d992338e5c1a7350fb5d26c9610828810","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"32ab21fd_f757f6a0","updated":"2025-02-20 03:06:58.000000000","message":"I think we got into a tangle with the revert in https://review.opendev.org/c/opendev/system-config/+/820250\n\nCurrently every infra-prod-service job runs https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/infra-prod/pre.yaml which sets up the executor to be able to log into bridge, copies system-config to bridge and runs the playbook. So yeah, the idea for parallel operation was that that step would only be \"sets up executor and runs playbook\"; with the copy being done once at the start.  That\u0027s where the split came from - https://opendev.org/opendev/base-jobs/commit/aa42212e7ed21dd9fb94e40f84d5edb99e125dae\n\nI agree that infra-prod-bootstrap-bridge isn\u0027t updating the system-config on bridge before it deploys the known hosts.  \n\nHowever, is this perhaps telling us that the `known_hosts` on the bridge should be setup from \"outside\" the bridge by the executor?\n\nSpecifically https://opendev.org/opendev/system-config/src/commit/52ad50a99fa10c5baecf4009a022e568438287b6/playbooks/roles/add-inventory-known-hosts/tasks/main.yaml#L4 we \"slurp\" the inventory from bridge and then mash it into the known_hosts.\n\nWhy would we not be taking that `inventory/base/hosts.yaml` from the copy on the executor that Zuul has provided for us?","commit_id":"5b0e59cf3d94a818c30c9e7d81bed5a9389a1bc6"},{"author":{"_account_id":7118,"name":"Ian Wienand","email":"iwienand@redhat.com","username":"iwienand"},"change_message_id":"d25adf80613739c853d1cba0e527b59fda9d3c3c","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"5b3b4027_9e2c4d10","in_reply_to":"32ab21fd_f757f6a0","updated":"2025-02-20 03:17:28.000000000","message":"https://review.opendev.org/c/opendev/system-config/+/942333 is an attempt at that ...","commit_id":"5b0e59cf3d94a818c30c9e7d81bed5a9389a1bc6"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"4f815d2bae122aba08c9992e63cb87613f6f7010","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"fa37dad4_60acc4ca","in_reply_to":"5b3b4027_9e2c4d10","updated":"2025-02-20 15:57:22.000000000","message":"Thanks I think the alternative approach should work fine. If we go with the alternative then I think we still want the dependency updates in this change but we can drop all of the edits to the bootstrap-bridge job in this change.","commit_id":"5b0e59cf3d94a818c30c9e7d81bed5a9389a1bc6"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"9a9a773e0fea9ae7ece634063784390a73c2beec","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"914f80d9_dc249906","in_reply_to":"7609cc65_ca686472","updated":"2025-02-24 21:01:41.000000000","message":"This rough plan makes sense to me and aligns with the original goals for parallelizing infra-prod job runs. Essentially we\u0027ll have one or two setup steps that run before everything else serially then we can run multiple jobs in parallel after that.\n\nThe main trick will be in ensuring we do so safely, not modifying shared resources on bridge accidentally. The idea of using a paused job that holds a semaphore to exclude any other buildset from proceeding until the current build set is done makes sense to me.\n\nThat said my primary concern is to ensure we can boot new servers today and deploy them without an additional round trip to update known_hosts. For that reason I\u0027ve updated this change to address the other two comments, but haven\u0027t tried to parallelize everything in this step. Instead I rebased ianw\u0027s DNM change to start sketching that out on top of this change and I think we can look in that direction for the future.","commit_id":"5b0e59cf3d94a818c30c9e7d81bed5a9389a1bc6"},{"author":{"_account_id":7118,"name":"Ian Wienand","email":"iwienand@redhat.com","username":"iwienand"},"change_message_id":"99d1f90fff58dcc316a721389993d4e1e88c4a5a","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"7609cc65_ca686472","in_reply_to":"fa37dad4_60acc4ca","updated":"2025-02-21 03:23:42.000000000","message":"OK, I think the insight I was missing that you\u0027re actually saying here is that there isn\u0027t really a separate \"source setup\" step - it\u0027s part of the bridge bootstrap.\n \nhttps://review.opendev.org/c/opendev/system-config/+/820250 shows that i was thinking there is a \"infra-prod-src-setup\" step somewhere in there.\n\nBut I don\u0027t think we need this, and I think we might be closer to cloning the source just once that I thought.\n\nDo we agree that the hierarchy we have in s-c:projects.yaml means that _all_ production jobs rely on a pre-run of `infra-prod-bootstrap-bridge`?\n\nEvery other job is parented to `infra-prod-playbook -\u003e opendev-infra-prod-base` which runs https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/infra-prod/pre.yaml - this setups keys so the executor can log into bridge _and_ copies source.\n\nSo -- can we make it so that `infra-prod-bootstrap-bridge` pauses, in say the same way as the `buildset-registry` job does, and \"locks out\" any other system-config runs starting while it is paused, until it\u0027s child prod runs finish?\n\nI think the reason we clone every job right now is that we hold a single semaphore for a production job, and then ensure system-config is where we want it (either our changeset, or for periodic jobs master), run the job, then give up the semaphore and start all over again.","commit_id":"5b0e59cf3d94a818c30c9e7d81bed5a9389a1bc6"}],"zuul.d/project.yaml":[{"author":{"_account_id":7118,"name":"Ian Wienand","email":"iwienand@redhat.com","username":"iwienand"},"change_message_id":"913f770d992338e5c1a7350fb5d26c9610828810","unresolved":true,"context_lines":[{"line_number":458,"context_line":"        - infra-prod-service-gitea-lb: \u0026infra-prod-service-gitea-lb"},{"line_number":459,"context_line":"            dependencies:"},{"line_number":460,"context_line":"              - name: infra-prod-base"},{"line_number":461,"context_line":"                soft: true"},{"line_number":462,"context_line":"              - name: system-config-promote-image-haproxy-statsd"},{"line_number":463,"context_line":"                soft: true"},{"line_number":464,"context_line":"        - infra-prod-service-grafana: \u0026infra-prod-service-grafana"}],"source_content_type":"text/x-yaml","patch_set":1,"id":"bfcf0478_b0001d96","line":461,"updated":"2025-02-20 03:06:58.000000000","message":"This is under the section \"letsencrypt depedencies\".  The LB probably does rely on that for certificates, so I think this should have\n\n```\ndependencies:\n  - name: infra-prod-letsencrypt\n    soft: true\n```\n\n?","commit_id":"5b0e59cf3d94a818c30c9e7d81bed5a9389a1bc6"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"9a9a773e0fea9ae7ece634063784390a73c2beec","unresolved":false,"context_lines":[{"line_number":458,"context_line":"        - infra-prod-service-gitea-lb: \u0026infra-prod-service-gitea-lb"},{"line_number":459,"context_line":"            dependencies:"},{"line_number":460,"context_line":"              - name: infra-prod-base"},{"line_number":461,"context_line":"                soft: true"},{"line_number":462,"context_line":"              - name: system-config-promote-image-haproxy-statsd"},{"line_number":463,"context_line":"                soft: true"},{"line_number":464,"context_line":"        - infra-prod-service-grafana: \u0026infra-prod-service-grafana"}],"source_content_type":"text/x-yaml","patch_set":1,"id":"58f5fd2b_031fe3b2","line":461,"in_reply_to":"96e85e19_50e8f57a","updated":"2025-02-24 21:01:41.000000000","message":"Done","commit_id":"5b0e59cf3d94a818c30c9e7d81bed5a9389a1bc6"},{"author":{"_account_id":7118,"name":"Ian Wienand","email":"iwienand@redhat.com","username":"iwienand"},"change_message_id":"99d1f90fff58dcc316a721389993d4e1e88c4a5a","unresolved":true,"context_lines":[{"line_number":458,"context_line":"        - infra-prod-service-gitea-lb: \u0026infra-prod-service-gitea-lb"},{"line_number":459,"context_line":"            dependencies:"},{"line_number":460,"context_line":"              - name: infra-prod-base"},{"line_number":461,"context_line":"                soft: true"},{"line_number":462,"context_line":"              - name: system-config-promote-image-haproxy-statsd"},{"line_number":463,"context_line":"                soft: true"},{"line_number":464,"context_line":"        - infra-prod-service-grafana: \u0026infra-prod-service-grafana"}],"source_content_type":"text/x-yaml","patch_set":1,"id":"96e85e19_50e8f57a","line":461,"in_reply_to":"9727e4c1_67dd7e68","updated":"2025-02-21 03:23:42.000000000","message":"yeah fair enough.  it\u0027s really only if it needs the certificate files on disk, which as you say doesn\u0027t apply for the lb.  still ... perhaps it should move out of this section into it\u0027s own.","commit_id":"5b0e59cf3d94a818c30c9e7d81bed5a9389a1bc6"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"4f815d2bae122aba08c9992e63cb87613f6f7010","unresolved":true,"context_lines":[{"line_number":458,"context_line":"        - infra-prod-service-gitea-lb: \u0026infra-prod-service-gitea-lb"},{"line_number":459,"context_line":"            dependencies:"},{"line_number":460,"context_line":"              - name: infra-prod-base"},{"line_number":461,"context_line":"                soft: true"},{"line_number":462,"context_line":"              - name: system-config-promote-image-haproxy-statsd"},{"line_number":463,"context_line":"                soft: true"},{"line_number":464,"context_line":"        - infra-prod-service-grafana: \u0026infra-prod-service-grafana"}],"source_content_type":"text/x-yaml","patch_set":1,"id":"9727e4c1_67dd7e68","line":461,"in_reply_to":"bfcf0478_b0001d96","updated":"2025-02-20 15:57:22.000000000","message":"The load balancers proxy at l3 not l4 and let the backends terminate ssl. I think a strict dependency on LE is not quite correct here though it wouldn\u0027t be totally wrong as the backends need LE.","commit_id":"5b0e59cf3d94a818c30c9e7d81bed5a9389a1bc6"},{"author":{"_account_id":7118,"name":"Ian Wienand","email":"iwienand@redhat.com","username":"iwienand"},"change_message_id":"913f770d992338e5c1a7350fb5d26c9610828810","unresolved":true,"context_lines":[{"line_number":549,"context_line":"        - infra-prod-service-zuul-lb: \u0026infra-prod-service-zuul-lb"},{"line_number":550,"context_line":"            dependencies:"},{"line_number":551,"context_line":"              - name: infra-prod-base"},{"line_number":552,"context_line":"                soft: true"},{"line_number":553,"context_line":"              - name: system-config-promote-image-haproxy-statsd"},{"line_number":554,"context_line":"                soft: true"},{"line_number":555,"context_line":"        - infra-prod-service-zuul-preview: \u0026infra-prod-service-zuul-preview"}],"source_content_type":"text/x-yaml","patch_set":1,"id":"0435bb62_378becdc","line":552,"updated":"2025-02-20 03:06:58.000000000","message":"Similar for these two - this is the list of things that relies on LE having run to put their certificates in place before they start.","commit_id":"5b0e59cf3d94a818c30c9e7d81bed5a9389a1bc6"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"4f815d2bae122aba08c9992e63cb87613f6f7010","unresolved":true,"context_lines":[{"line_number":549,"context_line":"        - infra-prod-service-zuul-lb: \u0026infra-prod-service-zuul-lb"},{"line_number":550,"context_line":"            dependencies:"},{"line_number":551,"context_line":"              - name: infra-prod-base"},{"line_number":552,"context_line":"                soft: true"},{"line_number":553,"context_line":"              - name: system-config-promote-image-haproxy-statsd"},{"line_number":554,"context_line":"                soft: true"},{"line_number":555,"context_line":"        - infra-prod-service-zuul-preview: \u0026infra-prod-service-zuul-preview"}],"source_content_type":"text/x-yaml","patch_set":1,"id":"33d5c5a6_82c18040","line":552,"in_reply_to":"0435bb62_378becdc","updated":"2025-02-20 15:57:22.000000000","message":"The zuul-db definitely doesn\u0027t need LE so I think that one should stay at least.","commit_id":"5b0e59cf3d94a818c30c9e7d81bed5a9389a1bc6"},{"author":{"_account_id":7118,"name":"Ian Wienand","email":"iwienand@redhat.com","username":"iwienand"},"change_message_id":"99d1f90fff58dcc316a721389993d4e1e88c4a5a","unresolved":true,"context_lines":[{"line_number":549,"context_line":"        - infra-prod-service-zuul-lb: \u0026infra-prod-service-zuul-lb"},{"line_number":550,"context_line":"            dependencies:"},{"line_number":551,"context_line":"              - name: infra-prod-base"},{"line_number":552,"context_line":"                soft: true"},{"line_number":553,"context_line":"              - name: system-config-promote-image-haproxy-statsd"},{"line_number":554,"context_line":"                soft: true"},{"line_number":555,"context_line":"        - infra-prod-service-zuul-preview: \u0026infra-prod-service-zuul-preview"}],"source_content_type":"text/x-yaml","patch_set":1,"id":"e24e17c5_1295e2a4","line":552,"in_reply_to":"33d5c5a6_82c18040","updated":"2025-02-21 03:23:42.000000000","message":"OK I guess that uses firewall rules; perhaps should be moved out too","commit_id":"5b0e59cf3d94a818c30c9e7d81bed5a9389a1bc6"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"9a9a773e0fea9ae7ece634063784390a73c2beec","unresolved":false,"context_lines":[{"line_number":549,"context_line":"        - infra-prod-service-zuul-lb: \u0026infra-prod-service-zuul-lb"},{"line_number":550,"context_line":"            dependencies:"},{"line_number":551,"context_line":"              - name: infra-prod-base"},{"line_number":552,"context_line":"                soft: true"},{"line_number":553,"context_line":"              - name: system-config-promote-image-haproxy-statsd"},{"line_number":554,"context_line":"                soft: true"},{"line_number":555,"context_line":"        - infra-prod-service-zuul-preview: \u0026infra-prod-service-zuul-preview"}],"source_content_type":"text/x-yaml","patch_set":1,"id":"7415f4eb_70c21c88","line":552,"in_reply_to":"e24e17c5_1295e2a4","updated":"2025-02-24 21:01:41.000000000","message":"Done","commit_id":"5b0e59cf3d94a818c30c9e7d81bed5a9389a1bc6"}]}