)]}'
{"/COMMIT_MSG":[{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"8d9d6dc8293ea14fa238ece2efd294aa070a54fe","unresolved":true,"context_lines":[{"line_number":9,"context_line":"The executor writes \u003cjobdir\u003e/home/.ssh/known_hosts from the host keys"},{"line_number":10,"context_line":"recorded when the node was booted.  OpenSSH\u0027s UpdateHostKeys, on by"},{"line_number":11,"context_line":"default, rewrites that file when a node offers a different key set and"},{"line_number":12,"context_line":"leaves a known_hosts.old backup next to it."},{"line_number":13,"context_line":""},{"line_number":14,"context_line":"Jobs that reboot the node hit this on every run.  The FIPS jobs are the"},{"line_number":15,"context_line":"obvious case: the reboot restricts the host key algorithms the node may"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"7db02ffe_f2f20f08","line":12,"updated":"2026-08-27 14:44:13.000000000","message":"What happens when there are no host keys because host-key-checking is disabled?","commit_id":"6c091b2fd0f154214a25643979655ceab8af0e13"},{"author":{"_account_id":6968,"name":"Christian Schwede","email":"cschwede@nvidia.com","username":"cschwede"},"change_message_id":"ecf0f592023e2562e12b1687376f07b92524a777","unresolved":true,"context_lines":[{"line_number":9,"context_line":"The executor writes \u003cjobdir\u003e/home/.ssh/known_hosts from the host keys"},{"line_number":10,"context_line":"recorded when the node was booted.  OpenSSH\u0027s UpdateHostKeys, on by"},{"line_number":11,"context_line":"default, rewrites that file when a node offers a different key set and"},{"line_number":12,"context_line":"leaves a known_hosts.old backup next to it."},{"line_number":13,"context_line":""},{"line_number":14,"context_line":"Jobs that reboot the node hit this on every run.  The FIPS jobs are the"},{"line_number":15,"context_line":"obvious case: the reboot restricts the host key algorithms the node may"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"f0fe7d27_8f675da5","line":12,"in_reply_to":"7db02ffe_f2f20f08","updated":"2026-08-28 10:50:46.000000000","message":"Thx @jim@acmegating.com for looking into this!\n\nTo answer your question - actually nothing is written then. That raised some questions on my side, and I run a couple checks in 1002671: DNM: Diagnose known_hosts.old creation | https://review.opendev.org/c/openstack/swift/+/1002671\n\nTurns out this patch won\u0027t fix it: it\u0027s the playbook synchronize task using rsync ssh that creates the known_hosts.old. Because `use_ssh_args` defaults to false in the Ansible synchronize module, this patch won\u0027t fix it.\n\nThere are a few options to go forward:\n1. Add `use_ssh_args: true` to all synchronize tasks in zuul-jobs. Doesn\u0027t lgtm.\n2. Let zuul write a `~/.ssh/config` with `UpdateHostKeys no` - should fix this.\n3. Add the `known_hosts.old` to `self.preserve_home_paths` - would disable the warning, but keep the file. Doesn\u0027t lgtm either.\n4. Do nothing, simply wait until zuul eventually deletes these files (assuming that this no longer raises a warning).\n\nWDYT?","commit_id":"6c091b2fd0f154214a25643979655ceab8af0e13"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"bec04d7ad1933ade68e71c2ee8a024d1d48dda08","unresolved":true,"context_lines":[{"line_number":9,"context_line":"The executor writes \u003cjobdir\u003e/home/.ssh/known_hosts from the host keys"},{"line_number":10,"context_line":"recorded when the node was booted.  OpenSSH\u0027s UpdateHostKeys, on by"},{"line_number":11,"context_line":"default, rewrites that file when a node offers a different key set and"},{"line_number":12,"context_line":"leaves a known_hosts.old backup next to it."},{"line_number":13,"context_line":""},{"line_number":14,"context_line":"Jobs that reboot the node hit this on every run.  The FIPS jobs are the"},{"line_number":15,"context_line":"obvious case: the reboot restricts the host key algorithms the node may"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"c9027565_65f93e41","line":12,"in_reply_to":"f0fe7d27_8f675da5","updated":"2026-08-29 15:28:23.000000000","message":"Thank you for looking into this as well!\n\nMy concern with adding \"UpdateHostKeys no\" (to either location) is mostly with the case where host-key-checking is disabled and we\u0027re not providing keys.  If the current behavior is to add keys to that file, effectively a \"trust on first use\" situation, I don\u0027t want to break that.  Zuul adds the \"-o StrictHostKeyChecking\u003dfalse\" option to the ansible_ssh_common_args host variable in order to facilitate this.\n\nI have experimented in a workstation environment, and it appears that even with UpdateHostKeys\u003dno, openssh will still add a new key to known_hosts (without writing a .old file).  I guess the idea is that it only writes a .old file if it\u0027s changing or removing an entry, not adding a new one.\n\nThat seems to suggest that UpdateHostKeys\u003dno should be safe, and I agree it\u0027s a good option.  I haven\u0027t tested this in a full Zuul+Ansible environment though; so if I\u0027m missing something, please let me know.\n\nI do also think that adding known_hosts.old to the allow list may be acceptable.  The goal of the warning, and eventual deletion, is to avoid a situation where an untrusted playbook tricks a later trusted playbook into doing something unintended by its author.  I don\u0027t see a way for known_hosts.old to do that, nor do I necessarily think that host_keys changing is automatically bad (it could be!  but if some attack relies on changing the host keys file, it seems very, very, likely that attack could also just clean up the .old file in order to avoid detection here, so while, sure, this could expose or prevent such attack if the attacker is inept, it\u0027s not a real defence since it\u0027s so easily bypassed).\n\nAccording to the Ansible docs, the use_ssh_args argument also controls the use of the ansible_ssh_common_args, so I agree, if we want to set UpdateHostKeys\u003dno, we need to do it in the .ssh/config file (which we do not currently write at all).\n\nRegarding doing nothing: that is also tempting; I am not sure if we will keep the warnings in that case.  Regardless, this seems like it is affecting a number of different use cases, and if we decide it\u0027s noise, we should probably reduce the noise.  Based on the security analysis I wrote above, I think adding known_hosts.old to the allow list is effectively the same as \"do nothing\" the way that you described it (eventually delete with no warning).\n\nI think either UpdateHostKeys\u003dno or adding to the allow list is okay.  I don\u0027t see any issues with adding the .ssh/config file, but it does have the potential to have ripple effects we don\u0027t foresee now.  Adding the .old file to the allow list seems easiest and minimally intrusive.  You said you didn\u0027t like that idea -- if you want to elaborate on why, I would welcome that (I may be missing something, or you may be extra cautious).\n\nI think I\u0027d also like to get some more opinions too before we make a decision.","commit_id":"6c091b2fd0f154214a25643979655ceab8af0e13"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"8d9d6dc8293ea14fa238ece2efd294aa070a54fe","unresolved":true,"context_lines":[{"line_number":21,"context_line":"UpdateHostKeys only runs after the host key has already been verified,"},{"line_number":22,"context_line":"so this cannot make a connection fail that would otherwise succeed."},{"line_number":23,"context_line":""},{"line_number":24,"context_line":"Co-Authored-By: Claude Opus 5 \u003cnoreply@anthropic.com\u003e"},{"line_number":25,"context_line":"Change-Id: I58bd1356a7ebf754d1c419b14600092240d64074"},{"line_number":26,"context_line":"Signed-off-by: Christian Schwede \u003ccschwede@mailbox.org\u003e"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"9abc800b_d4ef7d05","line":24,"updated":"2026-08-27 14:44:13.000000000","message":"For future reference, this should have a Generated-By or Assisted-By label; please see the policies at\n* https://opendev.org/zuul/zuul#ai-generated-content\n* https://openinfra.org/legal/ai-policy","commit_id":"6c091b2fd0f154214a25643979655ceab8af0e13"},{"author":{"_account_id":6968,"name":"Christian Schwede","email":"cschwede@nvidia.com","username":"cschwede"},"change_message_id":"ecf0f592023e2562e12b1687376f07b92524a777","unresolved":true,"context_lines":[{"line_number":21,"context_line":"UpdateHostKeys only runs after the host key has already been verified,"},{"line_number":22,"context_line":"so this cannot make a connection fail that would otherwise succeed."},{"line_number":23,"context_line":""},{"line_number":24,"context_line":"Co-Authored-By: Claude Opus 5 \u003cnoreply@anthropic.com\u003e"},{"line_number":25,"context_line":"Change-Id: I58bd1356a7ebf754d1c419b14600092240d64074"},{"line_number":26,"context_line":"Signed-off-by: Christian Schwede \u003ccschwede@mailbox.org\u003e"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"ba509a7c_b7c0a645","line":24,"in_reply_to":"9abc800b_d4ef7d05","updated":"2026-08-28 10:50:46.000000000","message":"Thanks, noted and will change if this patch gets another patchset.","commit_id":"6c091b2fd0f154214a25643979655ceab8af0e13"}],"/PATCHSET_LEVEL":[{"author":{"_account_id":6968,"name":"Christian Schwede","email":"cschwede@nvidia.com","username":"cschwede"},"change_message_id":"4410ef22cb50c18eefa95207aff39425253925f7","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"99286cd9_09b99247","updated":"2026-08-27 14:36:46.000000000","message":"@cboylan@sapwetik.org I saw the patch from you while I was debugging the known_hosts.old warnings - would be happy to know your views on this.","commit_id":"6c091b2fd0f154214a25643979655ceab8af0e13"},{"author":{"_account_id":6968,"name":"Christian Schwede","email":"cschwede@nvidia.com","username":"cschwede"},"change_message_id":"0de76a2828e2e6853d9613b2eae7753e0f81fe18","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"2b6194c3_84bf82f3","updated":"2026-08-27 14:37:46.000000000","message":"For reference, example warnings from a recent Swift patch: https://review.opendev.org/c/openstack/swift/+/1002636#message-b717e4abcff3de4cdcaa75aa4b739e62ab9be883","commit_id":"6c091b2fd0f154214a25643979655ceab8af0e13"}],"releasenotes/notes/no-update-host-keys-a99dd5ebbe950841.yaml":[{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"8d9d6dc8293ea14fa238ece2efd294aa070a54fe","unresolved":true,"context_lines":[{"line_number":1,"context_line":"---"},{"line_number":2,"context_line":"upgrade:"},{"line_number":3,"context_line":"  - |"},{"line_number":4,"context_line":"    The executor now passes ``UpdateHostKeys\u003dno`` to ssh.  The build\u0027s"},{"line_number":5,"context_line":"    ``known_hosts`` file is written by the executor from the host keys"}],"source_content_type":"text/x-yaml","patch_set":1,"id":"61f0107c_45559d30","line":2,"updated":"2026-08-27 14:44:13.000000000","message":"I would suggest putting this under either \"fixes\" or \"features\"; we usually only put things that people need to do to manage an upgrade here.","commit_id":"6c091b2fd0f154214a25643979655ceab8af0e13"},{"author":{"_account_id":6968,"name":"Christian Schwede","email":"cschwede@nvidia.com","username":"cschwede"},"change_message_id":"ecf0f592023e2562e12b1687376f07b92524a777","unresolved":false,"context_lines":[{"line_number":1,"context_line":"---"},{"line_number":2,"context_line":"upgrade:"},{"line_number":3,"context_line":"  - |"},{"line_number":4,"context_line":"    The executor now passes ``UpdateHostKeys\u003dno`` to ssh.  The build\u0027s"},{"line_number":5,"context_line":"    ``known_hosts`` file is written by the executor from the host keys"}],"source_content_type":"text/x-yaml","patch_set":1,"id":"aa3e222d_8c4a886c","line":2,"in_reply_to":"61f0107c_45559d30","updated":"2026-08-28 10:50:46.000000000","message":"Acknowledged","commit_id":"6c091b2fd0f154214a25643979655ceab8af0e13"}]}