)]}'
{"/PATCHSET_LEVEL":[{"author":{"_account_id":27900,"name":"Artem Goncharov","email":"artem.goncharov@gmail.com","username":"gtema"},"change_message_id":"94656cfc2184f8ca48a39f8df80a968985b0673b","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"8a5e9e3f_9526c2e0","updated":"2022-02-16 09:31:30.000000000","message":"I think my brain have after a while distorted what I was reading on https://docs.opendev.org/opendev/system-config/latest/zuul.html#secrets about secrets. On the other side we should still have something like key exhaustion (https://docs.opendev.org/opendev/system-config/latest/zuul.html#secrets) and similar stuff (yes, we rotate secret, but not the key used to encrypt the secret). I am not against of having some secrets encrypted by Zuul, but clearly rotating this data often without rotating the private key is like a red flag for any security person.\n\nOn the other side, as you mentioned, we also have people accessing bridge and using it further for ops reasons (and believe we do this daily). Moving secrets to Zuul is clearly going to make this harder, since secrets can not be fetched on the bridge. In our approach playbooks are being executed on the bridge exactly like before (both by Zuul and manually). It\u0027s just that the secrets are being fetched by Ansible from Vault directly (where absolutely necessary). Where it is not necessary we modify system deployments so that they are able to fetch secrets they require for work directly from Vault (i.e. Grafana deployment fetches itself DB password from Vault).\n\nLast, but not least, is the issue with rotation itself, that mostly that need to be a synchronous process. If you rotate DB password and update it in Git i.e. Grafana (as from example above) is not capable to access DB anymore until change is reviewed/merged/deployed - we know from practice it takes days rather then minutes. Not every secret is so complex, but it is very often the case (especially rotating password for OpenStack user i.e. nodepool is using).\n\nHonestly speaking our approach is not really resolving problem with bridge bootstrapping, and instead is making it even more challenging (depends from which angle you look). But at least now we are sure that from the security point of view we are moving to the safe side (having rotation/audit/least_required_privilege addressed).\n\nMy biggest concern with Zuul is that one you (accidentally or not, corruption) loose Zuul (encryption keys, or maybe even completely) - you are completely screwed. You can not recover even Zuul itself without manual steps on the bridge, but it most likely lacks required secrets. To avoid this you will not put all secrets into Zuul. But that, on the other side, just increases amount of different approaches used.\n\nWe have actually submitted talk for Summit to describe how we are doing this. If there is interest we can present this also in a smaller round for infra folks beforehand.","commit_id":"b219eb72ba325da6590324535dc81aad95464971"},{"author":{"_account_id":27900,"name":"Artem Goncharov","email":"artem.goncharov@gmail.com","username":"gtema"},"change_message_id":"6bedfe4576826c7f9eb652212aee1f09a6cf1b9c","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"c07bc458_e03a8440","updated":"2022-02-17 17:07:27.000000000","message":"Sure. Things just start as a spec and it is exactly time to raise points before they get more complex.","commit_id":"b219eb72ba325da6590324535dc81aad95464971"},{"author":{"_account_id":27900,"name":"Artem Goncharov","email":"artem.goncharov@gmail.com","username":"gtema"},"change_message_id":"bd1cb0a61e9abfe70eabd49a4036c062dd020528","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"d0753f42_6848365c","updated":"2022-02-15 07:06:59.000000000","message":"in most cases security regulations of companies are requiring that any type of secret is being periodically rotated. Storing them in Zuul is making that both hard and reduces secret safety with each iteration (as also written in Zuul docs if I read them correctly). We will not go this way. Instead we created analog sys-config project in the private repository and merge their states on the bridge. In addition private repo is also not having secrets directly, but knows where to get them from (Hashi Vault in our case). With this secrets themselves are stored in Vaul where they can be easily rotated, and jobs/playbooks are knowing how to get secrets (actually we rely mostly on lookup plugins not even ti take care of that).\nThis is correct that bootstraping step need to be rethought, but simply puting secrets as zuul secrets is not proper from security pov.","commit_id":"b219eb72ba325da6590324535dc81aad95464971"},{"author":{"_account_id":7118,"name":"Ian Wienand","email":"iwienand@redhat.com","username":"iwienand"},"change_message_id":"aeb6c421e13fdb5e3edc33c1b923aad2a9db61a0","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"89db8365_229a5b4d","in_reply_to":"8a5e9e3f_9526c2e0","updated":"2022-02-17 02:30:15.000000000","message":"\u003e Moving secrets to Zuul is clearly going to make this harder, since secrets can not be fetched on the bridge.\n\n\u003e My biggest concern with Zuul is that one you (accidentally or not, corruption) loose Zuul (encryption keys, or maybe even completely) - you are completely screwed. You can not recover even Zuul itself without manual steps on the bridge, but it most likely lacks required secrets.\n\nNote that this proposal is only suggesting to write the secrets kept in Zuul secret definitions to the bastion host for ansible to consume.  This is ~ equivalent to the current situation where humans write the secrets to the bastion host (except of course for the obvious caveat that Zuul has now has an encrypted copy at rest, as opposed to it being essentially discarded from the human\u0027s brain by assuming they can\u0027t remember it 😊.  But in terms of being able to manually run ansible on the bastion host, or recover if Zuul is lost, I believe we\u0027re in exactly the same situation.\n\nYou have made me think though -- is it a better idea to just encrypt the value with a public key, with the private portion kept on the bastion host, and then write that out with a Zuul job (that has access to the private key on the bastion host).  \n\nThis is an option because the bastion host is static, but still allows anyone to commit a key to the repository.\n\nWe\u0027re already keeping the secrets on the bastion host, so having the private key there doesn\u0027t seem any different; if you can access the key you can access the secrets anyway.","commit_id":"b219eb72ba325da6590324535dc81aad95464971"},{"author":{"_account_id":7118,"name":"Ian Wienand","email":"iwienand@redhat.com","username":"iwienand"},"change_message_id":"b9d88bbedee9cc61ac502c11a1fc4085ebad821e","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"b61912d7_0bdd05c5","in_reply_to":"d0753f42_6848365c","updated":"2022-02-15 20:15:32.000000000","message":"Thanks, I\u0027m definitely interested in iterating on any ideas that we can come up with.\n\nI agree on the need for rotation; I\u0027m not sure it\u0027s particularly easier or harder than any other way to rotate.  Ultimately you just have to run a little script and propose a change with this model.\n\nI\u0027m not sure that rotating and re-encrypting a zuul value makes it any less secure. Can you clarify this a bit more?  I will admit that this creates a side-channel where you give away some information by committing a change that tells everyone the key was just rotated, but I\u0027m not aware it would weaken the secret value as such.\n\nThe back-end system on the bastion/bridge we can consider opaque I guess (our bastion has something similar to get keys into place from repos for ansible to read).  The general point is that this system is managed by selected operators who need explicit permission to add/remove/rotate secrets.\n\nWe are trying to live an open-infrastructure model, where we would ideally like non-privileged users to be able to manage production systems.  (Sometimes, when things have really gone wrong, you need to log in.  But we want to make this the absolute exception, as much as possible.)\n\nI\u0027ve just put up for review some related work that will allow production deployment logs to be viewed by people who have a GPG key committed to the system-config repo [1].  That is in a similar orbit of allowing anyone with interest to help manage the systems.\n\nSo this does have to be filtered though a lens of trying to achieve these goals. This obviously isn\u0027t every project\u0027s goal, but it is ours 😊\n\n[1] https://review.opendev.org/q/topic:bridge-encrypt-logs","commit_id":"b219eb72ba325da6590324535dc81aad95464971"},{"author":{"_account_id":7118,"name":"Ian Wienand","email":"iwienand@redhat.com","username":"iwienand"},"change_message_id":"610468928bb3f6fbb11a9fca4477c798f08b063d","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":2,"id":"ed105b65_6dd3977b","updated":"2023-04-12 22:44:15.000000000","message":"I was just playing with this to see the operation of the rolecall-vote copyConditions when you upload a change that modified the author, but nothing else.","commit_id":"dea3d2b6897c5fe080e0a47f6b6814d749018742"}]}