)]}'
{"/PATCHSET_LEVEL":[{"author":{"_account_id":9311,"name":"Tristan Cacqueray","email":"tdecacqu@redhat.com","username":"tristanC"},"change_message_id":"4ba063a82b9001fd8eff5f526c2595a375fac9ec","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"0465dd01_6a9c0891","updated":"2022-03-17 21:48:06.000000000","message":"I do think this is reducing our security posture, the restriction imposed on untrusted jobs may not be perfect, but they provide an extra layer to prevent remote code execution on the executor instances.\n\nIf I understand correctly, this spec has a significant impact for operators, thus if we can improve the compartmentalizations along the way that may be worth the effort. For example, it seems like nodeless job may cause a \"noisy neighbor\" effect resulting in a negative impact.\n\nThat being said, I am in favor of this spec because the benefits outweigh the risk. \n","commit_id":"66384d0d46403ec30cbf15577d152b918fb12fb5"}],"doc/source/developer/specs/unrestricted-ansible.rst":[{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"d428dcf2dff5d42dc1d4f8f7f4c008d34cb56664","unresolved":true,"context_lines":[{"line_number":36,"context_line":"#. The number of plugins included in the community edition of Ansible"},{"line_number":37,"context_line":"   (the \"batteries-included\") is considerably larger than that in"},{"line_number":38,"context_line":"   Ansible 2.9 (meanwhile, the set in Ansible core is smaller than"},{"line_number":39,"context_line":"   that in 2.9)."},{"line_number":40,"context_line":""},{"line_number":41,"context_line":"#. The process of loading plugins differs depending on how they are"},{"line_number":42,"context_line":"   named (e.g., using the `csvfile` lookup plugin causes different"}],"source_content_type":"text/x-rst","patch_set":2,"id":"bc5dc3fc_cbc2a4d6","line":39,"updated":"2022-03-17 16:48:40.000000000","message":"Is it possible for Zuul to install an ansible core + limited batteries included to more closely match 2.9? Would that simplify things enough to be a possibility?","commit_id":"1685cf528f4ca158ef655c1c9585ff3024440a30"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"0e30d538e6ae62092c051c11f8dbb75e0ccdac9f","unresolved":false,"context_lines":[{"line_number":36,"context_line":"#. The number of plugins included in the community edition of Ansible"},{"line_number":37,"context_line":"   (the \"batteries-included\") is considerably larger than that in"},{"line_number":38,"context_line":"   Ansible 2.9 (meanwhile, the set in Ansible core is smaller than"},{"line_number":39,"context_line":"   that in 2.9)."},{"line_number":40,"context_line":""},{"line_number":41,"context_line":"#. The process of loading plugins differs depending on how they are"},{"line_number":42,"context_line":"   named (e.g., using the `csvfile` lookup plugin causes different"}],"source_content_type":"text/x-rst","patch_set":2,"id":"225a5d85_431d528d","line":39,"updated":"2022-03-17 17:22:18.000000000","message":"Yes.  But then we are curating our own Ansible distribution, and we have to do the work to qualify the collections that we install.  We would have to install several to obtain the basic level of functionality we expect.\n\nEven if we don\u0027t accept this proposal, I think we should use Ansible community so that we continue to approximate what \"install ansible\" means to people.\n\nHowever, if we don\u0027t accept this proposal, we may want to consider going the other direction and say since we\u0027re not going to have a fully-functional Ansible anyway, let\u0027s reduce the functionality further and maybe install fewer collections so that we don\u0027t support everything we have now in 2.9.  That might be a win for maintainability.\n\nAnother approach to consider (especially regarding your concerns below) would be to lift the file/network access permissions from our restricted ansible and try to only focus on limiting untrusted execution.  That might further reduce the scope of work (we would still need to figure out how to override the new collections loading mechanism, but at least then we would have far fewer plugins to try to override).","commit_id":"1685cf528f4ca158ef655c1c9585ff3024440a30"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"3745e1648fa78656afabaec887981932874d4c4b","unresolved":false,"context_lines":[{"line_number":36,"context_line":"#. The number of plugins included in the community edition of Ansible"},{"line_number":37,"context_line":"   (the \"batteries-included\") is considerably larger than that in"},{"line_number":38,"context_line":"   Ansible 2.9 (meanwhile, the set in Ansible core is smaller than"},{"line_number":39,"context_line":"   that in 2.9)."},{"line_number":40,"context_line":""},{"line_number":41,"context_line":"#. The process of loading plugins differs depending on how they are"},{"line_number":42,"context_line":"   named (e.g., using the `csvfile` lookup plugin causes different"}],"source_content_type":"text/x-rst","patch_set":2,"id":"0d788a13_c0c771e0","line":39,"in_reply_to":"225a5d85_431d528d","updated":"2022-03-17 17:41:09.000000000","message":"That is the good point about diverging from what people expect from Ansible.","commit_id":"1685cf528f4ca158ef655c1c9585ff3024440a30"},{"author":{"_account_id":9061,"name":"David Moreau Simard","email":"moi@dmsimard.com","username":"dmsimard"},"change_message_id":"272c7b68e292b703e0b62ff705ac6f67d249ac47","unresolved":true,"context_lines":[{"line_number":36,"context_line":"#. The number of plugins included in the community edition of Ansible"},{"line_number":37,"context_line":"   (the \"batteries-included\") is considerably larger than that in"},{"line_number":38,"context_line":"   Ansible 2.9 (meanwhile, the set in Ansible core is smaller than"},{"line_number":39,"context_line":"   that in 2.9)."},{"line_number":40,"context_line":""},{"line_number":41,"context_line":"#. The process of loading plugins differs depending on how they are"},{"line_number":42,"context_line":"   named (e.g., using the `csvfile` lookup plugin causes different"}],"source_content_type":"text/x-rst","patch_set":2,"id":"abf4f1cb_f3ac212a","line":39,"in_reply_to":"bc5dc3fc_cbc2a4d6","updated":"2022-03-17 18:13:43.000000000","message":"I will not speak for Zuul\u0027s ability to do that or not but it is an intended use case to be able to install just ansible-core and then cherry-pick the collections that users are interested in.\n\nThere\u0027s a galaxy requirements file that is available and kept up to date based on the collections that are included in the latest release of the ``ansible`` package from which you can remove the things you don\u0027t have a use for: https://github.com/ansible-community/ansible-build-data/blob/main/5/galaxy-requirements.yaml","commit_id":"1685cf528f4ca158ef655c1c9585ff3024440a30"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"d428dcf2dff5d42dc1d4f8f7f4c008d34cb56664","unresolved":true,"context_lines":[{"line_number":123,"context_line":""},{"line_number":124,"context_line":"The ability to execute arbitrary code locally combined with a"},{"line_number":125,"context_line":"potential future local root exploit could allow an attacker to gain"},{"line_number":126,"context_line":"control of the Zuul system."},{"line_number":127,"context_line":""},{"line_number":128,"context_line":"Operators will need to be cognizant of the risk and keep systems up to"},{"line_number":129,"context_line":"date and pro-actively rebuild executor servers and rotate credentials"}],"source_content_type":"text/x-rst","patch_set":2,"id":"cf3b2a6a_85cf19d1","line":126,"updated":"2022-03-17 16:48:40.000000000","message":"It may be worth noting that the linux kernel has had several such bugs in the last ~month. I think operators would have to consider immediately shutting down Zuul to patch things like the kernel when these local privilege escalation bugs are disclosed.\n\nWe have always tried to patch quickly anyway, but the added layer of protection gives us some breathing room which has been nice.\n\nAdditionally rotating credentials is a very expensive endeavor now that executors can talk to zookeeper.\n\nThese two things combined make me wary of this change. Executors have more access than ever and linux is seeing major exploits right now.\n\nMy initial thought on this is that if the change were made OpenDev should consider not driving CD through Zuul any more. To decouple the risk of executors from the rest of the system more fully. Maybe run a secondary small zuul just for this instead with restricted access? Haven\u0027t fully thought this through.","commit_id":"1685cf528f4ca158ef655c1c9585ff3024440a30"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"3af030a5df92a83bbda35109a9e99113c64388cc","unresolved":false,"context_lines":[{"line_number":123,"context_line":""},{"line_number":124,"context_line":"The ability to execute arbitrary code locally combined with a"},{"line_number":125,"context_line":"potential future local root exploit could allow an attacker to gain"},{"line_number":126,"context_line":"control of the Zuul system."},{"line_number":127,"context_line":""},{"line_number":128,"context_line":"Operators will need to be cognizant of the risk and keep systems up to"},{"line_number":129,"context_line":"date and pro-actively rebuild executor servers and rotate credentials"}],"source_content_type":"text/x-rst","patch_set":2,"id":"5eb42a84_080441eb","line":126,"updated":"2022-03-17 18:01:43.000000000","message":"The original idea of Zuul v3 was simple: use your production Ansible playbooks in testing and run both directly from Zuul.  That\u0027s why Nodesets are Ansible inventory, etc.  You just swap one for the other.\n\nThe restricted Ansible environment prevents us from achieving that because an untrusted playbook can\u0027t use enough of Ansible to actually directly execute a real production playbook.  Generally this is solved by nesting Ansible.  At this point, it hardly matters what Zuul itself is running.  It\u0027s just a multi-node execution coordinator that happens to implement a subset of Ansible.\n\nSo the motivation to do this to improve the CD story is to get back to that original idea: just have Zuul run Ansible.\n\nAn OpenDev-like system could just \"run: playbooks/eavesdrop.yaml\" using the built-in ssh keys, etc, and not have to worry about anything else.  Now, OpenDev itself may still not end up doing that because of logs and a desire to run things from bridge.  But the potential is there, and anyone without OpenDev\u0027s concerns about logs might be more comfortable with it.\n\nOpenDev could certainly directly execute simple playbooks like tenant reconfiguration loading, or publishing meeting calendars, or other similar event-driven CD things where the unique requirements aren\u0027t present.\n\nIf we\u0027re inclined to accept the risk and accept this proposal, then I think that\u0027s the story.  If we are not inclined, then I think the CD story becomes \"Zuul runs a tiny subset of mostly-compatible-Ansible that you use to bootstrap whatever it is you want to use to CD, but you\u0027ll need an external node.  More or less what we have today, but you\u0027ve got me thinking that if we can\u0027t fully use Ansible then maybe we should minimize it even more and start really pushing the angle that Zuul\u0027s Ansible is only there to bootstrap stuff.","commit_id":"1685cf528f4ca158ef655c1c9585ff3024440a30"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"0e30d538e6ae62092c051c11f8dbb75e0ccdac9f","unresolved":false,"context_lines":[{"line_number":123,"context_line":""},{"line_number":124,"context_line":"The ability to execute arbitrary code locally combined with a"},{"line_number":125,"context_line":"potential future local root exploit could allow an attacker to gain"},{"line_number":126,"context_line":"control of the Zuul system."},{"line_number":127,"context_line":""},{"line_number":128,"context_line":"Operators will need to be cognizant of the risk and keep systems up to"},{"line_number":129,"context_line":"date and pro-actively rebuild executor servers and rotate credentials"}],"source_content_type":"text/x-rst","patch_set":2,"id":"62c8fa41_96bcce3b","line":126,"updated":"2022-03-17 17:22:18.000000000","message":"This change has two benefits: 1) it reduces the amount of holes we have to try to find and plug every time there is a new Ansible; and 2) it makes CD directly from Zuul more viable.\n\nIf we don\u0027t think that improves the CD story, then that removes that benefit and makes this much less compelling.\n\nPersonally (as I noted at the bottom) I think the current protection is so weak as to be nearly nonexistent, but I do agree it is something, and it may take someone a little longer to figure out how to run something on the executor.\n\nA determined attacker of course would find and sit on a Zuul restriction bypass until they obtained the local root exploit so they could use them simultaneously.","commit_id":"1685cf528f4ca158ef655c1c9585ff3024440a30"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"2dbd05f1f01e90892bab4baac3fc875f3e2b6f08","unresolved":false,"context_lines":[{"line_number":123,"context_line":""},{"line_number":124,"context_line":"The ability to execute arbitrary code locally combined with a"},{"line_number":125,"context_line":"potential future local root exploit could allow an attacker to gain"},{"line_number":126,"context_line":"control of the Zuul system."},{"line_number":127,"context_line":""},{"line_number":128,"context_line":"Operators will need to be cognizant of the risk and keep systems up to"},{"line_number":129,"context_line":"date and pro-actively rebuild executor servers and rotate credentials"}],"source_content_type":"text/x-rst","patch_set":2,"id":"6772220b_908c89f4","line":126,"updated":"2022-03-17 18:14:01.000000000","message":"Tristan, yes, a successful exploit would get access to ZK and therefore all Zuul secrets.","commit_id":"1685cf528f4ca158ef655c1c9585ff3024440a30"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"3745e1648fa78656afabaec887981932874d4c4b","unresolved":false,"context_lines":[{"line_number":123,"context_line":""},{"line_number":124,"context_line":"The ability to execute arbitrary code locally combined with a"},{"line_number":125,"context_line":"potential future local root exploit could allow an attacker to gain"},{"line_number":126,"context_line":"control of the Zuul system."},{"line_number":127,"context_line":""},{"line_number":128,"context_line":"Operators will need to be cognizant of the risk and keep systems up to"},{"line_number":129,"context_line":"date and pro-actively rebuild executor servers and rotate credentials"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9ea0ef24_233bdb72","line":126,"in_reply_to":"62c8fa41_96bcce3b","updated":"2022-03-17 17:41:09.000000000","message":"Where the CD story becomes problematic for me is the exposure level in using Zuul for CD. With layers of security in theory our exposure is more limited. If we remove one layer of protection we become more exposed potentially.\n\nI think it may be worth expanding on the impact of CD on changes like this and how we can do better in general. Perhaps we can couple the ansible changes with stronger improvements in other areas to make CD more viable.","commit_id":"1685cf528f4ca158ef655c1c9585ff3024440a30"},{"author":{"_account_id":9311,"name":"Tristan Cacqueray","email":"tdecacqu@redhat.com","username":"tristanC"},"change_message_id":"34885eb40e6173d818906aef99733cde2fb93b14","unresolved":true,"context_lines":[{"line_number":123,"context_line":""},{"line_number":124,"context_line":"The ability to execute arbitrary code locally combined with a"},{"line_number":125,"context_line":"potential future local root exploit could allow an attacker to gain"},{"line_number":126,"context_line":"control of the Zuul system."},{"line_number":127,"context_line":""},{"line_number":128,"context_line":"Operators will need to be cognizant of the risk and keep systems up to"},{"line_number":129,"context_line":"date and pro-actively rebuild executor servers and rotate credentials"}],"source_content_type":"text/x-rst","patch_set":2,"id":"283228f6_f46ce1e4","line":126,"in_reply_to":"cf3b2a6a_85cf19d1","updated":"2022-03-17 17:48:10.000000000","message":"I agree this is a major concern, would a successful exploit get access to all the secrets known to Zuul?","commit_id":"1685cf528f4ca158ef655c1c9585ff3024440a30"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"d428dcf2dff5d42dc1d4f8f7f4c008d34cb56664","unresolved":true,"context_lines":[{"line_number":162,"context_line":"server and retrieve information, operators will need to ensure that no"},{"line_number":163,"context_line":"sensitive data are provided to the executors via the metadata service,"},{"line_number":164,"context_line":"and that it is not provided with any IAM profiles which should not be"},{"line_number":165,"context_line":"available to jobs."},{"line_number":166,"context_line":""},{"line_number":167,"context_line":"WinRM Credentials"},{"line_number":168,"context_line":"~~~~~~~~~~~~~~~~~"}],"source_content_type":"text/x-rst","patch_set":2,"id":"b5eec9b1_0ceaa702","line":165,"updated":"2022-03-17 16:48:40.000000000","message":"I think it was GKE that basically gives you full cluster root if you have access to this by default. Might be worth explicitly calling out known cases where this is a problem?","commit_id":"1685cf528f4ca158ef655c1c9585ff3024440a30"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"0e30d538e6ae62092c051c11f8dbb75e0ccdac9f","unresolved":false,"context_lines":[{"line_number":162,"context_line":"server and retrieve information, operators will need to ensure that no"},{"line_number":163,"context_line":"sensitive data are provided to the executors via the metadata service,"},{"line_number":164,"context_line":"and that it is not provided with any IAM profiles which should not be"},{"line_number":165,"context_line":"available to jobs."},{"line_number":166,"context_line":""},{"line_number":167,"context_line":"WinRM Credentials"},{"line_number":168,"context_line":"~~~~~~~~~~~~~~~~~"}],"source_content_type":"text/x-rst","patch_set":2,"id":"f961b26f_3df9ad03","line":165,"updated":"2022-03-17 17:22:18.000000000","message":"Yeah, wouldn\u0027t hurt to list specific examples.","commit_id":"1685cf528f4ca158ef655c1c9585ff3024440a30"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"0e30d538e6ae62092c051c11f8dbb75e0ccdac9f","unresolved":false,"context_lines":[{"line_number":196,"context_line":"these measures and we have relied entirely on bubblewrap to contain"},{"line_number":197,"context_line":"the fallout.  Removing the restricted environment does remove a layer"},{"line_number":198,"context_line":"from our defense in depth, but that layer may not be very effective in"},{"line_number":199,"context_line":"the first place."}],"source_content_type":"text/x-rst","patch_set":2,"id":"0a3b2eb3_22b27607","line":199,"updated":"2022-03-17 17:22:18.000000000","message":"I think there are more things we can do (and maybe that operators could do independent of us), but I think that most of the network and filesystem risks are actively manageable right now.  Even in most corporate environments, I don\u0027t think being able to winrm from the zuul executor is a big concern.  It seems like the local root exploit is the most concerning to you, so if we want to keep the \"no-untrusted-exec\" requirement, I don\u0027t think this is feasible.\n\n(Also, FWIW, I think we were happy to rely on only the restricted Ansible when we started v3 development; bwrap was a nice extra benefit, and then we realized it\u0027s actually doing most of the work.)","commit_id":"1685cf528f4ca158ef655c1c9585ff3024440a30"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"3af030a5df92a83bbda35109a9e99113c64388cc","unresolved":false,"context_lines":[{"line_number":196,"context_line":"these measures and we have relied entirely on bubblewrap to contain"},{"line_number":197,"context_line":"the fallout.  Removing the restricted environment does remove a layer"},{"line_number":198,"context_line":"from our defense in depth, but that layer may not be very effective in"},{"line_number":199,"context_line":"the first place."}],"source_content_type":"text/x-rst","patch_set":2,"id":"993190fb_8e857288","line":199,"updated":"2022-03-17 18:01:43.000000000","message":"I worry that assuming selinux/apparmor in Zuul assumes too much about deploymont.  We could run bwrap as a secondary user, as long as it doesn\u0027t entail giving the executor more privileges.","commit_id":"1685cf528f4ca158ef655c1c9585ff3024440a30"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"d428dcf2dff5d42dc1d4f8f7f4c008d34cb56664","unresolved":true,"context_lines":[{"line_number":196,"context_line":"these measures and we have relied entirely on bubblewrap to contain"},{"line_number":197,"context_line":"the fallout.  Removing the restricted environment does remove a layer"},{"line_number":198,"context_line":"from our defense in depth, but that layer may not be very effective in"},{"line_number":199,"context_line":"the first place."}],"source_content_type":"text/x-rst","patch_set":2,"id":"4cc5e343_fcd806cc","line":199,"updated":"2022-03-17 16:48:40.000000000","message":"Thinking about this from a defense in depth perspective are we able to leverage bwrap to restrict activity a bit more? For example could we restrict network connectivity to ssh/winrm/etc ports in untrusted contexts? Maybe limit system capabilites if we have room there as well?\n\nMostly thinking out loud here as I don\u0027t think we ever wanted to rely on any single defense mechanism and instead layer them to ensure as much safety as possible.\n\nMaybe Selinux or apparmor or similar would be useful here too?","commit_id":"1685cf528f4ca158ef655c1c9585ff3024440a30"},{"author":{"_account_id":4146,"name":"Clark Boylan","email":"cboylan@sapwetik.org","username":"cboylan"},"change_message_id":"3745e1648fa78656afabaec887981932874d4c4b","unresolved":false,"context_lines":[{"line_number":196,"context_line":"these measures and we have relied entirely on bubblewrap to contain"},{"line_number":197,"context_line":"the fallout.  Removing the restricted environment does remove a layer"},{"line_number":198,"context_line":"from our defense in depth, but that layer may not be very effective in"},{"line_number":199,"context_line":"the first place."}],"source_content_type":"text/x-rst","patch_set":2,"id":"ef846761_f165a784","line":199,"in_reply_to":"0a3b2eb3_22b27607","updated":"2022-03-17 17:41:09.000000000","message":"Yes, I think the major concerns I\u0027ve got are a privilege escalation on the executor and possibly a container escape given executor\u0027s zk access.\n\nThinking about zk access a bit I wonder if we can/should run the bwrap processes as a secondary user. And possibly something like selinux/apparmor can restrict file access further.\n\nI agree that existing deployments could probably do more alongside Zuul, but not needing to do that lifting separately in every deployment and instead share it via common zuul expectations and deployment tooling would go a long way I think.","commit_id":"1685cf528f4ca158ef655c1c9585ff3024440a30"},{"author":{"_account_id":9311,"name":"Tristan Cacqueray","email":"tdecacqu@redhat.com","username":"tristanC"},"change_message_id":"34885eb40e6173d818906aef99733cde2fb93b14","unresolved":true,"context_lines":[{"line_number":196,"context_line":"these measures and we have relied entirely on bubblewrap to contain"},{"line_number":197,"context_line":"the fallout.  Removing the restricted environment does remove a layer"},{"line_number":198,"context_line":"from our defense in depth, but that layer may not be very effective in"},{"line_number":199,"context_line":"the first place."}],"source_content_type":"text/x-rst","patch_set":2,"id":"e0088b50_b48487e9","line":199,"in_reply_to":"4cc5e343_fcd806cc","updated":"2022-03-17 17:48:10.000000000","message":"It is likely impossible for a container to change it\u0027s context after it started, so that may not be useful for executor already running a container. Also this may be difficult to setup as it depends a lot on the host system configuration.","commit_id":"1685cf528f4ca158ef655c1c9585ff3024440a30"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"e3c22c5b9e2ca5fb3c5acb0d3232567495a4679d","unresolved":false,"context_lines":[{"line_number":27,"context_line":"adjacent to playbooks."},{"line_number":28,"context_line":""},{"line_number":29,"context_line":"Currently, Zuul supports Ansible 2.9 as the latest version.  It is no"},{"line_number":30,"context_line":"longer maintained."},{"line_number":31,"context_line":""},{"line_number":32,"context_line":"More recent Ansible versions have significantly altered the internal"},{"line_number":33,"context_line":"plugin loading framework to accommodate Ansible Collections.  This"}],"source_content_type":"text/x-rst","patch_set":3,"id":"b90b6922_187e56c6","line":30,"updated":"2022-03-17 18:17:13.000000000","message":"Mea culpa!","commit_id":"66384d0d46403ec30cbf15577d152b918fb12fb5"},{"author":{"_account_id":9061,"name":"David Moreau Simard","email":"moi@dmsimard.com","username":"dmsimard"},"change_message_id":"272c7b68e292b703e0b62ff705ac6f67d249ac47","unresolved":true,"context_lines":[{"line_number":26,"context_line":"to verify that Ansible will not attempt to load any plugins which are"},{"line_number":27,"context_line":"adjacent to playbooks."},{"line_number":28,"context_line":""},{"line_number":29,"context_line":"Currently, Zuul supports Ansible 2.9 as the latest version.  It is no"},{"line_number":30,"context_line":"longer maintained."},{"line_number":31,"context_line":""},{"line_number":32,"context_line":"More recent Ansible versions have significantly altered the internal"},{"line_number":33,"context_line":"plugin loading framework to accommodate Ansible Collections.  This"}],"source_content_type":"text/x-rst","patch_set":3,"id":"478dccf8_7fd6d6f6","line":30,"range":{"start_line":29,"start_character":61,"end_line":30,"end_character":18},"updated":"2022-03-17 18:13:43.000000000","message":"This is not exactly accurate, the upstream end of life for Ansible 2.9 is currently planned for May 2022: https://groups.google.com/g/ansible-announce/c/kegIH5_okmg/","commit_id":"66384d0d46403ec30cbf15577d152b918fb12fb5"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"80d94ea6799943d112f53857af41774913480978","unresolved":false,"context_lines":[{"line_number":54,"context_line":""},{"line_number":55,"context_line":"At the same time, the restricted Ansible environment has proven to be"},{"line_number":56,"context_line":"a hindrance to using Zuul in many use cases, notably for lightweight"},{"line_number":57,"context_line":"jobs which don\u0027t need a remote node, or continuous deployment where a"},{"line_number":58,"context_line":"nested Ansible must be run in order to use certain features of"},{"line_number":59,"context_line":"Ansible."},{"line_number":60,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"bada3af5_6bbea3d6","line":57,"updated":"2022-03-17 18:11:37.000000000","message":"As a quick example: a git mirroring job that just needs to \"git push\" from the executor.  I\u0027m sure there are better ones, but most zero-node jobs that actually do anything need to have trusted playbooks at this point.","commit_id":"66384d0d46403ec30cbf15577d152b918fb12fb5"},{"author":{"_account_id":9311,"name":"Tristan Cacqueray","email":"tdecacqu@redhat.com","username":"tristanC"},"change_message_id":"34885eb40e6173d818906aef99733cde2fb93b14","unresolved":true,"context_lines":[{"line_number":54,"context_line":""},{"line_number":55,"context_line":"At the same time, the restricted Ansible environment has proven to be"},{"line_number":56,"context_line":"a hindrance to using Zuul in many use cases, notably for lightweight"},{"line_number":57,"context_line":"jobs which don\u0027t need a remote node, or continuous deployment where a"},{"line_number":58,"context_line":"nested Ansible must be run in order to use certain features of"},{"line_number":59,"context_line":"Ansible."},{"line_number":60,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"862e2204_05032f99","line":57,"updated":"2022-03-17 17:48:10.000000000","message":"Is it really a hindrance for lightweight jobs, do we have example?","commit_id":"66384d0d46403ec30cbf15577d152b918fb12fb5"},{"author":{"_account_id":9061,"name":"David Moreau Simard","email":"moi@dmsimard.com","username":"dmsimard"},"change_message_id":"272c7b68e292b703e0b62ff705ac6f67d249ac47","unresolved":true,"context_lines":[{"line_number":84,"context_line":""},{"line_number":85,"context_line":"* Updating or removing tests which verify the custom plugin behavior."},{"line_number":86,"context_line":""},{"line_number":87,"context_line":"* Adding support for Ansible 5.4."},{"line_number":88,"context_line":""},{"line_number":89,"context_line":"* Documenting the security considerations described below."},{"line_number":90,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"d038057c_9bb617f6","line":87,"range":{"start_line":87,"start_character":0,"end_line":87,"end_character":33},"updated":"2022-03-17 18:13:43.000000000","message":"FWIW: Ansible 5.5.0 was released last week -- the current pace of release for the \u0027ansible\u0027 package is a minor release every 3 weeks and a major release every 6 months (which aligns with the 6 months cycle of ansible-core).\n\nAnsible 6.0.0 is currently expected in May 2022 which aligns with the new release of ansible-core planned at the same time, 2.13.","commit_id":"66384d0d46403ec30cbf15577d152b918fb12fb5"},{"author":{"_account_id":9311,"name":"Tristan Cacqueray","email":"tdecacqu@redhat.com","username":"tristanC"},"change_message_id":"34885eb40e6173d818906aef99733cde2fb93b14","unresolved":true,"context_lines":[{"line_number":84,"context_line":""},{"line_number":85,"context_line":"* Updating or removing tests which verify the custom plugin behavior."},{"line_number":86,"context_line":""},{"line_number":87,"context_line":"* Adding support for Ansible 5.4."},{"line_number":88,"context_line":""},{"line_number":89,"context_line":"* Documenting the security considerations described below."},{"line_number":90,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"d8ddeec8_d004f6f4","line":87,"updated":"2022-03-17 17:48:10.000000000","message":"Should we consider leveraging execution environment https://docs.ansible.com/automation-controller/latest/html/userguide/execution_environments.html?","commit_id":"66384d0d46403ec30cbf15577d152b918fb12fb5"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"80d94ea6799943d112f53857af41774913480978","unresolved":false,"context_lines":[{"line_number":84,"context_line":""},{"line_number":85,"context_line":"* Updating or removing tests which verify the custom plugin behavior."},{"line_number":86,"context_line":""},{"line_number":87,"context_line":"* Adding support for Ansible 5.4."},{"line_number":88,"context_line":""},{"line_number":89,"context_line":"* Documenting the security considerations described below."},{"line_number":90,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"b9b2aaa5_0ff04980","line":87,"updated":"2022-03-17 18:11:37.000000000","message":"That could be an interesting way to use Ansible within Zuul, though since it\u0027s essentially building and running a container it\u0027s sort of parallel to what we\u0027re doing with bwrap.  We might even consider it a replacement for bwrap at some point, but I think we can defer that for now since I think fundamentally the security issues are the same.","commit_id":"66384d0d46403ec30cbf15577d152b918fb12fb5"},{"author":{"_account_id":9061,"name":"David Moreau Simard","email":"moi@dmsimard.com","username":"dmsimard"},"change_message_id":"272c7b68e292b703e0b62ff705ac6f67d249ac47","unresolved":true,"context_lines":[{"line_number":84,"context_line":""},{"line_number":85,"context_line":"* Updating or removing tests which verify the custom plugin behavior."},{"line_number":86,"context_line":""},{"line_number":87,"context_line":"* Adding support for Ansible 5.4."},{"line_number":88,"context_line":""},{"line_number":89,"context_line":"* Documenting the security considerations described below."},{"line_number":90,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"60fc1edb_95ea4f9d","line":87,"in_reply_to":"d8ddeec8_d004f6f4","updated":"2022-03-17 18:13:43.000000000","message":"Execution environments are container images built with ansible-builder (https://github.com/ansible/ansible-builder) and they\u0027re meant to contain particular versions of ansible-core, ansible-runner as well as Ansible collections necessary for the users\u0027 playbooks.\n\nIf you\u0027re curious to see what it looks like in practice, I have a simple example here which adds the ansible and ara packages on top of a \"base\" execution environment: https://github.com/ansible-community/images/tree/main/execution-environments\n\nansible-navigator is a TUI that, amongst other things, knows about execution environments and can run playbooks with them.","commit_id":"66384d0d46403ec30cbf15577d152b918fb12fb5"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"2ed94eb7633928f16a621c9f47b61aad79a0e2b5","unresolved":false,"context_lines":[{"line_number":89,"context_line":"* Documenting the security considerations described below."},{"line_number":90,"context_line":""},{"line_number":91,"context_line":"This has significant impact to operators and so will be communicated"},{"line_number":92,"context_line":"with a Zuul major version increase."},{"line_number":93,"context_line":""},{"line_number":94,"context_line":"Security Considerations"},{"line_number":95,"context_line":"-----------------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"23d15371_736e8ee4","line":92,"updated":"2022-03-21 22:01:44.000000000","message":"I wasn\u0027t aware of list-tasks, it looks interesting, but after a quick check, it looks like it doesn\u0027t recurse into include_role.","commit_id":"66384d0d46403ec30cbf15577d152b918fb12fb5"},{"author":{"_account_id":9311,"name":"Tristan Cacqueray","email":"tdecacqu@redhat.com","username":"tristanC"},"change_message_id":"34885eb40e6173d818906aef99733cde2fb93b14","unresolved":true,"context_lines":[{"line_number":89,"context_line":"* Documenting the security considerations described below."},{"line_number":90,"context_line":""},{"line_number":91,"context_line":"This has significant impact to operators and so will be communicated"},{"line_number":92,"context_line":"with a Zuul major version increase."},{"line_number":93,"context_line":""},{"line_number":94,"context_line":"Security Considerations"},{"line_number":95,"context_line":"-----------------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"97c6b9bf_639beb7d","line":92,"updated":"2022-03-17 17:48:10.000000000","message":"I wonder if we could implement the restriction using static analysis instead. Before running an unstrusted playbook, perhaps we could manually inspect its content to iterate over its tasks and prevent forbidden usage? When a task is not iterable, for example if it includes a dynamically generated tasks, then we could fail by default?","commit_id":"66384d0d46403ec30cbf15577d152b918fb12fb5"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"80d94ea6799943d112f53857af41774913480978","unresolved":false,"context_lines":[{"line_number":89,"context_line":"* Documenting the security considerations described below."},{"line_number":90,"context_line":""},{"line_number":91,"context_line":"This has significant impact to operators and so will be communicated"},{"line_number":92,"context_line":"with a Zuul major version increase."},{"line_number":93,"context_line":""},{"line_number":94,"context_line":"Security Considerations"},{"line_number":95,"context_line":"-----------------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"6c64e761_abe22e0a","line":92,"updated":"2022-03-17 18:11:37.000000000","message":"That seems like a distinct possibility.  We might miss something, but we miss things with our current system too.\n\nI worry that we might spend a long time figuring out all the possible ways of including tasks.  :)","commit_id":"66384d0d46403ec30cbf15577d152b918fb12fb5"},{"author":{"_account_id":9311,"name":"Tristan Cacqueray","email":"tdecacqu@redhat.com","username":"tristanC"},"change_message_id":"9da6a52c30aa9c2c0be6bb7617df6c4c4a1ddfb1","unresolved":false,"context_lines":[{"line_number":89,"context_line":"* Documenting the security considerations described below."},{"line_number":90,"context_line":""},{"line_number":91,"context_line":"This has significant impact to operators and so will be communicated"},{"line_number":92,"context_line":"with a Zuul major version increase."},{"line_number":93,"context_line":""},{"line_number":94,"context_line":"Security Considerations"},{"line_number":95,"context_line":"-----------------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"86179b5d_1e7c01d6","line":92,"in_reply_to":"6c64e761_abe22e0a","updated":"2022-03-17 18:27:39.000000000","message":"Are the only ways to include tasks: roles, include, include_tasks, import_tasks and  import_playbooks ?\n\nI think the main issue is implementing the semantic of jinja variable precedences, but that can be done.","commit_id":"66384d0d46403ec30cbf15577d152b918fb12fb5"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"e2f28ec4a84a2f0d9d696e3e14d968ce4bd648f2","unresolved":true,"context_lines":[{"line_number":89,"context_line":"* Documenting the security considerations described below."},{"line_number":90,"context_line":""},{"line_number":91,"context_line":"This has significant impact to operators and so will be communicated"},{"line_number":92,"context_line":"with a Zuul major version increase."},{"line_number":93,"context_line":""},{"line_number":94,"context_line":"Security Considerations"},{"line_number":95,"context_line":"-----------------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"c97f7c8a_67344dcf","line":92,"in_reply_to":"86179b5d_1e7c01d6","updated":"2022-03-18 13:01:28.000000000","message":"I don\u0027t think this is feasible for us zuul maintainers. Such a static analysis sounds like it would require an entire project with several people maintaining this. To me this sounds like even more work like we have to do with the current approach.","commit_id":"66384d0d46403ec30cbf15577d152b918fb12fb5"},{"author":{"_account_id":9311,"name":"Tristan Cacqueray","email":"tdecacqu@redhat.com","username":"tristanC"},"change_message_id":"84d56ee8114ca11dc58eb298217dde8d3992e2d4","unresolved":true,"context_lines":[{"line_number":89,"context_line":"* Documenting the security considerations described below."},{"line_number":90,"context_line":""},{"line_number":91,"context_line":"This has significant impact to operators and so will be communicated"},{"line_number":92,"context_line":"with a Zuul major version increase."},{"line_number":93,"context_line":""},{"line_number":94,"context_line":"Security Considerations"},{"line_number":95,"context_line":"-----------------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"66d0ea9d_d2c60dc0","line":92,"in_reply_to":"c97f7c8a_67344dcf","updated":"2022-03-18 18:46:06.000000000","message":"Perhaps this does not have to be implemented within zuul, and maybe that could be useful to others. For example, the `--list-task` argument of ansible-playbook seems promising, but it might need some tweak to be useful here.\n\nIf such a static analysis tool existed, then it seems like we would be able to use ansible without modification, while preserving the restriction for untrusted playbook.","commit_id":"66384d0d46403ec30cbf15577d152b918fb12fb5"},{"author":{"_account_id":9311,"name":"Tristan Cacqueray","email":"tdecacqu@redhat.com","username":"tristanC"},"change_message_id":"34885eb40e6173d818906aef99733cde2fb93b14","unresolved":true,"context_lines":[{"line_number":117,"context_line":"have access to more files which are made available to the bubblewrap"},{"line_number":118,"context_line":"environment than before.  See `WinRM Credentials` below for one"},{"line_number":119,"context_line":"specific case."},{"line_number":120,"context_line":""},{"line_number":121,"context_line":"Local Code Execution"},{"line_number":122,"context_line":"~~~~~~~~~~~~~~~~~~~~"},{"line_number":123,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"8ad5a897_1771bf03","line":120,"updated":"2022-03-17 17:48:10.000000000","message":"Perhaps we should consider CPU and memory resources?","commit_id":"66384d0d46403ec30cbf15577d152b918fb12fb5"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"80d94ea6799943d112f53857af41774913480978","unresolved":false,"context_lines":[{"line_number":117,"context_line":"have access to more files which are made available to the bubblewrap"},{"line_number":118,"context_line":"environment than before.  See `WinRM Credentials` below for one"},{"line_number":119,"context_line":"specific case."},{"line_number":120,"context_line":""},{"line_number":121,"context_line":"Local Code Execution"},{"line_number":122,"context_line":"~~~~~~~~~~~~~~~~~~~~"},{"line_number":123,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"29e45d92_0c0b45b6","line":120,"updated":"2022-03-17 18:11:37.000000000","message":"Yes; the ability to run more zero-node jobs will probably increase CPU and memory pressure.","commit_id":"66384d0d46403ec30cbf15577d152b918fb12fb5"},{"author":{"_account_id":9311,"name":"Tristan Cacqueray","email":"tdecacqu@redhat.com","username":"tristanC"},"change_message_id":"4ba063a82b9001fd8eff5f526c2595a375fac9ec","unresolved":true,"context_lines":[{"line_number":117,"context_line":"have access to more files which are made available to the bubblewrap"},{"line_number":118,"context_line":"environment than before.  See `WinRM Credentials` below for one"},{"line_number":119,"context_line":"specific case."},{"line_number":120,"context_line":""},{"line_number":121,"context_line":"Local Code Execution"},{"line_number":122,"context_line":"~~~~~~~~~~~~~~~~~~~~"},{"line_number":123,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"7c72165f_18d7e6bc","line":120,"in_reply_to":"29e45d92_0c0b45b6","updated":"2022-03-17 21:48:06.000000000","message":"What about resources exhaustion, is there something we should do to prevent one job to hog all the resources?","commit_id":"66384d0d46403ec30cbf15577d152b918fb12fb5"},{"author":{"_account_id":1,"name":"James E. Blair","email":"jim@acmegating.com","username":"corvus"},"change_message_id":"eb152a37ca9ebe13c509b2f77102690a4dd80b6a","unresolved":false,"context_lines":[{"line_number":132,"context_line":"Local Network Access"},{"line_number":133,"context_line":"~~~~~~~~~~~~~~~~~~~~"},{"line_number":134,"context_line":""},{"line_number":135,"context_line":"If the Zuul executor is run in a network environment which is trusted,"},{"line_number":136,"context_line":"then users may be able to take advantage of that to access restricted"},{"line_number":137,"context_line":"systems."},{"line_number":138,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"19ba2061_f5577caa","line":135,"updated":"2022-03-17 17:23:30.000000000","message":"Good point.  This is an excellent example of how we may think we are more protected than we really are.","commit_id":"66384d0d46403ec30cbf15577d152b918fb12fb5"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"7d76709722ab56cdb5322ad979bbf5fd82848469","unresolved":true,"context_lines":[{"line_number":132,"context_line":"Local Network Access"},{"line_number":133,"context_line":"~~~~~~~~~~~~~~~~~~~~"},{"line_number":134,"context_line":""},{"line_number":135,"context_line":"If the Zuul executor is run in a network environment which is trusted,"},{"line_number":136,"context_line":"then users may be able to take advantage of that to access restricted"},{"line_number":137,"context_line":"systems."},{"line_number":138,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"017a236b_9b60c544","line":135,"updated":"2022-03-17 17:04:36.000000000","message":"Maybe we can note that e.g. the ansible uri module is already now available for local execution so everything that can be done with the uri module is already possible without this change.","commit_id":"66384d0d46403ec30cbf15577d152b918fb12fb5"},{"author":{"_account_id":16068,"name":"Tobias Henkel","email":"tobias.henkel@bmw.de","username":"tobias.henkel"},"change_message_id":"7d76709722ab56cdb5322ad979bbf5fd82848469","unresolved":true,"context_lines":[{"line_number":154,"context_line":"Cloud Metadata"},{"line_number":155,"context_line":"~~~~~~~~~~~~~~"},{"line_number":156,"context_line":""},{"line_number":157,"context_line":"A special case of local network access is the ability to access"},{"line_number":158,"context_line":"metadata servers if the executor is running in a cloud environment."},{"line_number":159,"context_line":""},{"line_number":160,"context_line":"Because a Zuul job would be able to open a connection to the metadata"}],"source_content_type":"text/x-rst","patch_set":3,"id":"8a9566da_7f8db4de","line":157,"updated":"2022-03-17 17:04:36.000000000","message":"This is already possible via the uri module (this is unrestricted).","commit_id":"66384d0d46403ec30cbf15577d152b918fb12fb5"}]}