)]}'
{"/COMMIT_MSG":[{"author":{"_account_id":7118,"name":"Ian Wienand","email":"iwienand@redhat.com","username":"iwienand"},"change_message_id":"baeb5aa8116e5941be58175ba9910983321e8b14","unresolved":true,"context_lines":[{"line_number":7,"context_line":"Test if DNS is up"},{"line_number":8,"context_line":""},{"line_number":9,"context_line":"In FIPS enabled jobs, we reboot the node after FIPS is enabled."},{"line_number":10,"context_line":"Sometimes, stack.sh runs before the local DNS server is up, resulting"},{"line_number":11,"context_line":"in errors when we try to install packages."},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"The added check makes sure DNS is up."}],"source_content_type":"text/x-gerrit-commit-message","patch_set":3,"id":"a0f874fd_5273fffa","line":10,"range":{"start_line":10,"start_character":32,"end_line":10,"end_character":58},"updated":"2022-05-19 00:37:16.000000000","message":"This is an opendev gate issue since the nodes use unbound?  The problem being that unbound isn\u0027t responding before devstack starts?\n\nI\u0027m not sure we\u0027ve seen this in other jobs, but that\u0027s probably because after node boot regular jobs do a lot of pre-work that would hide any quick startup race or ordering issues.\n\nIf that is the problem, perhaps we should be looking more closely at the nodepool side -- it seems likely the systemd service bringing this up should not let boot progress until unbound is listening properly.  It not doing that is more likely an oversight in unbound installation than desired behavior.","commit_id":"5fada16514a6074d61101576d86cb9070af30609"}],"/PATCHSET_LEVEL":[{"author":{"_account_id":11628,"name":"Michael Johnson","email":"johnsomor@gmail.com","username":"johnsom"},"change_message_id":"bcfc665a486518167de11dfa949e8adf201244be","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":2,"id":"8927d311_030a2010","updated":"2022-05-17 21:06:49.000000000","message":"I think we should have an upper bound on the retries.","commit_id":"01a8a889dec86eea89bc3ba0e3b4061cd6bc7be9"},{"author":{"_account_id":7118,"name":"Ian Wienand","email":"iwienand@redhat.com","username":"iwienand"},"change_message_id":"6c8cff78b35f29394faeb1714d2c81d30390bfb3","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"bbfc31ed_403231bf","updated":"2022-05-19 23:25:02.000000000","message":"\u003e Patch Set 3:\n\u003e \n\u003e Ade Lee wrote:\n\u003e \n\u003e \u003e Whatever sets up the unbound server shouldn\u0027t proceed till\n\u003e \u003e we know DNS resolution works, to prevent race conditions.\n\u003e \n\u003e It\u0027s really more that the init shouldn\u0027t treat the system as up until DNS resolution is working (unbound is set up before the reboot, it\u0027s the init at boot currently treating unbound\u0027s start completion as optional that contributes to this problem).\n\nTo be concrete, we do publish the images @\n\nhttps://nb0\u003c1|2|3\u003e.opendev.org/images/\n\nand most of the unbound setup is done @ https://opendev.org/openstack/project-config/src/branch/master/nodepool/elements/nodepool-base/finalise.d/89-boot-settings\n\nI *think* we are just setting up unbound config and then using the service file from the distro\n\nI found at least one bug that seems to describe this, e.g. https://github.com/NLnetLabs/unbound/issues/296\n","commit_id":"5fada16514a6074d61101576d86cb9070af30609"},{"author":{"_account_id":13252,"name":"Dr. Jens Harbott","display_name":"Jens Harbott (frickler)","email":"frickler@offenerstapel.de","username":"jrosenboom"},"change_message_id":"225ee375ea8b9ecbb4ec82536449949ef1635ae5","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"9a9d4187_47f1dfd5","updated":"2022-05-19 12:16:50.000000000","message":"\u003e Patch Set 3:\n\u003e \n\u003e Dr. Jens Harbott wrote:\n\u003e \n\u003e \u003e IMO this doesn\u0027t belong here. Devstack should be able to expect a working host\n\u003e \u003e setup. If you reboot the host in the fips playbook, put that check there.\n\u003e \n\u003e The questions then become: What criteria should fips-setup use to determine when a host is \"working\" after reboot? If the definition of \"working\" from the perspective of fips-setup in zuul-jobs is also \"I can resolve records via DNS\" then what DNS record(s) should it attempt to query?\n\nThe answer to the last question should be the same as the answer proposed here. But I also agree with ianw\u0027s comment that there may be an even deeper issue with the way we set up unbound vs. system startup.","commit_id":"5fada16514a6074d61101576d86cb9070af30609"},{"author":{"_account_id":9914,"name":"Ade Lee","email":"alee@redhat.com","username":"alee"},"change_message_id":"aaeb85520934be3c8f0b596ecb2bb425cc874f59","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":3,"id":"fc8a24e1_d5d8b973","updated":"2022-05-19 15:27:14.000000000","message":"\u003e Patch Set 3:\n\u003e \n\u003e Dr. Jens Harbott wrote:\n\u003e \n\u003e \u003e The answer to the last question should be the same as the answer proposed here.\n\u003e \n\u003e With my zuul-jobs reviewer hat on, I\u0027m not sure having a generic role hard-code an assumption (or even default to it) that systems should try resolving opendev.org is a good idea. At a minimum, it would need some escape hatch for jobs being run in environments where nodes may not have access to a fully open recursive resolver, either to skip the check or substitute an alternative record which is known to be resolvable by those nodes... but beyond that, should the fips-setup role necessarily even depend on having working DNS resolution?\n\nSeems to me that the fips-setup-role has really nothing to do with whether DNS resolution is working or not.  The only reason it comes up here is because the role happens to reboot the node.\n\nTo Jen\u0027s point though, devstack has every right to expect a \"working system\" to begin with - and while having a check here would be very useful and informative, we don\u0027t check all the many other things that make up a \"working system\".\n\nIan\u0027s suggestion makes the most sense to me.  Whatever sets up the unbound server shouldn\u0027t proceed till we know DNS resolution works, to prevent race conditions.","commit_id":"5fada16514a6074d61101576d86cb9070af30609"},{"author":{"_account_id":11628,"name":"Michael Johnson","email":"johnsomor@gmail.com","username":"johnsom"},"change_message_id":"75e012e559806e14dc1d4e7f9c013b0ed07ff2ff","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"cd3b2172_951e3b7d","updated":"2022-05-18 15:03:45.000000000","message":"FYI, the test patch shows this works:\nhttps://zuul.opendev.org/t/openstack/build/654645cad8d444c48de7595ed5f5b1c2/log/job-output.txt#2062\n\nIt required one retry (5 second interval) before the DNS resolved.","commit_id":"5fada16514a6074d61101576d86cb9070af30609"},{"author":{"_account_id":13252,"name":"Dr. Jens Harbott","display_name":"Jens Harbott (frickler)","email":"frickler@offenerstapel.de","username":"jrosenboom"},"change_message_id":"dfaee1e7e9ba0f0b9a874b552c3c6b676f9198bd","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"dbb9d621_21d43639","updated":"2022-05-18 08:50:39.000000000","message":"IMO this doesn\u0027t belong here. Devstack should be able to expect a working host setup. If you reboot the host in the fips playbook, put that check there.","commit_id":"5fada16514a6074d61101576d86cb9070af30609"},{"author":{"_account_id":11628,"name":"Michael Johnson","email":"johnsomor@gmail.com","username":"johnsom"},"change_message_id":"135a2827464366569b110459a6900313b11ad19b","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"41912281_7f1b9405","in_reply_to":"dbb9d621_21d43639","updated":"2022-05-18 14:58:52.000000000","message":"I feel the opposite. Currently if DNS isn\u0027t working, devstack continues to run until it can\u0027t find a package. This error, that a package isn\u0027t found is misleading (bad mirror, bad repo, etc.). I think it\u0027s better to have an explicit test that calls out the true problem early.","commit_id":"5fada16514a6074d61101576d86cb9070af30609"}],"stack.sh":[{"author":{"_account_id":11628,"name":"Michael Johnson","email":"johnsomor@gmail.com","username":"johnsom"},"change_message_id":"bcfc665a486518167de11dfa949e8adf201244be","unresolved":true,"context_lines":[{"line_number":282,"context_line":""},{"line_number":283,"context_line":"# ensure that dns is available before trying to install packages"},{"line_number":284,"context_line":"if [[ \"$OFFLINE\" !\u003d \"True\" ]]; then"},{"line_number":285,"context_line":"    while ! nslookup opendev.org; do"},{"line_number":286,"context_line":"        echo \"checking if dns is up by looking up opendev.org\""},{"line_number":287,"context_line":"        sleep 2"},{"line_number":288,"context_line":"    done"}],"source_content_type":"text/x-sh","patch_set":2,"id":"ace692d1_0eb77dc3","line":285,"updated":"2022-05-17 21:06:49.000000000","message":"I think there is an easier way to do this.\n\nnslookup -retry\u003d60 opendev.org\n\nThis will give 60 connect attempts with a 5 second delay (default), so it will try for about five minutes and give up.","commit_id":"01a8a889dec86eea89bc3ba0e3b4061cd6bc7be9"}]}