)]}'
{"/PATCHSET_LEVEL":[{"author":{"_account_id":36080,"name":"Erkin Mussurmankulov","display_name":"Eric","email":"erkin.mussurmankulov@ps.kz","username":"mongoose404","status":"PS Cloud services employee"},"change_message_id":"c493f1ce758f11b1c420596ca9f3e9364c9c6528","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"c9f6aad8_fc06709e","updated":"2026-06-17 17:13:36.000000000","message":"recheck","commit_id":"27f5e6f76f1457001658e35d5cab65eb987cda59"},{"author":{"_account_id":36080,"name":"Erkin Mussurmankulov","display_name":"Eric","email":"erkin.mussurmankulov@ps.kz","username":"mongoose404","status":"PS Cloud services employee"},"change_message_id":"ce4e71f159f0fade1dc81a1b6d0a1fa5df8336ac","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":4,"id":"511cbe6e_104f15d6","updated":"2026-07-10 10:54:05.000000000","message":"recheck","commit_id":"d5f9827d1888a7f912ff577f67de6268e4f20ae2"},{"author":{"_account_id":36080,"name":"Erkin Mussurmankulov","display_name":"Eric","email":"erkin.mussurmankulov@ps.kz","username":"mongoose404","status":"PS Cloud services employee"},"change_message_id":"669e271ffcf405b85395cf64c98007d337f25c9e","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":6,"id":"e251423f_cfaed2fc","updated":"2026-08-11 16:54:08.000000000","message":"@kispear@gmail.com\n\nWithout considering that the default 5GB GUEST_IMAGE_SIZE wasn\u0027t enough, everything else goes smoothly with this MR. (I don\u0027t understand how tests didn\u0027t show this previously, but it doesn\u0027t matter; a mere increase to 6GB works)\n\n```\nAsync summary\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\nTime spent in the background minus waits: 664 sec\nElapsed time: 827 sec\nTime if we did everything serially: 1491 sec\nSpeedup:  1.8029\n```\n\nAlso from another report with backup image building:\n```\nTime spent in the background minus waits: 903 sec\nElapsed time: 964 sec\nTime if we did everything serially: 1867 sec\nSpeedup:  1.93672\n```\n\n31 minutes for mariadb basic\n33 minutes for mysql basic\n34 minutes for postgresql basic\n\nAnd all this during busy evening hours in Zuul. I bet in the morning these numbers should be better.\n\nI consider this a HUGE achievement 😄🥳\n\nWe have very little room for optimization left, but we can move it a little bit further, though.\n\n\nThere is still an issue left with this MR: if the `create_registry_and_guest_image` task fails, the entire CI pipeline freezes on the line `async_wait create_registry_and_guest_image`. Currently, I\u0027m investigating how we can avoid that.","commit_id":"596472916917d406d886c8ed445242314b5233fe"},{"author":{"_account_id":5733,"name":"Kieran Spear","email":"kispear@gmail.com","username":"kspear"},"change_message_id":"8e2089d3f945d90bfdc457c88d3882a34b84b401","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":10,"id":"84746779_99b3dada","updated":"2026-08-12 04:04:22.000000000","message":"I looked a bit into the cinder failures, and I think the unconditional docker install is breaking guest egress? [guestagent-log](https://zuul.opendev.org/t/openstack/build/98dd16ef8ce64fb9a13a4e909915611a/log/controller/logs/guest-agent-logs/79e638a6-6dbf-4d01-9880-ed009cfa6f04/log/trove/trove-guestagent_log.txt#808)\n\nThis matters more for these jobs since TROVE_ENABLE_LOCAL_REGISTRY\u003d\u003dFalse so they pull from quay.io.","commit_id":"1ee25922798ba8354d1557cb262a5e1e2ac9359c"},{"author":{"_account_id":36080,"name":"Erkin Mussurmankulov","display_name":"Eric","email":"erkin.mussurmankulov@ps.kz","username":"mongoose404","status":"PS Cloud services employee"},"change_message_id":"ec8858e06b53e5b5b742388eb0871ece2f019736","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":10,"id":"c209ff9e_67dc46d4","in_reply_to":"84746779_99b3dada","updated":"2026-08-12 21:04:23.000000000","message":"This patch works well if [this MR](https://review.opendev.org/c/openstack/trove/+/1000732) is added as Depends-on.\n__________\n\nAbout the missing network: a kind of detective story here.\n\nIn my local DevStack, iptables had a default `FORWARD DROP` rule; it was causing a lack of egress traffic in the guest instances. Also, the same was in Zuul. \nSo after some deduction, I found the root cause: my new Docker installation code in async mode was missing the config:\n```\n{\n    \"bridge\": \"none\",\n    \"ip-forward\": false,\n    \"iptables\": false\n}\n```\nWhich is present in the `trovestack install-docker`. Because of that, `apt docker install` replaced the default `FORWARD ACCEPT` with `FORWARD DROP`.\n\n________\n\nI can do a final polishing now, and then this MR will be ready for review.","commit_id":"1ee25922798ba8354d1557cb262a5e1e2ac9359c"},{"author":{"_account_id":36080,"name":"Erkin Mussurmankulov","display_name":"Eric","email":"erkin.mussurmankulov@ps.kz","username":"mongoose404","status":"PS Cloud services employee"},"change_message_id":"9d70e1560f66349ad11a0908799d911d4eb3859c","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":15,"id":"85a90224_30080a4a","updated":"2026-08-12 12:12:35.000000000","message":"Looks like we implemented image embedding just in time.\nIt seems Neutron routing doesn\u0027t work as it did before; probably this is caused by the async installation process.\nSo as a result, guests don\u0027t have external connectivity. That\u0027s why the Cinder tests are failing: they require connectivity to quay.io because the guest agent still runs `docker pull` at startup.\n\nIronically, with image embedding, we don\u0027t need external connectivity at all.\nJust checking whether the image exists in the local Docker image store and skipping `docker pull` fixes the issue.\nI will create a separate MR for this issue.\n\nBut I will also debug the installation process, so the external connectivity should work as well.","commit_id":"d3abd14549bdd49d532ea4ecd7b90fab42907255"},{"author":{"_account_id":36080,"name":"Erkin Mussurmankulov","display_name":"Eric","email":"erkin.mussurmankulov@ps.kz","username":"mongoose404","status":"PS Cloud services employee"},"change_message_id":"69a13d91bb37189e492bc48937670943eae4d02a","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":23,"id":"7a0424ce_2d2aba9d","updated":"2026-08-21 07:55:31.000000000","message":"recheck","commit_id":"8ca879e35e7db2a6e01b8e55278a96fdc552957e"}],"devstack/plugin.sh":[{"author":{"_account_id":5733,"name":"Kieran Spear","email":"kispear@gmail.com","username":"kspear"},"change_message_id":"8e2089d3f945d90bfdc457c88d3882a34b84b401","unresolved":true,"context_lines":[{"line_number":858,"context_line":"        # asynchronously"},{"line_number":859,"context_line":"        async_runfunc add_image_to_glance"},{"line_number":860,"context_line":""},{"line_number":861,"context_line":"        init_trove_db"},{"line_number":862,"context_line":"        config_nova_keypair"},{"line_number":863,"context_line":"        config_cinder_volume_type"},{"line_number":864,"context_line":"        config_mgmt_security_group"}],"source_content_type":"text/x-sh","patch_set":4,"id":"97560f3c_1699b1f4","line":861,"updated":"2026-08-12 04:04:22.000000000","message":"Can this second call be removed?","commit_id":"d5f9827d1888a7f912ff577f67de6268e4f20ae2"},{"author":{"_account_id":36080,"name":"Erkin Mussurmankulov","display_name":"Eric","email":"erkin.mussurmankulov@ps.kz","username":"mongoose404","status":"PS Cloud services employee"},"change_message_id":"cde163b29653b874cb4b4a74a804f27c7d2903fd","unresolved":false,"context_lines":[{"line_number":858,"context_line":"        # asynchronously"},{"line_number":859,"context_line":"        async_runfunc add_image_to_glance"},{"line_number":860,"context_line":""},{"line_number":861,"context_line":"        init_trove_db"},{"line_number":862,"context_line":"        config_nova_keypair"},{"line_number":863,"context_line":"        config_cinder_volume_type"},{"line_number":864,"context_line":"        config_mgmt_security_group"}],"source_content_type":"text/x-sh","patch_set":4,"id":"4f127440_620fe962","line":861,"in_reply_to":"97560f3c_1699b1f4","updated":"2026-08-12 08:52:25.000000000","message":"Done, thanks.\nLooks like a typo.\n\nA will get the MVP first, and then do the final polishing.","commit_id":"d5f9827d1888a7f912ff577f67de6268e4f20ae2"},{"author":{"_account_id":39300,"name":"Olessya Khussainova","display_name":"Olessya Khussainova","email":"olessya.khussainova@ps.kz","username":"Cellofun"},"change_message_id":"e2134caa289c7518301e12b5d701af9d903b18d7","unresolved":true,"context_lines":[{"line_number":546,"context_line":"    echo \"Add the image to glance\""},{"line_number":547,"context_line":"    image_name\u003d${TROVE_IMAGE_NAME:-\"trove-guest-${TROVE_IMAGE_OS}-${TROVE_IMAGE_OS_RELEASE}\"}"},{"line_number":548,"context_line":"    image_file\u003d${TROVE_IMAGE_FILE:-\"$HOME/images/${image_name}.qcow2\"}"},{"line_number":549,"context_line":"    glance_image_id\u003d$(openstack --os-cloud trove \\"},{"line_number":550,"context_line":"      image create \"$image_name\" \\"},{"line_number":551,"context_line":"      --disk-format qcow2 --container-format bare \\"},{"line_number":552,"context_line":"      --tag trove \\"}],"source_content_type":"text/x-sh","patch_set":20,"id":"899cb07c_f9d4f8b1","line":549,"updated":"2026-08-18 12:09:00.000000000","message":"Looks like errors may be silently ignored here. `async_inner` calls the worker as `if $*`, so `set -e` may not stop the function if `openstack image create` fails. The following commands could then make the function return `0`, and `async_wait` would report success even though the image was not uploaded.\n\nMaybe it would be safer to propagate errors explicitly here, for example with `|| return $?`. The same may apply to commands in `create_registry_container`.","commit_id":"9513a2afe8ffd9016e43e104466bb2dd587ee268"},{"author":{"_account_id":36080,"name":"Erkin Mussurmankulov","display_name":"Eric","email":"erkin.mussurmankulov@ps.kz","username":"mongoose404","status":"PS Cloud services employee"},"change_message_id":"ea5001841a3cf0c06d77bf91a35675fdc25a774f","unresolved":false,"context_lines":[{"line_number":546,"context_line":"    echo \"Add the image to glance\""},{"line_number":547,"context_line":"    image_name\u003d${TROVE_IMAGE_NAME:-\"trove-guest-${TROVE_IMAGE_OS}-${TROVE_IMAGE_OS_RELEASE}\"}"},{"line_number":548,"context_line":"    image_file\u003d${TROVE_IMAGE_FILE:-\"$HOME/images/${image_name}.qcow2\"}"},{"line_number":549,"context_line":"    glance_image_id\u003d$(openstack --os-cloud trove \\"},{"line_number":550,"context_line":"      image create \"$image_name\" \\"},{"line_number":551,"context_line":"      --disk-format qcow2 --container-format bare \\"},{"line_number":552,"context_line":"      --tag trove \\"}],"source_content_type":"text/x-sh","patch_set":20,"id":"33847dbf_eb5e771a","line":549,"in_reply_to":"899cb07c_f9d4f8b1","updated":"2026-08-19 15:19:00.000000000","message":"Nice catch, thanks! Resolved.","commit_id":"9513a2afe8ffd9016e43e104466bb2dd587ee268"}]}