)]}'
{"/PATCHSET_LEVEL":[{"author":{"_account_id":36080,"name":"Erkin Mussurmankulov","display_name":"Eric","email":"erkin.mussurmankulov@ps.kz","username":"mongoose404","status":"PS Cloud services employee"},"change_message_id":"c7b72be7c8c445e2403fbcf44f1c8166459440c1","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"7f5ca348_b06d2705","updated":"2026-08-07 06:15:47.000000000","message":"Hi! I already tried this a few months ago [here](https://review.opendev.org/c/openstack/trove/+/990725)\nBaking tarball images doesn\u0027t show any significant boost compared with a normal image pull.\n\nBut these patches will increase zuul jobs significantly (10-15 min for some jobs):\n\n - [Add MySQL image optimized for zuul test jobs](https://review.opendev.org/c/openstack/trove/+/993085)\n - [Add PostgreSQL image optimized for zuul test jobs](https://review.opendev.org/c/openstack/trove/+/993089)\n - [Add backup images optimized for zuul test jobs](https://review.opendev.org/c/openstack/trove/+/994210)\n - [Build guest images asynchronously in DevStack](https://review.opendev.org/c/openstack/trove/+/993305)\n - [Fix guest log collection startup bottleneck](https://review.opendev.org/c/openstack/trove/+/994159)\nFor the last item, the more advanced alternative is [here](https://review.opendev.org/c/openstack/trove/+/998524), but it doesn\u0027t work yet, it needs some debugging)","commit_id":"1bb8fa41de2cc88bbc6ddd6fec39b1ab5d5d2590"},{"author":{"_account_id":36080,"name":"Erkin Mussurmankulov","display_name":"Eric","email":"erkin.mussurmankulov@ps.kz","username":"mongoose404","status":"PS Cloud services employee"},"change_message_id":"d027e8e8e84d344a461f2a25ea7e4e848e81a59f","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"eed8fc2a_10051f34","in_reply_to":"29263854_16552d45","updated":"2026-08-07 09:27:46.000000000","message":"20%? It\u0027s a good result, but my measurements showed a difference of merely 5-10 seconds 🙂\nE.g., for docker pull it was 5 min and for docker load tarball it was 4m 50sec. Not a big deal 😆\n\nI also doubt about supporting such `optimized` images, but reducing the image size really shows good results. Right now it\u0027s just a concept. Probably someone will figure out a better way to reduce image size; probably some new tools will appear soon, dunno.\n\nBut for backup Docker images, I have a solution in mind: we can get rid of backup images altogether. We have all we need in the main database image (backup utility of the exact version for this database). We should just run a container with the required command for creating a backup and pass the output to the backup driver (Swift or S3 in the future). But it\u0027s a lot of work, pending for the next release 😃\n\nI\u0027m not sure why backup containers were created in the first place. Maintaining them is a tedious process, and they are a limiting factor for supporting legacy database versions (e.g., MySQL 5.7, MariaDB 10, PostgreSQL 12, etc...). You can\u0027t build backup images for old DB versions anymore; no required packages are left in the repositories.","commit_id":"1bb8fa41de2cc88bbc6ddd6fec39b1ab5d5d2590"},{"author":{"_account_id":36080,"name":"Erkin Mussurmankulov","display_name":"Eric","email":"erkin.mussurmankulov@ps.kz","username":"mongoose404","status":"PS Cloud services employee"},"change_message_id":"e657a31a5b09a0680c5b9047d4be8363ecfd528f","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"f5fc6317_4e9ccc24","in_reply_to":"7f5ca348_b06d2705","updated":"2026-08-07 06:24:17.000000000","message":"I also tried to use ramdisk for the `/var/lib/docker` directory, without success. The most expensive operation is unpacking the image; it produces dozens of thousands of files, so the tarball is also a packed image, and the most expensive operation here is unpacking. So the only viable vector for optimizing with Docker is to reduce image size and the number of layers.","commit_id":"1bb8fa41de2cc88bbc6ddd6fec39b1ab5d5d2590"},{"author":{"_account_id":5733,"name":"Kieran Spear","email":"kispear@gmail.com","username":"kspear"},"change_message_id":"c8b6f9ba343d8c69a5f86fab1711d4ac1a4eb74c","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"29263854_16552d45","in_reply_to":"f5fc6317_4e9ccc24","updated":"2026-08-07 08:58:53.000000000","message":"Yeah, it looks like this is only around a 20% improvement, which doesn\u0027t seem worth it.\n\nI skimmed some of those patches, I will take a closer look next week. My concern with the custom mysql/postgres images is it\u0027s a pretty high ongoing maintenance cost to have custom images just for the gate.\n\nIt looks like magnum actually does a chrooted docker pull while building an image, to get rid of the docker save/load entirely. I\u0027m going to try that as well.","commit_id":"1bb8fa41de2cc88bbc6ddd6fec39b1ab5d5d2590"},{"author":{"_account_id":5733,"name":"Kieran Spear","email":"kispear@gmail.com","username":"kspear"},"change_message_id":"144dedcb23e65c7834cfafdaf0ed7150000c1777","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":4,"id":"99a893e0_d27005ed","updated":"2026-08-10 05:31:15.000000000","message":"recheck","commit_id":"c90a442a783676f7db73563ddf974b51e11fe30d"},{"author":{"_account_id":5733,"name":"Kieran Spear","email":"kispear@gmail.com","username":"kspear"},"change_message_id":"3a6a0e3a6dccfb8db5d83d763f3cecc0e46360ba","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":5,"id":"92ecb035_ca6ed8f7","updated":"2026-08-10 07:38:09.000000000","message":"@erkin.mussurmankulov@ps.kz could you check the latest patchsets (4/5)? They show a significant improvement by pulling the images during the image build, so the guest pull is a no-op essentially. This should also help a lot with build time variance.\n\nI think this will also work alongside your async image build patch to reduce build times even further.","commit_id":"5e56c58176a9b707e6b9843ff4cd2bf9c7a80a34"},{"author":{"_account_id":5733,"name":"Kieran Spear","email":"kispear@gmail.com","username":"kspear"},"change_message_id":"f98a7a7479d1bfdaf1bdcc791241e52d63a821f2","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":5,"id":"2fded79d_95fc94ec","in_reply_to":"1e21e8fb_338ce758","updated":"2026-08-10 10:54:29.000000000","message":"The latest patchset is doing more than just putting a tar on the guest image though. It\u0027s doing a full layer extraction on the image build host via docker and baking _that_ into the image, so when each guest is booted it doesn\u0027t need to pull (or extract layers) at all, everything is already in containerd storage.\n\nFor example if you look at this job, it has gone from around 1h30m to 1h (the cluster of three runs to the right around the 1h duration are all from this patch):\n\nhttps://zuul.opendev.org/t/openstack/runtime?job_name\u003dtrove-tempest-cinder-storage-driver-postgresql\u0026project\u003dopenstack%2Ftrove\u0026branch\u003dmaster\u0026pipeline\u003dcheck","commit_id":"5e56c58176a9b707e6b9843ff4cd2bf9c7a80a34"},{"author":{"_account_id":36080,"name":"Erkin Mussurmankulov","display_name":"Eric","email":"erkin.mussurmankulov@ps.kz","username":"mongoose404","status":"PS Cloud services employee"},"change_message_id":"f5a5a4953675931a1e3f7cb229730d28308116d0","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":5,"id":"2f877c17_e8c3dc5c","in_reply_to":"2c61f4d5_80518177","updated":"2026-08-10 13:01:30.000000000","message":"I took a closer look, and the `integration/scripts/files/elements/guest-agent/install.d/guest-agent-source-install/32-embedded-docker-images` is needed; we can\u0027t remove it.\nBut we can safely remove `/opt/trove-docker-images.tar` after successful or failed loading of the images; we don\u0027t need it. So the total guest image size should be smaller without this file.","commit_id":"5e56c58176a9b707e6b9843ff4cd2bf9c7a80a34"},{"author":{"_account_id":5733,"name":"Kieran Spear","email":"kispear@gmail.com","username":"kspear"},"change_message_id":"4bd0f69dc5300978c4b4c40ef64f090bd3cd840b","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":5,"id":"9e8892b4_27a636f8","in_reply_to":"2f877c17_e8c3dc5c","updated":"2026-08-10 23:31:20.000000000","message":"True, I\u0027ll cut this down to just the necessary parts.","commit_id":"5e56c58176a9b707e6b9843ff4cd2bf9c7a80a34"},{"author":{"_account_id":36080,"name":"Erkin Mussurmankulov","display_name":"Eric","email":"erkin.mussurmankulov@ps.kz","username":"mongoose404","status":"PS Cloud services employee"},"change_message_id":"7e8d7e07732b6c1efae0c6d40a7e88b4742d9014","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":5,"id":"2c61f4d5_80518177","in_reply_to":"2fded79d_95fc94ec","updated":"2026-08-10 11:57:40.000000000","message":"Oh, I see now. WOW.\n\nWait, but in that case, you don\u0027t need these files: \n - integration/scripts/files/elements/guest-agent/install.d/guest-agent-source-install/50-trove-image-loader\n - integration/scripts/files/elements/guest-agent/install.d/guest-agent-source-install/32-embedded-docker-images\n\nCan you remove them, please?\nAnd we\u0027ll see if it actually works.","commit_id":"5e56c58176a9b707e6b9843ff4cd2bf9c7a80a34"},{"author":{"_account_id":36080,"name":"Erkin Mussurmankulov","display_name":"Eric","email":"erkin.mussurmankulov@ps.kz","username":"mongoose404","status":"PS Cloud services employee"},"change_message_id":"fd87cf2d9794170d8b25a338e461c910a3b2ad14","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":5,"id":"1e21e8fb_338ce758","in_reply_to":"92ecb035_ca6ed8f7","updated":"2026-08-10 09:17:07.000000000","message":"I was testing the same approach, and it didn\u0027t show a significant speed increase.\n\nHere are the results on my local host:\n\n```\n$ time docker pull localhost:4000/trove-datastores/postgresql:18\n18: Pulling from trove-datastores/postgresql\nf96fde5c7579: Pull complete \n...\nDigest: sha256:c2cd7b05457b161725a471aaeef427e46c509cb37c0936d81ae71e1b22a4558e\nStatus: Downloaded newer image for localhost:4000/trove-datastores/postgresql:18\nlocalhost:4000/trove-datastores/postgresql:18\n\nreal\t0m3.749s\nuser\t0m0.012s\nsys\t0m0.012s\n\n$ docker save localhost:4000/trove-datastores/postgresql:18 \u003e image.tar\n$ docker image rm localhost:4000/trove-datastores/postgresql:18\nUntagged: localhost:4000/trove-datastores/postgresql:18\nUntagged: localhost:4000/trove-datastores/postgresql@sha256:c2cd7b05457b161725a471aaeef427e46c509cb37c0936d81ae71e1b22a4558e\n...\n\n$ time docker load -i image.tar\nf2ec4de84f55: Loading layer [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003e]  81.05MB/81.05MB\ne0b1b7835da3: Loading layer [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003e]  12.29kB/12.29kB\n5d89e95cad74: Loading layer [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003e]  16.14MB/16.14MB\n00b74b5f4bfd: Loading layer [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003e]  3.652MB/3.652MB\n4a6a14f26c3d: Loading layer [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003e]  26.18MB/26.18MB\na527b9f6806c: Loading layer [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003e]  3.483MB/3.483MB\n2e62347dc0bb: Loading layer [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003e]  1.536kB/1.536kB\n6c981ebf1296: Loading layer [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003e]   7.68kB/7.68kB\n4c5575c3daa1: Loading layer [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003e]  332.9MB/332.9MB\na8ef8b8b985b: Loading layer [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003e]  74.75kB/74.75kB\n063c1642e1a5: Loading layer [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003e]  2.048kB/2.048kB\n87a1644bff90: Loading layer [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003e]  20.99kB/20.99kB\n422ae5421b5f: Loading layer [\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003e]  3.072kB/3.072kB\nLoaded image: localhost:4000/trove-datastores/postgresql:18\n\nreal\t0m4.226s\nuser\t0m0.046s\nsys\t0m0.193s\n```\n\nAs you can see, no speed increase for tarballs; they may run slower in some cases.\n\nThis approach may work if the bottleneck is image transferring, but in our case the bottleneck is the high load on the host system itself.\nExtracting archives is a very CPU-dependent task, no matter if it\u0027s a tarball or a remote image from the registry on the host system.\n\nAlso, we\u0027re probably observing an \"overselling\": when a lot of concurrent zuul jobs are running, the overall performance is suffering, and zuul jobs may run\neven slower, more than 1 hour for basic tests, but during non-busy hours,\nit may be a strong 40 minutes run.\n\nAlso, the amount of introduced code that we should support in this MR is significant, so I still think that this approach is a *dead end*.\n\n___________________\n\nBy the way, cooling down guest-log-collector helps to reduce CPU usage a lot, increasing overall performance.\nI have two options, but the second isn\u0027t working properly yet:\n 1. https://review.opendev.org/c/openstack/trove/+/994159?usp\u003dsearch\n 2. https://review.opendev.org/c/openstack/trove/+/998524?usp\u003dsearch","commit_id":"5e56c58176a9b707e6b9843ff4cd2bf9c7a80a34"},{"author":{"_account_id":5733,"name":"Kieran Spear","email":"kispear@gmail.com","username":"kspear"},"change_message_id":"ce20b521d78f784672828105df0c382964aadc72","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":5,"id":"012012e6_67980ca4","in_reply_to":"9e8892b4_27a636f8","updated":"2026-08-11 00:17:58.000000000","message":"Ok, I\u0027ve removed any trove-image-loader changes from this patchset, so we don\u0027t need to embed the tar file on the image. I think the trove-image-loader pull will just end up being a no-op, so we can remove it completely in a later change.","commit_id":"5e56c58176a9b707e6b9843ff4cd2bf9c7a80a34"},{"author":{"_account_id":36080,"name":"Erkin Mussurmankulov","display_name":"Eric","email":"erkin.mussurmankulov@ps.kz","username":"mongoose404","status":"PS Cloud services employee"},"change_message_id":"2b038bd66f5efa4b4db34073593328bbb2b6b593","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":6,"id":"5a538dc3_597f29da","updated":"2026-08-11 08:44:02.000000000","message":"That\u0027s awesome! It works like a charm! 😃\n```\n00:51:11 systemd[1]: Starting trove-image-loader.service - OpenStack Trove image loader Service...\n00:51:19 trove-image-loader[1022]: datastore_image\u003dnpa10a813ecb194:4000/trove-datastores/mysql\n00:51:19 trove-image-loader[1022]: backup_image\u003dnpa10a813ecb194:4000/trove-datastores/db-backup-mysql\n00:51:22 trove-image-loader[1022]: Image already exists, skip loading.\n00:51:23 systemd[1]: trove-image-loader.service: Deactivated successfully.\n00:51:23 systemd[1]: Finished trove-image-loader.service - OpenStack Trove image loader Service.\n00:51:23 systemd[1]: trove-image-loader.service: Consumed 10.107s CPU time.\n```\nTrove image loader [shows](https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_90c/openstack/90cbbf18d2354632980f961b1fa99216/controller/logs/guest-agent-logs/82e844f4-7502-42cc-9458-6c30e5b9af0f/log/syslog) completion in only 12 seconds 🥳\n\nI\u0027m eager to merge it, but I found a couple of discrepancies in the code and docs which are worth mentioning. Also, I added serial numbers for convenience.\nThe rest LGTM.\n\n\nI love that this MR allows us to use current DB images without optimizing, and the big size of the images wouldn\u0027t affect the overall test performance at all.\nAfter coupling with the async DIB build MR, we probably will run under 30mins for basic tests 😄","commit_id":"ab42fb75498067aae4d739478ac005b9c433720b"},{"author":{"_account_id":5733,"name":"Kieran Spear","email":"kispear@gmail.com","username":"kspear"},"change_message_id":"e4684a08cd9e5e217141253d522951c7ab0d1e94","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":6,"id":"cc372522_7c0a6071","in_reply_to":"5a538dc3_597f29da","updated":"2026-08-11 10:26:20.000000000","message":"Yes, looking forward to seeing the results with your changes too!","commit_id":"ab42fb75498067aae4d739478ac005b9c433720b"},{"author":{"_account_id":36080,"name":"Erkin Mussurmankulov","display_name":"Eric","email":"erkin.mussurmankulov@ps.kz","username":"mongoose404","status":"PS Cloud services employee"},"change_message_id":"6933a013ba3e13394bfa96dc2638b9f10b7fbe09","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":7,"id":"d3700a03_b70a7a20","updated":"2026-08-11 11:47:32.000000000","message":"Great job! Thank you.\nMR is ready for departure.","commit_id":"224524d82173b59cdaca8834d7ae53c8f287a7e4"},{"author":{"_account_id":5733,"name":"Kieran Spear","email":"kispear@gmail.com","username":"kspear"},"change_message_id":"e1bf38f0eda079e73699c89528f52e209054c6c6","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":7,"id":"b0045979_cb70f08e","updated":"2026-08-11 10:24:58.000000000","message":"Ok, I think I\u0027ve addressed all your comments, Eric. I\u0027ve rebased on top of your just-merged mirror patch, so hopefully the build is faster this time.","commit_id":"224524d82173b59cdaca8834d7ae53c8f287a7e4"},{"author":{"_account_id":5733,"name":"Kieran Spear","email":"kispear@gmail.com","username":"kspear"},"change_message_id":"5081273dd5dac52cc746ee3795337ffba391397d","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":7,"id":"e229809e_098185d5","in_reply_to":"8d506659_b5e51c42","updated":"2026-08-12 06:22:42.000000000","message":"Ah, yes good point. Happy to look into this tomorrow unless you get to it first.","commit_id":"224524d82173b59cdaca8834d7ae53c8f287a7e4"},{"author":{"_account_id":36080,"name":"Erkin Mussurmankulov","display_name":"Eric","email":"erkin.mussurmankulov@ps.kz","username":"mongoose404","status":"PS Cloud services employee"},"change_message_id":"1914ecc079dda5574030ad0fec45c98072755d55","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":7,"id":"8d506659_b5e51c42","in_reply_to":"d3700a03_b70a7a20","updated":"2026-08-11 12:24:46.000000000","message":"Interesting side effect:\nEven if tags are different, the hash sums of the images are similar. So a preloaded image with tags exported as `quay.io/...` works for local registry tags`hostname:4000/...` too.\n\nBut it looks like we forgot about the necessity of building backup images. Currently, all Zuul jobs that run tests with backup changes use the `TROVE_ENABLE_LOCAL_REGISTRY\u003dTrue` and `TROVE_BUILD_BACKUP_IMAGES\u003dTrue` combination; this invokes the building of the backup images using current changes from git instead of using pre-built backup images from quay.io.\n\nI propose creating another patch to fix this issue. The first idea is:\n - Move invocation of `create_registry_container` before `create_guest_image`\n - Add code into `create_registry_container` that will set the correct values for `DIB_TROVE_DOCKER_IMAGES`, considering the newly built backup image\n - If `create_registry_container` wasn\u0027t called, we may call `prepare_embedded_docker_images_vars` instead, if all prerequisites are met\n\nI already did something like that when I was experimenting with tarballs.","commit_id":"224524d82173b59cdaca8834d7ae53c8f287a7e4"}],"devstack/plugin.sh":[{"author":{"_account_id":36080,"name":"Erkin Mussurmankulov","display_name":"Eric","email":"erkin.mussurmankulov@ps.kz","username":"mongoose404","status":"PS Cloud services employee"},"change_message_id":"2b038bd66f5efa4b4db34073593328bbb2b6b593","unresolved":true,"context_lines":[{"line_number":462,"context_line":"# the guest image (via DIB_TROVE_DOCKER_IMAGES). This saves guest instances"},{"line_number":463,"context_line":"# from pulling ~1GB of images at create time, which in the gate happens on an"},{"line_number":464,"context_line":"# emulated CPU and routinely takes 15-30 minutes."},{"line_number":465,"context_line":"function prepare_embedded_docker_images {"},{"line_number":466,"context_line":"    local quay_alias\u003d${TROVE_DATASTORE_TYPE}"},{"line_number":467,"context_line":"    [[ \"${quay_alias}\" \u003d\u003d \"postgresql\" ]] \u0026\u0026 quay_alias\u003d\"postgres\""},{"line_number":468,"context_line":""}],"source_content_type":"text/x-sh","patch_set":6,"id":"6a578adf_44f4f411","line":465,"updated":"2026-08-11 08:44:02.000000000","message":"5.\nAlso, maybe after all the code moves, we can rename it to `prepare_embedded_docker_images_vars`? Or something like that- how do you think?","commit_id":"ab42fb75498067aae4d739478ac005b9c433720b"},{"author":{"_account_id":5733,"name":"Kieran Spear","email":"kispear@gmail.com","username":"kspear"},"change_message_id":"e1bf38f0eda079e73699c89528f52e209054c6c6","unresolved":false,"context_lines":[{"line_number":462,"context_line":"# the guest image (via DIB_TROVE_DOCKER_IMAGES). This saves guest instances"},{"line_number":463,"context_line":"# from pulling ~1GB of images at create time, which in the gate happens on an"},{"line_number":464,"context_line":"# emulated CPU and routinely takes 15-30 minutes."},{"line_number":465,"context_line":"function prepare_embedded_docker_images {"},{"line_number":466,"context_line":"    local quay_alias\u003d${TROVE_DATASTORE_TYPE}"},{"line_number":467,"context_line":"    [[ \"${quay_alias}\" \u003d\u003d \"postgresql\" ]] \u0026\u0026 quay_alias\u003d\"postgres\""},{"line_number":468,"context_line":""}],"source_content_type":"text/x-sh","patch_set":6,"id":"64b9919e_c5b7176f","line":465,"in_reply_to":"6a578adf_44f4f411","updated":"2026-08-11 10:24:58.000000000","message":"Done","commit_id":"ab42fb75498067aae4d739478ac005b9c433720b"},{"author":{"_account_id":36080,"name":"Erkin Mussurmankulov","display_name":"Eric","email":"erkin.mussurmankulov@ps.kz","username":"mongoose404","status":"PS Cloud services employee"},"change_message_id":"2b038bd66f5efa4b4db34073593328bbb2b6b593","unresolved":true,"context_lines":[{"line_number":481,"context_line":""},{"line_number":482,"context_line":"    local pair source target refs attempt"},{"line_number":483,"context_line":"    refs\u003d\"\""},{"line_number":484,"context_line":"    for pair in ${pairs}; do"},{"line_number":485,"context_line":"        source\u003d${pair%%\u003d*}"},{"line_number":486,"context_line":"        target\u003d${pair##*\u003d}"},{"line_number":487,"context_line":"        # Fail fast (rather than silently falling back to pulling at"}],"source_content_type":"text/x-sh","patch_set":6,"id":"08884ec5_0546f781","line":484,"updated":"2026-08-11 08:44:02.000000000","message":"4.\nThis part duplicates almost the same code in `70-trove-docker-images` script.\nI propose moving this part, including `docker tag` to `70-trove-docker-images` entirely.\nSo the responsibility of this method remains to set the correct `DIB_TROVE_DOCKER_IMAGES` variable only.","commit_id":"ab42fb75498067aae4d739478ac005b9c433720b"},{"author":{"_account_id":5733,"name":"Kieran Spear","email":"kispear@gmail.com","username":"kspear"},"change_message_id":"e1bf38f0eda079e73699c89528f52e209054c6c6","unresolved":false,"context_lines":[{"line_number":481,"context_line":""},{"line_number":482,"context_line":"    local pair source target refs attempt"},{"line_number":483,"context_line":"    refs\u003d\"\""},{"line_number":484,"context_line":"    for pair in ${pairs}; do"},{"line_number":485,"context_line":"        source\u003d${pair%%\u003d*}"},{"line_number":486,"context_line":"        target\u003d${pair##*\u003d}"},{"line_number":487,"context_line":"        # Fail fast (rather than silently falling back to pulling at"}],"source_content_type":"text/x-sh","patch_set":6,"id":"f1123192_d033766b","line":484,"in_reply_to":"08884ec5_0546f781","updated":"2026-08-11 10:24:58.000000000","message":"Done","commit_id":"ab42fb75498067aae4d739478ac005b9c433720b"},{"author":{"_account_id":36080,"name":"Erkin Mussurmankulov","display_name":"Eric","email":"erkin.mussurmankulov@ps.kz","username":"mongoose404","status":"PS Cloud services employee"},"change_message_id":"2b038bd66f5efa4b4db34073593328bbb2b6b593","unresolved":true,"context_lines":[{"line_number":487,"context_line":"        # Fail fast (rather than silently falling back to pulling at"},{"line_number":488,"context_line":"        # instance boot) so a broken embedding path can\u0027t masquerade as a"},{"line_number":489,"context_line":"        # working one. Retry a couple of times to ride out registry blips."},{"line_number":490,"context_line":"        for attempt in 1 2 3; do"},{"line_number":491,"context_line":"            sudo docker pull \"${source}\" \u0026\u0026 break"},{"line_number":492,"context_line":"            [[ ${attempt} \u003d\u003d 3 ]] \u0026\u0026 die $LINENO \"Unable to pull ${source} for embedding into the guest image\""},{"line_number":493,"context_line":"            sleep 10"}],"source_content_type":"text/x-sh","patch_set":6,"id":"134d72de_d82b51b6","line":490,"updated":"2026-08-11 08:44:02.000000000","message":"3.\nProbably doing several attempts isn\u0027t required here. If the OpenDev or quay.io infrastructure is in stampede mode, this wouldn\u0027t help at all. The test or some parallel job will mostly fail somewhere else anyway.\nAlso, as far as I remember, Docker will do several attempts if some of the layers failed to download by itself, so small network hiccups aren\u0027t a big deal.\n\nI think we can do a failproof `docker pull` by using a helper method for the entire CI later in a separate MR, if necessary.","commit_id":"ab42fb75498067aae4d739478ac005b9c433720b"},{"author":{"_account_id":5733,"name":"Kieran Spear","email":"kispear@gmail.com","username":"kspear"},"change_message_id":"e1bf38f0eda079e73699c89528f52e209054c6c6","unresolved":false,"context_lines":[{"line_number":487,"context_line":"        # Fail fast (rather than silently falling back to pulling at"},{"line_number":488,"context_line":"        # instance boot) so a broken embedding path can\u0027t masquerade as a"},{"line_number":489,"context_line":"        # working one. Retry a couple of times to ride out registry blips."},{"line_number":490,"context_line":"        for attempt in 1 2 3; do"},{"line_number":491,"context_line":"            sudo docker pull \"${source}\" \u0026\u0026 break"},{"line_number":492,"context_line":"            [[ ${attempt} \u003d\u003d 3 ]] \u0026\u0026 die $LINENO \"Unable to pull ${source} for embedding into the guest image\""},{"line_number":493,"context_line":"            sleep 10"}],"source_content_type":"text/x-sh","patch_set":6,"id":"b3e09e9e_18762aba","line":490,"in_reply_to":"134d72de_d82b51b6","updated":"2026-08-11 10:24:58.000000000","message":"Agreed. Fixed.","commit_id":"ab42fb75498067aae4d739478ac005b9c433720b"},{"author":{"_account_id":36080,"name":"Erkin Mussurmankulov","display_name":"Eric","email":"erkin.mussurmankulov@ps.kz","username":"mongoose404","status":"PS Cloud services employee"},"change_message_id":"2b038bd66f5efa4b4db34073593328bbb2b6b593","unresolved":true,"context_lines":[{"line_number":509,"context_line":"        return 0"},{"line_number":510,"context_line":"    fi"},{"line_number":511,"context_line":""},{"line_number":512,"context_line":"    if [[ \"${TROVE_EMBED_DATASTORE_IMAGES}\" \u003d\u003d \"True\" \u0026\u0026 \\"},{"line_number":513,"context_line":"          -n \"${TROVE_DATASTORE_TYPE:-}\" \u0026\u0026 -n \"${TROVE_DATASTORE_VERSION:-}\" ]]; then"},{"line_number":514,"context_line":"        prepare_embedded_docker_images"},{"line_number":515,"context_line":"    fi"}],"source_content_type":"text/x-sh","patch_set":6,"id":"34de21a5_808fdd62","line":512,"updated":"2026-08-11 08:44:02.000000000","message":"6.\nI think we can move this part outside of this method.\n\nEven if guest image building isn\u0027t enabled (I can\u0027t imagine why you need it that way, though), the `prepare_embedded_docker_images` method will just do a mere variable set, without downloading anything, if we remove the `docker pull` part away from it.\n\n_________\n\nLater, in my MR, I will move this part into a separate method for an async call by DevStack.","commit_id":"ab42fb75498067aae4d739478ac005b9c433720b"},{"author":{"_account_id":5733,"name":"Kieran Spear","email":"kispear@gmail.com","username":"kspear"},"change_message_id":"e1bf38f0eda079e73699c89528f52e209054c6c6","unresolved":false,"context_lines":[{"line_number":509,"context_line":"        return 0"},{"line_number":510,"context_line":"    fi"},{"line_number":511,"context_line":""},{"line_number":512,"context_line":"    if [[ \"${TROVE_EMBED_DATASTORE_IMAGES}\" \u003d\u003d \"True\" \u0026\u0026 \\"},{"line_number":513,"context_line":"          -n \"${TROVE_DATASTORE_TYPE:-}\" \u0026\u0026 -n \"${TROVE_DATASTORE_VERSION:-}\" ]]; then"},{"line_number":514,"context_line":"        prepare_embedded_docker_images"},{"line_number":515,"context_line":"    fi"}],"source_content_type":"text/x-sh","patch_set":6,"id":"44b68ccb_8f61f38f","line":512,"in_reply_to":"34de21a5_808fdd62","updated":"2026-08-11 10:24:58.000000000","message":"Done, if I understood you correctly.","commit_id":"ab42fb75498067aae4d739478ac005b9c433720b"}],"integration/scripts/files/elements/guest-agent/README.rst":[{"author":{"_account_id":36080,"name":"Erkin Mussurmankulov","display_name":"Eric","email":"erkin.mussurmankulov@ps.kz","username":"mongoose404","status":"PS Cloud services employee"},"change_message_id":"2b038bd66f5efa4b4db34073593328bbb2b6b593","unresolved":true,"context_lines":[{"line_number":6,"context_line":"Environment variables:"},{"line_number":7,"context_line":""},{"line_number":8,"context_line":"``DIB_TROVE_DOCKER_IMAGES``"},{"line_number":9,"context_line":"  Optional, space separated list of docker image references (``name:tag``)."},{"line_number":10,"context_line":"  The images are fetched with the build host\u0027s docker daemon, saved into a"},{"line_number":11,"context_line":"  tarball embedded in the guest image, and loaded by trove-image-loader on"},{"line_number":12,"context_line":"  first boot so guest instances don\u0027t need to pull them at create time. The"}],"source_content_type":"text/x-rst","patch_set":6,"id":"8e6b8176_b4980b6a","line":9,"updated":"2026-08-11 08:44:02.000000000","message":"1.\nI think we should move this documentation away from this file. The chances that somebody will look here are very faint.\n\nI propose moving this into `doc/source/install/install-devstack.rst`, probably a new section after the \"Running DevStack\" section. Something like \"Other DevStack/DIB variables\".\nThis info would get into search engines and AI bots knowledge bases eventually.\n\nProbably we should move a lot of comments outside of `devstack/settings` to this section later. Or even create a separate file in docs for this. But not within the scope of this MR, though.\n\nAlso, please add a disclaimer that this special mode is intended to be used only in the Zuul environment due to limited resources on testing infrastructure. It is highly **not recommended** to use this in production because embedding Docker images inside virtual images is not how Trove should work; cloud operators may get unexpected results. Also, a normal `docker image pull` in a production environment should be very fast (mere seconds), so this mode isn\u0027t helpful for production at all.\n\nIn addition, it\u0027s worth mentioning that for the DevStack setup, this option will work only if `$TROVE_DATASTORE_TYPE` and `$TROVE_DATASTORE_VERSION` are set (idk whether *\"single datastore version builds\"* is an appropriate term? 🤔)","commit_id":"ab42fb75498067aae4d739478ac005b9c433720b"},{"author":{"_account_id":5733,"name":"Kieran Spear","email":"kispear@gmail.com","username":"kspear"},"change_message_id":"e1bf38f0eda079e73699c89528f52e209054c6c6","unresolved":false,"context_lines":[{"line_number":6,"context_line":"Environment variables:"},{"line_number":7,"context_line":""},{"line_number":8,"context_line":"``DIB_TROVE_DOCKER_IMAGES``"},{"line_number":9,"context_line":"  Optional, space separated list of docker image references (``name:tag``)."},{"line_number":10,"context_line":"  The images are fetched with the build host\u0027s docker daemon, saved into a"},{"line_number":11,"context_line":"  tarball embedded in the guest image, and loaded by trove-image-loader on"},{"line_number":12,"context_line":"  first boot so guest instances don\u0027t need to pull them at create time. The"}],"source_content_type":"text/x-rst","patch_set":6,"id":"6f809ac8_20960d14","line":9,"in_reply_to":"8e6b8176_b4980b6a","updated":"2026-08-11 10:24:58.000000000","message":"Ok, I\u0027ve moved the bulk into install-devstack.rst where it will be easier to discover. I\u0027ve still left a basic description here since it\u0027s DIB convention.","commit_id":"ab42fb75498067aae4d739478ac005b9c433720b"},{"author":{"_account_id":36080,"name":"Erkin Mussurmankulov","display_name":"Eric","email":"erkin.mussurmankulov@ps.kz","username":"mongoose404","status":"PS Cloud services employee"},"change_message_id":"2b038bd66f5efa4b4db34073593328bbb2b6b593","unresolved":true,"context_lines":[{"line_number":8,"context_line":"``DIB_TROVE_DOCKER_IMAGES``"},{"line_number":9,"context_line":"  Optional, space separated list of docker image references (``name:tag``)."},{"line_number":10,"context_line":"  The images are fetched with the build host\u0027s docker daemon, saved into a"},{"line_number":11,"context_line":"  tarball embedded in the guest image, and loaded by trove-image-loader on"},{"line_number":12,"context_line":"  first boot so guest instances don\u0027t need to pull them at create time. The"},{"line_number":13,"context_line":"  references must match the ``docker_image``/``backup_docker_image`` values"},{"line_number":14,"context_line":"  the guest agent is configured with. At build time the tarball is extracted"}],"source_content_type":"text/x-rst","patch_set":6,"id":"8507b427_7d94b617","line":11,"updated":"2026-08-11 08:44:02.000000000","message":"2.\nI think, no need to mention `trove-image-loader` here at all, as I plan to remove it soon. Also, images are already loaded into the local containerd, so `trove-image-loader` does nothing right now, except that it shows that the required images are in place.","commit_id":"ab42fb75498067aae4d739478ac005b9c433720b"},{"author":{"_account_id":5733,"name":"Kieran Spear","email":"kispear@gmail.com","username":"kspear"},"change_message_id":"e1bf38f0eda079e73699c89528f52e209054c6c6","unresolved":false,"context_lines":[{"line_number":8,"context_line":"``DIB_TROVE_DOCKER_IMAGES``"},{"line_number":9,"context_line":"  Optional, space separated list of docker image references (``name:tag``)."},{"line_number":10,"context_line":"  The images are fetched with the build host\u0027s docker daemon, saved into a"},{"line_number":11,"context_line":"  tarball embedded in the guest image, and loaded by trove-image-loader on"},{"line_number":12,"context_line":"  first boot so guest instances don\u0027t need to pull them at create time. The"},{"line_number":13,"context_line":"  references must match the ``docker_image``/``backup_docker_image`` values"},{"line_number":14,"context_line":"  the guest agent is configured with. At build time the tarball is extracted"}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba10903b_d6c6d0cc","line":11,"in_reply_to":"8507b427_7d94b617","updated":"2026-08-11 10:24:58.000000000","message":"Done","commit_id":"ab42fb75498067aae4d739478ac005b9c433720b"}],"integration/scripts/files/elements/guest-agent/post-install.d/35-preload-docker-images":[{"author":{"_account_id":36080,"name":"Erkin Mussurmankulov","display_name":"Eric","email":"erkin.mussurmankulov@ps.kz","username":"mongoose404","status":"PS Cloud services employee"},"change_message_id":"4b6a5b1d22e72cfac2c241d0726732125ebbf4d5","unresolved":true,"context_lines":[{"line_number":21,"context_line":"# chroot root is not a mountpoint, so containerd\u0027s mountinfo lookups fail"},{"line_number":22,"context_line":"# without them)."},{"line_number":23,"context_line":"#"},{"line_number":24,"context_line":"# Best-effort: if the daemons cannot run in this environment or the load"},{"line_number":25,"context_line":"# fails, the tarball is left in place and trove-image-loader loads it on"},{"line_number":26,"context_line":"# first boot instead (which in turn falls back to pulling)."},{"line_number":27,"context_line":"#"}],"source_content_type":"application/x-shellscript","patch_set":5,"id":"adb6c7a3_074b877f","line":24,"updated":"2026-08-10 12:05:50.000000000","message":"After removing tarball we also may remove this comment.\n\nI can see the idea behind it, but we\u0027ll get four ways in total:\n- standard docker pull by guest-agent\n- trove-image-loader pull\n- docker load from tarball\n- pre-baked images by DIB\n\nI think everything in between the first and last item may be removed, it\u0027s too much 🙂\n\nWe will remove `trove-image-loader` later, in a separate MR. Pre-baking images directly into the guest image resolves the issue that `trove-image-loader` is trying to accomplish.","commit_id":"5e56c58176a9b707e6b9843ff4cd2bf9c7a80a34"},{"author":{"_account_id":5733,"name":"Kieran Spear","email":"kispear@gmail.com","username":"kspear"},"change_message_id":"4bd0f69dc5300978c4b4c40ef64f090bd3cd840b","unresolved":true,"context_lines":[{"line_number":21,"context_line":"# chroot root is not a mountpoint, so containerd\u0027s mountinfo lookups fail"},{"line_number":22,"context_line":"# without them)."},{"line_number":23,"context_line":"#"},{"line_number":24,"context_line":"# Best-effort: if the daemons cannot run in this environment or the load"},{"line_number":25,"context_line":"# fails, the tarball is left in place and trove-image-loader loads it on"},{"line_number":26,"context_line":"# first boot instead (which in turn falls back to pulling)."},{"line_number":27,"context_line":"#"}],"source_content_type":"application/x-shellscript","patch_set":5,"id":"ae0bbdd8_260afced","line":24,"in_reply_to":"adb6c7a3_074b877f","updated":"2026-08-10 23:31:20.000000000","message":"You\u0027re right! The tarball was only relevant for the first patchset so not needed at all now. I also didn\u0027t realise that trove-image-loader is purely a CI thing.","commit_id":"5e56c58176a9b707e6b9843ff4cd2bf9c7a80a34"},{"author":{"_account_id":5733,"name":"Kieran Spear","email":"kispear@gmail.com","username":"kspear"},"change_message_id":"ce20b521d78f784672828105df0c382964aadc72","unresolved":false,"context_lines":[{"line_number":21,"context_line":"# chroot root is not a mountpoint, so containerd\u0027s mountinfo lookups fail"},{"line_number":22,"context_line":"# without them)."},{"line_number":23,"context_line":"#"},{"line_number":24,"context_line":"# Best-effort: if the daemons cannot run in this environment or the load"},{"line_number":25,"context_line":"# fails, the tarball is left in place and trove-image-loader loads it on"},{"line_number":26,"context_line":"# first boot instead (which in turn falls back to pulling)."},{"line_number":27,"context_line":"#"}],"source_content_type":"application/x-shellscript","patch_set":5,"id":"53f6b47f_f4503e1a","line":24,"in_reply_to":"ae0bbdd8_260afced","updated":"2026-08-11 00:17:58.000000000","message":"Done","commit_id":"5e56c58176a9b707e6b9843ff4cd2bf9c7a80a34"}]}