)]}'
{"/PATCHSET_LEVEL":[{"author":{"_account_id":36080,"name":"Erkin Mussurmankulov","display_name":"Eric","email":"erkin.mussurmankulov@ps.kz","username":"mongoose404","status":"PS Cloud services employee"},"change_message_id":"fc28ecbd29d9fd00d03d438ec9f380f2e35ce591","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"eb5c7948_c4a88518","updated":"2026-05-28 07:28:20.000000000","message":"Hello Wu,\n\nCould you please take a look at this patch?\n\nDoes your local DevStack work without DNS in the `private` network?\n\nAlso, I\u0027ve inspected the Zuul jobs. Somehow, it gets two default routes from cloud-init:\n\n```\n| Route |   Destination   |    Gateway    |     Genmask     | Interface | Flags |\n+-------+-----------------+---------------+-----------------+-----------+-------+\n|   0   |     0.0.0.0     |    10.1.0.1   |     0.0.0.0     |    ens3   |   UG  |\n|   1   |     0.0.0.0     | 192.168.254.1 |     0.0.0.0     |    ens4   |   UG  |\n|   2   |     8.8.8.8     | 192.168.254.1 | 255.255.255.255 |    ens4   |  UGH  |\n|   3   |     10.1.0.0    |    0.0.0.0    | 255.255.255.192 |    ens3   |   U   |\n|   4   |     10.1.0.1    |    0.0.0.0    | 255.255.255.255 |    ens3   |   UH  |\n|   5   |     10.1.0.2    |    0.0.0.0    | 255.255.255.255 |    ens3   |   UH  |\n|   6   |    127.0.0.1    |    0.0.0.0    | 255.255.255.255 |    ens3   |   UH  |\n|   7   | 169.254.169.254 |    10.1.0.2   | 255.255.255.255 |    ens3   |  UGH  |\n|   8   | 169.254.169.254 | 192.168.254.2 | 255.255.255.255 |    ens4   |  UGH  |\n|   9   |  192.168.254.0  |    0.0.0.0    |  255.255.255.0  |    ens4   |   U   |\n|   10  |  192.168.254.1  |    0.0.0.0    | 255.255.255.255 |    ens4   |   UH  |\n|   11  |  192.168.254.2  |    0.0.0.0    | 255.255.255.255 |    ens4   |   UH  |\n+-------+-----------------+---------------+-----------------+-----------+-------+\n```\n\nHaving two default routes is not correct behavior. Using the `trove-mgmt` network for internet access isn\u0027t correct either. But somehow, it works 🤔\n\nWe\u0027ll probably need to revisit this issue later. Right now, we at least need to make local DevStack setups work properly.","commit_id":"9715c3e10e238ad1ea6dad4d995f1bcc92f7ce9a"},{"author":{"_account_id":26285,"name":"wu.chunyang","email":"wchy1001@gmail.com","username":"wu.chunyang"},"change_message_id":"3887945aaad607cc08f11c3a3fdc29f60a4705d0","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"430456c7_a2c86f33","in_reply_to":"2437baad_733bc1db","updated":"2026-06-11 01:18:15.000000000","message":"Hi, I am sorry for the confusion. On the contrary, the ens4 I referred to is the client interface, not the trove-mgmt interface.\nWe need to force cloud operators to use trove-mgmt as the default network instead of the client network, and remove all client network configurations from the default network namespace.\nEven in production environments, instances may need to pull images from quay.io, so the `TROVE_MGMT_GATEWAY` remains required.\nThe client interface only takes effect once it is attached to the database container. Before that happens, all network traffic must go through the trove-mgmt interface.\n\nIn a nutshell, the client interface should be used only for user network traffic, and all other traffic should go through the management interface.\n\nSo the simplest way to fix this issue is just to shut down the client interface before any network traffic happens, and we have already done this within the guest-agent service for non-dev guest images.","commit_id":"9715c3e10e238ad1ea6dad4d995f1bcc92f7ce9a"},{"author":{"_account_id":26285,"name":"wu.chunyang","email":"wchy1001@gmail.com","username":"wu.chunyang"},"change_message_id":"ddf44fab93bd12c0d2ad926b28735a30afa58c6d","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"c11eaa12_dceaa5c8","in_reply_to":"26fb1617_e61d96c0","updated":"2026-06-09 02:29:52.000000000","message":"When designing the Trove network, trove-mgmt handles all network access, including trove-guestagent deployment, Docker Hub access and OpenStack control plane traffic. Since we cannot predict how users configure their private networks, the user network should not undertake this role.","commit_id":"9715c3e10e238ad1ea6dad4d995f1bcc92f7ce9a"},{"author":{"_account_id":26285,"name":"wu.chunyang","email":"wchy1001@gmail.com","username":"wu.chunyang"},"change_message_id":"10261e7335abededcc705c98cefe9a78b54c2b69","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"d0e8cc6d_c417230e","in_reply_to":"6bd0a7e6_a36f69a0","updated":"2026-06-10 09:29:55.000000000","message":"Hi. Our goal is to use trove-mgmt as the default route. The ens4 interface should never be used for external communication, as it is dedicated to business traffic. So I don\u0027t think there is an issue with the current design.\nThe problem lies in the development environment. The guest-agent installs packages in the pre-exec stage prior to startup. It brings ens4 down right after launch to resolve duplicate routes.\nHowever, network traffic occurs before ens4 is shut down, so the duplicate route remains active.\n\nTo fix this, the simplest approach is to disable ens4 right after system startup. Its associated default route will then be removed as well.","commit_id":"9715c3e10e238ad1ea6dad4d995f1bcc92f7ce9a"},{"author":{"_account_id":36080,"name":"Erkin Mussurmankulov","display_name":"Eric","email":"erkin.mussurmankulov@ps.kz","username":"mongoose404","status":"PS Cloud services employee"},"change_message_id":"bc30ab7daea5f9d81e879fadd92497cd7237910b","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"6bd0a7e6_a36f69a0","in_reply_to":"c11eaa12_dceaa5c8","updated":"2026-06-10 07:17:22.000000000","message":"Hello, Wu!\n\n\nI can see a fundamental problem with our current network setup in tests.\n\nHere is the list of facts I\u0027m currently working with:\n\n1. Tempest scenarios use the `TROVE_MGMT_GATEWAY` variable, which attaches the `trove-mgmt` interface to the router `$ROUTER_ID` created by Neutron during startup. This causes cloud-init to use `trove-mgmt` as the default route inside the instance:\n\n  | Route |   Destination   |    Gateway    |     Genmask     | Interface | Flags |\n  +-------+-----------------+---------------+-----------------+-----------+-------+\n  |   0   |     0.0.0.0     |    10.1.0.1   |     0.0.0.0     |    ens3   |   UG  |\n  |   1   |     0.0.0.0     | 192.168.254.1 |     0.0.0.0     |    ens4   |   UG  | \u003c- here it is\n  |   2   |     8.8.8.8     | 192.168.254.1 | 255.255.255.255 |    ens4   |  UGH  |\n  |   3   |     10.1.0.0    |    0.0.0.0    | 255.255.255.192 |    ens3   |   U   |\n  |   4   |     10.1.0.1    |    0.0.0.0    | 255.255.255.255 |    ens3   |   UH  |\n  |   5   |     10.1.0.2    |    0.0.0.0    | 255.255.255.255 |    ens3   |   UH  |\n  |   6   |    127.0.0.1    |    0.0.0.0    | 255.255.255.255 |    ens3   |   UH  |\n  |   7   | 169.254.169.254 |    10.1.0.2   | 255.255.255.255 |    ens3   |  UGH  |\n  |   8   | 169.254.169.254 | 192.168.254.2 | 255.255.255.255 |    ens4   |  UGH  |\n  |   9   |  192.168.254.0  |    0.0.0.0    |  255.255.255.0  |    ens4   |   U   |\n  |   10  |  192.168.254.1  |    0.0.0.0    | 255.255.255.255 |    ens4   |   UH  |\n  |   11  |  192.168.254.2  |    0.0.0.0    | 255.255.255.255 |    ens4   |   UH  |\n  +-------+-----------------+---------------+-----------------+-----------+-------+\n\n2. The problem with having two default routes is that it may cause asymmetric routing issues, such as:\n\n   * DNS failures (I encountered this recently);\n   * intermittent connectivity;\n   * TCP handshakes that never complete;\n   * connections that work only intermittently;\n   * services becoming unreachable from certain networks.\n\n3. At the moment, tests are not running reliably, and in some cases this is caused by network-related issues.\n\n4. Throughout the previous week, I struggled with faulty DNS resolution during instance startup, which was caused by the presence of two default routes. At the moment, after the `libnss`/`systemd-resolved` update, the problem seems to be gone. However, it may reappear at any time because both routes currently have the same default metric of 100.\n\nI\u0027m currently exploring two approaches:\n\n1. Remove `TROVE_MGMT_GATEWAY` and do not attach `trove-mgmt` to the router at all. In this case, the default route will be provided by the client\u0027s network. I understand that a client may create a misconfigured network. However, if the cloud provider has configured the `trove-mgmt` interface correctly, the instance should still boot successfully even without a client network and without any Internet connectivity.\n\n2. Change the route metrics for the `trove-mgmt` interface during startup. The default value is 100, and the script changes it to 50, so that Linux prioritizes the route with the lower metric. This should eliminate potential networking issues caused by route ambiguity.\n\nThis problem is related only to the testing environment in Zuul, so it should not affect production deployments. However, in my opinion, testing environments should resemble production setups as closely as possible.\n\nWhat do you think?\n\nP.S. This MR is not directly related to the issue with `TROVE_MGMT_GATEWAY`. It is the final step required to make DevStack work for developers using the default `./stack` installation workflow. At the moment, users still need to configure DNS manually after installation or rely on `TROVE_MGMT_GATEWAY`.","commit_id":"9715c3e10e238ad1ea6dad4d995f1bcc92f7ce9a"},{"author":{"_account_id":36080,"name":"Erkin Mussurmankulov","display_name":"Eric","email":"erkin.mussurmankulov@ps.kz","username":"mongoose404","status":"PS Cloud services employee"},"change_message_id":"e31b7aa67fb53ad271e5fdf6eb968a92bf02c25b","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"2437baad_733bc1db","in_reply_to":"d0e8cc6d_c417230e","updated":"2026-06-10 10:53:14.000000000","message":"Hello, Wu, thank you for the fast reply.\n\nThere is confusion in names in your messages: `ens4` is the trove-mgmt interface, `ens3` is the client (business traffic) interface (it is true for my DevStack and for Zuul jobs instances)\n\nOk, I\u0027m fine with this approach, as long as we don\u0027t force cloud operators to make ens4 (trove-mgmt) interface as the default network. In a production environment, the instance doesn\u0027t need to access the Internet at all, so the `TROVE_MGMT_GATEWAY` parameter remains only in the Zuul testing environment.\n\nI will implement this new approach asap.","commit_id":"9715c3e10e238ad1ea6dad4d995f1bcc92f7ce9a"},{"author":{"_account_id":26285,"name":"wu.chunyang","email":"wchy1001@gmail.com","username":"wu.chunyang"},"change_message_id":"a2f955af3aaaf375c4327bfeb989072fd8a75301","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"26fb1617_e61d96c0","in_reply_to":"eb5c7948_c4a88518","updated":"2026-06-09 02:23:17.000000000","message":"hello, The default DNS should work correctly, because the private network is associated with the router, and the router has the external network access.\n\nFrom the cloud init logs: the ens3 is the management network adapter, and the ens4 is the user network adapter, after the trove-agent starts, the ens4 will be set down before performing all network operations, as a result, the default routes via the ens4 will be deleted by the kernel.\n\nSo, what problem have you encountered?\n\nmore information: https://github.com/openstack/trove/blob/master/trove/cmd/guest.py#L77","commit_id":"9715c3e10e238ad1ea6dad4d995f1bcc92f7ce9a"}],"devstack/settings":[{"author":{"_account_id":37598,"name":"Ivan Anfimov","display_name":"Ivan Anfimov","email":"lazekteam@gmail.com","username":"anfimovir"},"change_message_id":"9dfe5f3718a2d3fe87e75c4d8706e84f2fa804bc","unresolved":true,"context_lines":[{"line_number":104,"context_line":"AMQP_PORT\u003d${AMQP_PORT:-5672}"},{"line_number":105,"context_line":"REGISTRY_PORT\u003d${REGISTRY_PORT:-4000}"},{"line_number":106,"context_line":""},{"line_number":107,"context_line":"TROVE_DNS\u003d${TROVE_DNS:-8.8.8.8}"}],"source_content_type":"application/octet-stream","patch_set":1,"id":"c6424308_81da1619","line":107,"updated":"2026-05-28 09:28:24.000000000","message":"May be add alternative DNS too? 1.1.1.1","commit_id":"9715c3e10e238ad1ea6dad4d995f1bcc92f7ce9a"},{"author":{"_account_id":36080,"name":"Erkin Mussurmankulov","display_name":"Eric","email":"erkin.mussurmankulov@ps.kz","username":"mongoose404","status":"PS Cloud services employee"},"change_message_id":"78c8fc74270acd59bdfaa7cec936dd60518d758b","unresolved":false,"context_lines":[{"line_number":104,"context_line":"AMQP_PORT\u003d${AMQP_PORT:-5672}"},{"line_number":105,"context_line":"REGISTRY_PORT\u003d${REGISTRY_PORT:-4000}"},{"line_number":106,"context_line":""},{"line_number":107,"context_line":"TROVE_DNS\u003d${TROVE_DNS:-8.8.8.8}"}],"source_content_type":"application/octet-stream","patch_set":1,"id":"e835bad3_be6f21b9","line":107,"in_reply_to":"c6424308_81da1619","updated":"2026-05-28 09:36:02.000000000","message":"For a long time, trove-mgmt has worked on a single `8.8.8.8`. I think it\u0027s fine for a testing environment. It either works or doesn\u0027t.\nIf it doesn\u0027t work, we\u0027ll see it easily and just wait until it goes up. If we\u0027ll add a second DNS, some timeouts during execution may occur, which may produce some ambiguity in test results.","commit_id":"9715c3e10e238ad1ea6dad4d995f1bcc92f7ce9a"}]}