)]}'
{"/PATCHSET_LEVEL":[{"author":{"_account_id":11628,"name":"Michael Johnson","email":"johnsomor@gmail.com","username":"johnsom"},"change_message_id":"585e3f40f039cc6c8e93c3fdaac0338c51e7993d","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"6bf7f2f3_8bca734c","updated":"2025-09-15 19:53:01.000000000","message":"LGTM","commit_id":"f17744640da168d5a4e3211731d8be8a5ffaaee7"},{"author":{"_account_id":37881,"name":"Wesley Hershberger","display_name":"Wesley Hershberger","email":"wesley.hershberger@canonical.com","username":"whershberger","status":"Support Engineering @ Canonical"},"change_message_id":"64281c1e435fc75853a069a52e5bd4bc187c9e55","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"6508e17c_d0a124cf","updated":"2025-08-18 14:47:14.000000000","message":"grenade.sh_log\n```\n2025-08-15 14:23:02.654 | + /opt/stack/new/octavia/devstack/upgrade/resources.sh:verify:78 :   openstack loadbalancer show -f value -c operating_status lb1\n2025-08-15 14:23:02.654 | + /opt/stack/new/octavia/devstack/upgrade/resources.sh:verify:78 :   grep -q ONLINE\n2025-08-15 14:23:03.889 | Unknown Error (HTTP 500) (Request-ID: None)\n2025-08-15 14:23:04.103 | + /opt/stack/new/octavia/devstack/upgrade/resources.sh:verify:1 :   exit_trap\n2025-08-15 14:23:04.108 | + /opt/stack/new/grenade/functions:exit_trap:115 :   local r\u003d1\n2025-08-15 14:23:04.113 | 1 verify /opt/stack/new/octavia/devstack/upgrade/resources.sh\n2025-08-15 14:23:04.113 | 123 main /opt/stack/new/octavia/devstack/upgrade/resources.sh\n2025-08-15 14:23:04.113 | Exit code: 1\n2025-08-15 14:23:06.833 | + inc/plugin:resources:123                 :   die 123 \u0027Failed to run /opt/stack/new/octavia/devstack/upgrade/resources.sh verify post-upgrade\u0027\n2025-08-15 14:23:06.837 | + /opt/stack/new/devstack/functions-common:die:285 :   local exitcode\u003d1\n2025-08-15 14:23:06.842 | [Call Trace]\n2025-08-15 14:23:06.842 | ./grenade.sh:361:resources\n2025-08-15 14:23:06.842 | /opt/stack/new/grenade/inc/plugin:123:die\n2025-08-15 14:23:06.851 | [ERROR] /opt/stack/new/grenade/inc/plugin:123 Failed to run /opt/stack/new/octavia/devstack/upgrade/resources.sh verify post-upgrade\n2025-08-15 14:23:07.856 | 1 die /opt/stack/new/devstack/functions-common\n2025-08-15 14:23:07.856 | 123 resources /opt/stack/new/grenade/inc/plugin\n2025-08-15 14:23:07.856 | 361 main ./grenade.sh\n2025-08-15 14:23:07.856 | Exit code: 1\n```\nscreen-o-api.txt\n```\nAug 15 14:23:03.887875 npf3e512846d6c4 devstack@o-api.service[183764]: --- no python application found, check your startup logs for errors ---\nAug 15 14:23:03.887875 npf3e512846d6c4 devstack@o-api.service[183764]: [pid: 183764|app: -1|req: -1/1] 199.204.45.122 () {56 vars in 1216 bytes} [Fri Aug 15 14:23:03 2025] GET /load-balancer/v2.0/lbaas/loadbalancers?name\u003dlb1 \u003d\u003e generated 21 bytes in 0 msecs (HTTP/1.1 500) 3 headers in 102 bytes (0 switches on core 0)\n```\nI\u0027m not sure which startup logs are being referred to here, but o-api was restarted just before this and spat out a backtrace with `pkg_resources.DistributionNotFound: The \u0027oslo.cache\u003e\u003d1.26.0\u0027 distribution was not found and is required by keystonemiddleware`\n\nLooks like https://bugs.launchpad.net/octavia/+bug/2109665; I haven\u0027t dug into the details but I\u0027m wondering if I7e0b17dbf9284db27e183a9a27e716266e26a237 actually fixed this...","commit_id":"f17744640da168d5a4e3211731d8be8a5ffaaee7"},{"author":{"_account_id":37881,"name":"Wesley Hershberger","display_name":"Wesley Hershberger","email":"wesley.hershberger@canonical.com","username":"whershberger","status":"Support Engineering @ Canonical"},"change_message_id":"1eba968746c1167de6149da33b56912e2db46013","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"79c344d2_d8e533c7","updated":"2025-08-18 16:16:00.000000000","message":"recheck Have looked at this a little more; other picks don\u0027t make much sense. Would like to make sure this isn\u0027t a one-off","commit_id":"f17744640da168d5a4e3211731d8be8a5ffaaee7"},{"author":{"_account_id":37881,"name":"Wesley Hershberger","display_name":"Wesley Hershberger","email":"wesley.hershberger@canonical.com","username":"whershberger","status":"Support Engineering @ Canonical"},"change_message_id":"562f6abab94ed351d1e10c495311febb356c1b22","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"b4deb92f_a280444f","in_reply_to":"1ea48ae3_29df9565","updated":"2025-08-26 21:10:43.000000000","message":"I missed this in my test run:\n\n```\n2025-08-22 18:53:04.765 | + /opt/stack/new/octavia/devstack/upgrade/resources.sh:verify:80 :   ls -la /opt/stack/data/venv/lib/python3.12/site-packages\n2025-08-22 18:53:04.776 | total 3788\n...\n2025-08-22 18:53:04.782 | drwxrwxr-x   6 stack stack    4096 Aug 22 18:48 oslo_cache\n2025-08-22 18:53:04.782 | drwxrwxr-x   2 stack stack    4096 Aug 22 18:48 oslo_cache-3.10.2.dist-info\n...\n```\n\nSo it does look like new setuptools installed oslo.cache in the global venv :/","commit_id":"f17744640da168d5a4e3211731d8be8a5ffaaee7"},{"author":{"_account_id":37881,"name":"Wesley Hershberger","display_name":"Wesley Hershberger","email":"wesley.hershberger@canonical.com","username":"whershberger","status":"Support Engineering @ Canonical"},"change_message_id":"a96e15d34b4b899b2c6c84708ca73392ebc12dba","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"4ae2fdf5_421d5afd","in_reply_to":"3b3397b1_8632e071","updated":"2025-08-25 22:16:13.000000000","message":"Hey Gregory,\n\nI have confirmed that the breakage is not caused by my patch (expected) with [test run](https://review.opendev.org/c/openstack/octavia/+/958283). We can also reject my above hypothesis; oslo.cache simply isn\u0027t present in `/opt/stack/data/venv`.\n\nI did a little research on venvs in devstack and the blame indicates that the octavia plugin was being implemented around the same time that venv support was added to devstack. It looks like octavia does not run in a venv ([docs](https://docs.openstack.org/devstack/latest/configuration.html#virtual-environments)). Is there any reason for this? I wonder if that\u0027s a viable solution to this problem.\n\nI\u0027m not the most familiar with devstack or grenade but would it be sufficient to simply include something in `devstack/settings` like the cinder plugin does in order to force grenade to run octavia in a venv?\n```\nif [[ ${USE_VENV} \u003d True ]]; then\n    PROJECT_VENV[\"cinder\"]\u003d${CINDER_DIR}.venv\n    CINDER_BIN_DIR\u003d${PROJECT_VENV[\"cinder\"]}/bin\nelse\n    CINDER_BIN_DIR\u003d$(get_python_exec_prefix)\nfi\n```\n\nSince the failure is clearly not caused by this patch I\u0027d really appreciate some assistance getting this sorted out; currently looks like all stable backports for octavia are blocked on this.","commit_id":"f17744640da168d5a4e3211731d8be8a5ffaaee7"},{"author":{"_account_id":29244,"name":"Gregory Thiemonge","email":"gthiemon@redhat.com","username":"gthiemonge"},"change_message_id":"15f19da4a27484ee772fef99197148568ff23917","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"6833420f_f31b0532","in_reply_to":"4ae2fdf5_421d5afd","updated":"2025-08-26 07:04:20.000000000","message":"thanks for the heads up, we will take a look\n\noctavia is installed in a global venv: /opt/stack/data/venv/\n\nin the logs I see that oslo.cache has been installed there too:\n\n```\n2025-08-26 00:55:57.615 | Requirement already satisfied: oslo.cache\u003e\u003d1.26.0 in /opt/stack/data/venv/lib/python3.12/site-packages (from keystonemiddleware\u003e\u003d9.5.0-\u003eoctavia\u003d\u003d16.0.1.dev3) (3.10.2)\n```","commit_id":"f17744640da168d5a4e3211731d8be8a5ffaaee7"},{"author":{"_account_id":37881,"name":"Wesley Hershberger","display_name":"Wesley Hershberger","email":"wesley.hershberger@canonical.com","username":"whershberger","status":"Support Engineering @ Canonical"},"change_message_id":"baf0730847a6035ec369c1d0edd531c29fa975b9","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"fb8ad62a_6310bda8","in_reply_to":"6508e17c_d0a124cf","updated":"2025-08-18 15:04:01.000000000","message":"Wondering if I5d9bb5ef52217ed5d2361e5855e9f5fc2c1b4111 is needed.","commit_id":"f17744640da168d5a4e3211731d8be8a5ffaaee7"},{"author":{"_account_id":29244,"name":"Gregory Thiemonge","email":"gthiemon@redhat.com","username":"gthiemonge"},"change_message_id":"a0407419ee591c1e2240e2510868e24a0875c99f","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"2bf1b2e0_12f6e201","in_reply_to":"6508e17c_d0a124cf","updated":"2025-08-18 15:02:45.000000000","message":"yeah it might be related to this octavia-wsgi patch","commit_id":"f17744640da168d5a4e3211731d8be8a5ffaaee7"},{"author":{"_account_id":29244,"name":"Gregory Thiemonge","email":"gthiemon@redhat.com","username":"gthiemonge"},"change_message_id":"bdc73f4f1deb0c9a6369c25ccc82ec21a00fb152","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"1ea48ae3_29df9565","in_reply_to":"6833420f_f31b0532","updated":"2025-08-26 07:06:47.000000000","message":"but this venv seems to use the old setuptools:\n\n```\n2025-08-26 00:55:57.740 | Requirement already satisfied: setuptools in /opt/stack/data/venv/lib/python3.12/site-packages (from Paste\u003e\u003d2.0.2-\u003eoslo.service\u003e\u003d1.24.0-\u003eoslo.messaging\u003e\u003d14.1.0-\u003eoctavia\u003d\u003d16.0.1.dev3) (75.3.2)\n```","commit_id":"f17744640da168d5a4e3211731d8be8a5ffaaee7"},{"author":{"_account_id":37881,"name":"Wesley Hershberger","display_name":"Wesley Hershberger","email":"wesley.hershberger@canonical.com","username":"whershberger","status":"Support Engineering @ Canonical"},"change_message_id":"e602c86d9695407527a0eeadfffd5dad699127a8","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":1,"id":"ffc138df_4d2fb3ac","in_reply_to":"b4deb92f_a280444f","updated":"2025-08-28 16:27:18.000000000","message":"For future travelers: https://bugs.launchpad.net/grenade/+bug/2121578","commit_id":"f17744640da168d5a4e3211731d8be8a5ffaaee7"},{"author":{"_account_id":37881,"name":"Wesley Hershberger","display_name":"Wesley Hershberger","email":"wesley.hershberger@canonical.com","username":"whershberger","status":"Support Engineering @ Canonical"},"change_message_id":"4524ba1bab24edd6ce2367791004bfa3c35a63ff","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"3b3397b1_8632e071","in_reply_to":"fb8ad62a_6310bda8","updated":"2025-08-21 17:31:08.000000000","message":"I\u0027ve done a bunch more research and I think stephfin\u0027s request in https://github.com/pypa/setuptools/issues/5051 would resolve this. All logs from the most recent test failure:\n\n/opt/stack/data has setuptools\u003d\u003d75.3.2\n```\n2025-08-18 16:47:32.957 | Requirement already satisfied: setuptools in /opt/stack/data/venv/lib/python3.12/site-packages (from pbr) (75.3.2)\n```\n\n/opt/stack/new has setuptools\u003d\u003d80.9.0\n```\n2025-08-18 16:47:24.736 | + lib/infra:install_infra:34               :   PIP_VIRTUAL_ENV\u003d/opt/stack/new/requirements/.venv\n2025-08-18 16:47:24.740 | + lib/infra:install_infra:34               :   pip_install -U pbr \u0027setuptools[core]\u0027\n2025-08-18 16:47:24.777 | + inc/python:pip_install:210               :   env http_proxy\u003d https_proxy\u003d no_proxy\u003d PIP_FIND_LINKS\u003d /opt/stack/new/requirements/.venv/bin/pip install -c /opt/stack/new/requirements/upper-constraints.txt -U pbr \u0027setuptools[core]\u0027\n2025-08-18 16:47:25.422 | Looking in indexes: https://mirror.ca-ymq-1.vexxhost.opendev.org/pypi/simple, https://mirror.ca-ymq-1.vexxhost.opendev.org/wheel/ubuntu-24.04-x86_64\n2025-08-18 16:47:25.501 | Collecting pbr\n2025-08-18 16:47:25.504 |   Downloading https://mirror.ca-ymq-1.vexxhost.opendev.org/pypifiles/packages/47/ac/684d71315abc7b1214d59304e23a982472967f6bf4bde5a98f1503f648dc/pbr-6.1.1-py2.py3-none-any.whl.metadata (3.4 kB)\n2025-08-18 16:47:25.744 | Collecting setuptools[core]\n2025-08-18 16:47:25.745 |   Using cached https://mirror.ca-ymq-1.vexxhost.opendev.org/pypifiles/packages/a3/dc/17031897dae0efacfea57dfd3a82fdd2a2aeb58e0ff71b77b87e44edc772/setuptools-80.9.0-py3-none-any.whl.metadata (6.6 kB)\n```\n\nBetween those versions, https://github.com/pypa/setuptools/pull/4766 landed, which replaces `.` in whl filenames with `_`.\n\nIf for some reason an oslo.cache whl downloaded by 80.9.0 was expected to work with 75.3.2, you\u0027d see the failure we\u0027re seeing here. Unclear why that would be...","commit_id":"f17744640da168d5a4e3211731d8be8a5ffaaee7"},{"author":{"_account_id":37881,"name":"Wesley Hershberger","display_name":"Wesley Hershberger","email":"wesley.hershberger@canonical.com","username":"whershberger","status":"Support Engineering @ Canonical"},"change_message_id":"1e2463694b35f33bc14c2b14b4b7ead9ef5449f3","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":1,"id":"62403483_0c41e5f5","in_reply_to":"ffc138df_4d2fb3ac","updated":"2025-09-10 19:29:53.000000000","message":"recheck Merged a temporary workaround https://review.opendev.org/c/openstack/octavia/+/958808","commit_id":"f17744640da168d5a4e3211731d8be8a5ffaaee7"}]}