)]}'
{"neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py":[{"author":{"_account_id":16688,"name":"Rodolfo Alonso","email":"ralonsoh@redhat.com","username":"rodolfo-alonso-hernandez"},"change_message_id":"9cee50665366908acfda6dc671984d89eeee7fb8","unresolved":true,"context_lines":[{"line_number":597,"context_line":"                    time.sleep(self.conf.AGENT.report_interval)"},{"line_number":598,"context_line":"                    continue"},{"line_number":599,"context_line":"            except (oslo_messaging.MessagingTimeout,"},{"line_number":600,"context_line":"                    oslo_messaging.RemoteError) as e:"},{"line_number":601,"context_line":"                LOG.warning(\u0027l2-agent cannot contact neutron server. \u0027"},{"line_number":602,"context_line":"                            \u0027Check connectivity to neutron server. \u0027"},{"line_number":603,"context_line":"                            \u0027Retrying... \u0027"}],"source_content_type":"text/x-python","patch_set":1,"id":"2840d504_3c713711","line":600,"range":{"start_line":600,"start_character":35,"end_line":600,"end_character":46},"updated":"2026-09-16 08:43:10.000000000","message":"The situation reported in the bug cannot happen. The `Upgrade strategy` [1] defined for Neutron sets the golden rule: `It\u0027s always assumed that neutron-server component is upgraded before agents.`\n\nThis is the fundamental upgrade order for Neutron:\n* First, all neutron-servers are upgraded.\n* Then, neutron agents are upgraded (per node).\n\nWhy this order matters? The documentation states two constraints that rolling upgrades must satisfy:\n* Older services should be able to talk with newer services.\n* Older services should not require the database to have older schema.\n\nBy upgrading the server first, Neutron avoids needing backwards compatibility code on the agent side. The server is the only component that accesses the database directly, so it\u0027s isolated as the upgrade boundary.\n\nThe critical AMQP consideration. \nThe documentation has a dedicated section on this exact issue:\n```\nSince it\u0027s always assumed that neutron-server component is upgraded before agents, only the former should handle both old and new RPC versions.\nThe implication of that is that no code that handles UnsupportedVersion oslo.messaging exceptions belongs to agent code.\n```\n\nWhen the bug actually happens? The bug occurs when this upgrade order is violated: an agent is restarted (or upgraded) before all server replicas have been upgraded. During a rolling server upgrade with multiple replicas, if an agent connects to a server replica that hasn\u0027t been upgraded yet, the agent (at version 1.4) requests an RPC interface the server (still at 1.3) doesn\u0027t implement → UnsupportedVersion.\n\n[1]https://docs.openstack.org/neutron/latest/contributor/internals/upgrade.html","commit_id":"1dea41effd6b2bf083a888d51366026cf6d369a4"},{"author":{"_account_id":39176,"name":"Sravan Sajeev","display_name":"Sravan Sajeev","email":"sravan.sajeev@windriver.com","username":"ssajeev"},"change_message_id":"9ec90013a5e88a7f4ab465a1de1d3aa39f2ab0c8","unresolved":true,"context_lines":[{"line_number":597,"context_line":"                    time.sleep(self.conf.AGENT.report_interval)"},{"line_number":598,"context_line":"                    continue"},{"line_number":599,"context_line":"            except (oslo_messaging.MessagingTimeout,"},{"line_number":600,"context_line":"                    oslo_messaging.RemoteError) as e:"},{"line_number":601,"context_line":"                LOG.warning(\u0027l2-agent cannot contact neutron server. \u0027"},{"line_number":602,"context_line":"                            \u0027Check connectivity to neutron server. \u0027"},{"line_number":603,"context_line":"                            \u0027Retrying... \u0027"}],"source_content_type":"text/x-python","patch_set":1,"id":"af2de3ee_08174fea","line":600,"range":{"start_line":600,"start_character":35,"end_line":600,"end_character":46},"in_reply_to":"2840d504_3c713711","updated":"2026-09-16 15:20:27.000000000","message":"Thanks for looking at this. I\u0027m not arguing with the doctrine, server-first is the documented order and I\u0027m not asking anyone to relax it. I just don\u0027t think it settles this particular case, for three reasons.\n\nThe first is that the agent code already does this. setup_rpc() is called at\novs_neutron_agent.py:306, and two lines later, at 308, we build OVSDVRNeutronAgent. Its __init__ calls get_dvr_mac_address(), which catches oslo_messaging.RemoteError specifically for the case\nwhere, in its own words, the server does not support this RPC API. The DHCP agent does something similar at agent.py:313 and 320, branching on RemoteError.exc_type.\n\nI want to be fair about how far that goes. The DVR call is guarded by enable_distributed_routing so it doesn\u0027t run everywhere, and it tolerates the error and moves on rather than retrying, so it isn\u0027t the same behaviour I\u0027m proposing. I\u0027m offering it as support for the principle, that an\nagent shouldn\u0027t treat a server-version RemoteError as an unhandled crash, not as precedent for retrying.\n\nThe second is the part I\u0027d most like you to look at, because it changed my own mind about what to write here. My first instinct was to sidestep the doctrine completely by copying get_dvr_mac_address() exactly: catch it, log it, sys.exit(1), tolerate nothing. That turns out not to work, and I don\u0027t think it works anywhere in this agent.\n\nThe agent body doesn\u0027t run on the process main thread, ovs_oskenapp.py starts it with hub.spawn(agent_main_wrapper, ...). SystemExit isn\u0027t an Exception subclass so it slips past agent_main_wrapper\u0027s except Exception, but the finally block still routes into AppManager.close(), which is the same teardown a crash takes. And that path can\u0027t end the process under os-ken\u0027s native hub: run_apps() calls t.kill(), which a native-hub thread doesn\u0027t have, and if you fix that to hub.kill(t) then hub.joinall() waits forever on threads that aren\u0027t daemons.\nI filed that separately as LP #2167375 with a small reproduction, since it\u0027s really an os-ken problem.\n\nWhich also means the seven sys.exit(1) and raise SystemExit(1) calls already in\novs_neutron_agent.py and ovs_dvr_neutron_agent.py don\u0027t terminate anything either, including the DVR one above.\n\nSo what actually happens today is that the main thread dies and the process keeps running. The pod stays Running with zero restarts, the agent shows as dead in openstack network agent list, nothing alarms, and br-int keeps forwarding its existing flows while programming nothing new, so you don\u0027t notice until someone launches or migrates an instance. I had hosts sit like that for\n19 hours before I spotted it. That\u0027s how I ended up at retry: the choice isn\u0027t between tolerating skew and failing cleanly, it\u0027s between tolerating skew and hanging quietly.\n\nThe third is that the ordering isn\u0027t always something a deployer can arrange. In openstack-helm the only dependency the ovs-agent declares on the server is\ndependencies.static.ovs_agent.services, set to oslo_messaging and network, and that resolves to whether the Service has ready endpoints. With more than one server replica that stays true right through a rolling update, so it never orders anything, and there are no hook weights between the\nserver Deployment and the agent DaemonSets. Any openstack-helm deployment doing a rolling serverupdate has this window open by default.\n\nNone of that is me insisting on this particular patch. If there\u0027s a shape you\u0027d rather have here, I\u0027m happy to rework it.","commit_id":"1dea41effd6b2bf083a888d51366026cf6d369a4"},{"author":{"_account_id":39176,"name":"Sravan Sajeev","display_name":"Sravan Sajeev","email":"sravan.sajeev@windriver.com","username":"ssajeev"},"change_message_id":"6e32e278bd9c670a47e3af5b64bb54965925de21","unresolved":true,"context_lines":[{"line_number":597,"context_line":"                    time.sleep(self.conf.AGENT.report_interval)"},{"line_number":598,"context_line":"                    continue"},{"line_number":599,"context_line":"            except (oslo_messaging.MessagingTimeout,"},{"line_number":600,"context_line":"                    oslo_messaging.RemoteError) as e:"},{"line_number":601,"context_line":"                LOG.warning(\u0027l2-agent cannot contact neutron server. \u0027"},{"line_number":602,"context_line":"                            \u0027Check connectivity to neutron server. \u0027"},{"line_number":603,"context_line":"                            \u0027Retrying... \u0027"}],"source_content_type":"text/x-python","patch_set":1,"id":"aa022cfb_c5a35415","line":600,"range":{"start_line":600,"start_character":35,"end_line":600,"end_character":46},"in_reply_to":"af2de3ee_08174fea","updated":"2026-09-21 14:02:23.000000000","message":"@ralonsoh@redhat.com please take a look at my previous comment when you can. Thanks!","commit_id":"1dea41effd6b2bf083a888d51366026cf6d369a4"}]}
