)]}'
{"/COMMIT_MSG":[{"author":{"_account_id":9816,"name":"Takashi Kajinami","email":"kajinamit@oss.nttdata.com","username":"kajinamit"},"change_message_id":"bcc5ee2174a20ceb33a29b3896a7b31dbe750122","unresolved":true,"context_lines":[{"line_number":8,"context_line":""},{"line_number":9,"context_line":"Under eventlet, native threads also take the logging module lock:"},{"line_number":10,"context_line":"logging.getLogger() and the first Logger.isEnabledFor() check after a"},{"line_number":11,"context_line":"log level change both need it. Examples are eventlet.tpool workers,"},{"line_number":12,"context_line":"oslo.messaging\u0027s heartbeat_in_pthread thread and asyncio loops that"},{"line_number":13,"context_line":"libraries run in their own threads. _fix_eventlet_logging() made that"},{"line_number":14,"context_line":"lock a green RLock. A green lock wakes its waiter from the hub of the"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":2,"id":"d612bca2_03ccc1e6","line":11,"range":{"start_line":11,"start_character":44,"end_line":11,"end_character":67},"updated":"2026-10-02 16:26:36.000000000","message":"I think the current patching hack was made to resolve this. Do you still see logging in eventlet.tpool broken ?","commit_id":"1c0454956224e922bc9a322682016d201f813e2d"},{"author":{"_account_id":10273,"name":"Adam Harwell","email":"flux.adam@gmail.com","username":"rm_you"},"change_message_id":"b4df429d5422e9741b4c750e6beb12db35089d49","unresolved":true,"context_lines":[{"line_number":8,"context_line":""},{"line_number":9,"context_line":"Under eventlet, native threads also take the logging module lock:"},{"line_number":10,"context_line":"logging.getLogger() and the first Logger.isEnabledFor() check after a"},{"line_number":11,"context_line":"log level change both need it. Examples are eventlet.tpool workers,"},{"line_number":12,"context_line":"oslo.messaging\u0027s heartbeat_in_pthread thread and asyncio loops that"},{"line_number":13,"context_line":"libraries run in their own threads. _fix_eventlet_logging() made that"},{"line_number":14,"context_line":"lock a green RLock. A green lock wakes its waiter from the hub of the"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":2,"id":"0abccb5a_cd5c1869","line":11,"range":{"start_line":11,"start_character":44,"end_line":11,"end_character":67},"in_reply_to":"d612bca2_03ccc1e6","updated":"2026-10-02 18:06:44.000000000","message":"Yes, in one case.\n\n- Bug 1983863 fixed the handler locks (PipeMutex).\n- This change is about the module lock, logging._lock. It is still a green RLock.\n- tpool workers also take it: in getLogger(), and in the first isEnabledFor() after a level change.\n\nOn master (eventlet 0.41.2, Python 3.12), when a tpool worker and a greenthread want this lock at the same time:\n\n- tpool worker waits for a greenthread: \"greenlet.error: Cannot switch to a different thread\", and the worker never wakes up.\n- greenthread waits for a tpool worker: the greenthread never wakes up, and later users of the lock block too.\n\nBoth cases pass with this change. The case is rare: a stress test did not hit it. But a test that forces it hits it every time. The reproducer is in the bug report.","commit_id":"1c0454956224e922bc9a322682016d201f813e2d"},{"author":{"_account_id":9816,"name":"Takashi Kajinami","email":"kajinamit@oss.nttdata.com","username":"kajinamit"},"change_message_id":"bcc5ee2174a20ceb33a29b3896a7b31dbe750122","unresolved":true,"context_lines":[{"line_number":9,"context_line":"Under eventlet, native threads also take the logging module lock:"},{"line_number":10,"context_line":"logging.getLogger() and the first Logger.isEnabledFor() check after a"},{"line_number":11,"context_line":"log level change both need it. Examples are eventlet.tpool workers,"},{"line_number":12,"context_line":"oslo.messaging\u0027s heartbeat_in_pthread thread and asyncio loops that"},{"line_number":13,"context_line":"libraries run in their own threads. _fix_eventlet_logging() made that"},{"line_number":14,"context_line":"lock a green RLock. A green lock wakes its waiter from the hub of the"},{"line_number":15,"context_line":"thread that releases it, and that hub cannot switch to a greenlet of"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":2,"id":"95d003f6_d200ca71","line":12,"range":{"start_line":12,"start_character":17,"end_line":12,"end_character":37},"updated":"2026-10-02 16:26:36.000000000","message":"We\u0027ve learned that using native thread within eventlet is nightmare and this behavior was eventually removed after we learned it just broken. I\u0027d rather not attempt to adding more hacks for that broken mixed usage but suggest using threading/async only after eventlet is removed.","commit_id":"1c0454956224e922bc9a322682016d201f813e2d"},{"author":{"_account_id":10273,"name":"Adam Harwell","email":"flux.adam@gmail.com","username":"rm_you"},"change_message_id":"b4df429d5422e9741b4c750e6beb12db35089d49","unresolved":true,"context_lines":[{"line_number":9,"context_line":"Under eventlet, native threads also take the logging module lock:"},{"line_number":10,"context_line":"logging.getLogger() and the first Logger.isEnabledFor() check after a"},{"line_number":11,"context_line":"log level change both need it. Examples are eventlet.tpool workers,"},{"line_number":12,"context_line":"oslo.messaging\u0027s heartbeat_in_pthread thread and asyncio loops that"},{"line_number":13,"context_line":"libraries run in their own threads. _fix_eventlet_logging() made that"},{"line_number":14,"context_line":"lock a green RLock. A green lock wakes its waiter from the hub of the"},{"line_number":15,"context_line":"thread that releases it, and that hub cannot switch to a greenlet of"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":2,"id":"da8fd00c_423e7ee0","line":12,"range":{"start_line":12,"start_character":17,"end_line":12,"end_character":37},"in_reply_to":"95d003f6_d200ca71","updated":"2026-10-02 18:06:44.000000000","message":"Agreed. heartbeat_in_pthread was removed in March (624f8572), so I removed it from the commit message.\n\nOur new v3 HTTP-RPC driver does not use native threads under Eventlet. Our v2 driver (https://review.opendev.org/c/openstack/oslo.messaging/+/997470) still does, and needs this fix.\n\nIf you prefer not to merge this now, I can set it to WIP. Then only our v2 chain uses it, until we decide if we move fully to v3.","commit_id":"1c0454956224e922bc9a322682016d201f813e2d"},{"author":{"_account_id":9816,"name":"Takashi Kajinami","email":"kajinamit@oss.nttdata.com","username":"kajinamit"},"change_message_id":"bcc5ee2174a20ceb33a29b3896a7b31dbe750122","unresolved":true,"context_lines":[{"line_number":10,"context_line":"logging.getLogger() and the first Logger.isEnabledFor() check after a"},{"line_number":11,"context_line":"log level change both need it. Examples are eventlet.tpool workers,"},{"line_number":12,"context_line":"oslo.messaging\u0027s heartbeat_in_pthread thread and asyncio loops that"},{"line_number":13,"context_line":"libraries run in their own threads. _fix_eventlet_logging() made that"},{"line_number":14,"context_line":"lock a green RLock. A green lock wakes its waiter from the hub of the"},{"line_number":15,"context_line":"thread that releases it, and that hub cannot switch to a greenlet of"},{"line_number":16,"context_line":"another OS thread (\"greenlet.error: Cannot switch to a different"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":2,"id":"f8c5ab79_e082870a","line":13,"range":{"start_line":13,"start_character":0,"end_line":13,"end_character":34},"updated":"2026-10-02 16:26:36.000000000","message":"Do you have any example of such libraries, causing the described problem ? I vaguely remember kombu or pyamqp has ones but there has been no similar report of the problem triggered by it.","commit_id":"1c0454956224e922bc9a322682016d201f813e2d"},{"author":{"_account_id":10273,"name":"Adam Harwell","email":"flux.adam@gmail.com","username":"rm_you"},"change_message_id":"b4df429d5422e9741b4c750e6beb12db35089d49","unresolved":true,"context_lines":[{"line_number":10,"context_line":"logging.getLogger() and the first Logger.isEnabledFor() check after a"},{"line_number":11,"context_line":"log level change both need it. Examples are eventlet.tpool workers,"},{"line_number":12,"context_line":"oslo.messaging\u0027s heartbeat_in_pthread thread and asyncio loops that"},{"line_number":13,"context_line":"libraries run in their own threads. _fix_eventlet_logging() made that"},{"line_number":14,"context_line":"lock a green RLock. A green lock wakes its waiter from the hub of the"},{"line_number":15,"context_line":"thread that releases it, and that hub cannot switch to a greenlet of"},{"line_number":16,"context_line":"another OS thread (\"greenlet.error: Cannot switch to a different"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":2,"id":"94478b30_223f02ce","line":13,"range":{"start_line":13,"start_character":0,"end_line":13,"end_character":34},"in_reply_to":"f8c5ab79_e082870a","updated":"2026-10-02 18:06:44.000000000","message":"Only our own code: our v2 HTTP-RPC driver runs an asyncio loop in a native thread, and logs from it. Changing log levels made the service hang within seconds. This will likely go away when we move to v3.\n\nI added the other uses of native threads that I found to the bug. They came from an automated code search, so the list may not be complete. I removed this sentence from the commit message.","commit_id":"1c0454956224e922bc9a322682016d201f813e2d"},{"author":{"_account_id":9816,"name":"Takashi Kajinami","email":"kajinamit@oss.nttdata.com","username":"kajinamit"},"change_message_id":"bcc5ee2174a20ceb33a29b3896a7b31dbe750122","unresolved":true,"context_lines":[{"line_number":15,"context_line":"thread that releases it, and that hub cannot switch to a greenlet of"},{"line_number":16,"context_line":"another OS thread (\"greenlet.error: Cannot switch to a different"},{"line_number":17,"context_line":"thread\"). The waiter is never woken but stays queued, so every later"},{"line_number":18,"context_line":"user of the lock blocks forever and the service hangs. With a native"},{"line_number":19,"context_line":"thread logging, a SIGHUP that reloads log_config_append triggers this"},{"line_number":20,"context_line":"within one or two reloads."},{"line_number":21,"context_line":""},{"line_number":22,"context_line":"Use a recursive lock built on a native lock instead. A thread that runs"},{"line_number":23,"context_line":"no eventlet hub blocks on it, and a greenthread retries after a short"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":2,"id":"6b2bb4ad_45164d84","line":20,"range":{"start_line":18,"start_character":55,"end_line":20,"end_character":26},"updated":"2026-10-02 16:26:36.000000000","message":"Can you report a bug for this, with some details about real scenarios where this problem is observed.","commit_id":"1c0454956224e922bc9a322682016d201f813e2d"},{"author":{"_account_id":10273,"name":"Adam Harwell","email":"flux.adam@gmail.com","username":"rm_you"},"change_message_id":"b4df429d5422e9741b4c750e6beb12db35089d49","unresolved":true,"context_lines":[{"line_number":15,"context_line":"thread that releases it, and that hub cannot switch to a greenlet of"},{"line_number":16,"context_line":"another OS thread (\"greenlet.error: Cannot switch to a different"},{"line_number":17,"context_line":"thread\"). The waiter is never woken but stays queued, so every later"},{"line_number":18,"context_line":"user of the lock blocks forever and the service hangs. With a native"},{"line_number":19,"context_line":"thread logging, a SIGHUP that reloads log_config_append triggers this"},{"line_number":20,"context_line":"within one or two reloads."},{"line_number":21,"context_line":""},{"line_number":22,"context_line":"Use a recursive lock built on a native lock instead. A thread that runs"},{"line_number":23,"context_line":"no eventlet hub blocks on it, and a greenthread retries after a short"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":2,"id":"14ccf40d_e5e03771","line":20,"range":{"start_line":18,"start_character":55,"end_line":20,"end_character":26},"in_reply_to":"6b2bb4ad_45164d84","updated":"2026-10-02 18:06:44.000000000","message":"Done: https://bugs.launchpad.net/oslo.log/+bug/2169320, with a reproducer for both cases, and the affected use cases in a comment.\n\nThe SIGHUP case was seen with our v2 driver, so PS3 describes the tpool case instead.","commit_id":"1c0454956224e922bc9a322682016d201f813e2d"}]}