)]}'
{"/COMMIT_MSG":[{"author":{"_account_id":28522,"name":"Hervé Beraud","email":"herveberaud.pro@gmail.com","username":"hberaud"},"change_message_id":"fb29b405a2229b5de1e5770a9a60fdd670fdd67f","unresolved":true,"context_lines":[{"line_number":8,"context_line":""},{"line_number":9,"context_line":"This reverts commit 8705f67bfbd733edf923003c0a5850e7a4a5735b."},{"line_number":10,"context_line":""},{"line_number":11,"context_line":"Reason for revert: Reverting this change as it is causing failures across multiple projects (nova and designate at least). OpenSearch is showing 580 related failures in the last 24 hours. As the comment states, this was an intentional setting based on how this \"mutex\" is implemented. I think this needs to stay in the code until we can move oslo service off of eventlet."},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"Change-Id: I562dc23c6ca41dcef6f40127f071b8422f677a0f"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"3eb83a06_af6f8ad9","line":11,"range":{"start_line":11,"start_character":341,"end_line":11,"end_character":371},"updated":"2024-07-11 14:53:20.000000000","message":"And services?\n\nPossibly some services are mixing native threads and green threads without using tpool\n\nhttps://eventlet.readthedocs.io/en/latest/threading.html\n\nIn this case, oslo.log may fail to properly accessing file descriptors which are related to logging. Logger are poorly shared between native threads and green threads.\n\nI\u0027d encourage to check if logging is used in native threads in a monkey patched environment. That could the real root cause of these errors (I do not deny the fact that removing the eventlet debug helper usages in oslo.log may have triggered the problem). Moving from threading to tpool could be a good compromize.\n\nCinder removed all their logging actions from native threads to avoid falling in such scenario.","commit_id":"563f013005bf997fa06f5a720908177932f18755"},{"author":{"_account_id":11628,"name":"Michael Johnson","email":"johnsomor@gmail.com","username":"johnsom"},"change_message_id":"9a0b2fa6846ee8e8ce5c0a8a7b7e398166465f47","unresolved":true,"context_lines":[{"line_number":8,"context_line":""},{"line_number":9,"context_line":"This reverts commit 8705f67bfbd733edf923003c0a5850e7a4a5735b."},{"line_number":10,"context_line":""},{"line_number":11,"context_line":"Reason for revert: Reverting this change as it is causing failures across multiple projects (nova and designate at least). OpenSearch is showing 580 related failures in the last 24 hours. As the comment states, this was an intentional setting based on how this \"mutex\" is implemented. I think this needs to stay in the code until we can move oslo service off of eventlet."},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"Change-Id: I562dc23c6ca41dcef6f40127f071b8422f677a0f"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"d2740ebe_67e3453b","line":11,"range":{"start_line":11,"start_character":341,"end_line":11,"end_character":371},"in_reply_to":"3eb83a06_af6f8ad9","updated":"2024-07-11 14:59:09.000000000","message":"The fix for native threads is a different patch that I am not proposing to revert[1].\n\nIn the case of designate, it is only using greenthreads. The producer service runs the main code via oslo service, but it also starts another greenthread via oslo service \"loopingcall\". Both of these have logging components in them which this change can cause an exception to be raised instead of ignored in the pipes based mutex implemented in [1].\n\n[1]https://github.com/openstack/oslo.log/commit/94b9dc32ec1f52a582adbd97fe2847f7c87d6c17","commit_id":"563f013005bf997fa06f5a720908177932f18755"},{"author":{"_account_id":28522,"name":"Hervé Beraud","email":"herveberaud.pro@gmail.com","username":"hberaud"},"change_message_id":"e9add4ae2e2897dfc435f82a6a60be1454f07f7d","unresolved":true,"context_lines":[{"line_number":8,"context_line":""},{"line_number":9,"context_line":"This reverts commit 8705f67bfbd733edf923003c0a5850e7a4a5735b."},{"line_number":10,"context_line":""},{"line_number":11,"context_line":"Reason for revert: Reverting this change as it is causing failures across multiple projects (nova and designate at least). OpenSearch is showing 580 related failures in the last 24 hours. As the comment states, this was an intentional setting based on how this \"mutex\" is implemented. I think this needs to stay in the code until we can move oslo service off of eventlet."},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"Change-Id: I562dc23c6ca41dcef6f40127f071b8422f677a0f"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"ef12583d_4af2208c","line":11,"range":{"start_line":11,"start_character":341,"end_line":11,"end_character":371},"in_reply_to":"d2740ebe_67e3453b","updated":"2024-07-11 15:15:49.000000000","message":"Thanks for these details","commit_id":"563f013005bf997fa06f5a720908177932f18755"}],"/PATCHSET_LEVEL":[{"author":{"_account_id":28522,"name":"Hervé Beraud","email":"herveberaud.pro@gmail.com","username":"hberaud"},"change_message_id":"fcab287b682c4c09208d804af514d69d825709f5","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":2,"id":"a3c8e672_1691a483","updated":"2024-07-12 11:28:28.000000000","message":"I not +Wed to lets time to see if we want to also remove https://review.opendev.org/#/q/94b9dc32ec1f52a582adbd97fe2847f7c87d6c17 like Takashi suggested. See inline comments.","commit_id":"f1dde6e3f15d46d3fd89fdbcdd4d4f802cd94ccc"},{"author":{"_account_id":9816,"name":"Takashi Kajinami","email":"kajinamit@oss.nttdata.com","username":"kajinamit"},"change_message_id":"052cf6e49ab4db26090235fd9609582617dfdd5d","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":2,"id":"f82f8259_ec83a3f3","updated":"2024-07-12 09:52:13.000000000","message":"OK now I\u0027m aware this broke heat. I thought the change may have been tested before being proposed but it was not :-(\n\nI\u0027m for reverting this change given the fact that we are trying to get away from eventlet.","commit_id":"f1dde6e3f15d46d3fd89fdbcdd4d4f802cd94ccc"},{"author":{"_account_id":28522,"name":"Hervé Beraud","email":"herveberaud.pro@gmail.com","username":"hberaud"},"change_message_id":"3309a17a4b1b8d59658237d8e63fbf13823d6b21","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":2,"id":"ab16bed9_5feac2aa","in_reply_to":"29587393_1373af3b","updated":"2024-07-16 15:22:41.000000000","message":"No problem, thanks","commit_id":"f1dde6e3f15d46d3fd89fdbcdd4d4f802cd94ccc"},{"author":{"_account_id":28522,"name":"Hervé Beraud","email":"herveberaud.pro@gmail.com","username":"hberaud"},"change_message_id":"fcab287b682c4c09208d804af514d69d825709f5","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":2,"id":"8e4235e9_e3cb8a29","in_reply_to":"50ff7698_cc0c9f9b","updated":"2024-07-12 11:28:28.000000000","message":"Before encouraging Daniel to propose this patch (the one we propose to remove)  I personnally made many checks and tests against some local code (not on an Openstack service like heat) to validate that removal assumption, unfortunatelly this doesn\u0027t seem enough.\n\nI\u0027m ok to revert that patch the time to find an alternative.\n\nI\u0027m also ok to simply remove all this eventlet non sense from oslo.log as proposed by Takashi. IMO oslo.log should be agnostic to eventlet and to these mutex things, that\u0027s not the goal and the semantic of oslo.log. \n\nThe fastener alternative seems a good option.\n\nFor this reason, I\u0027d rather advocate to remove the whole eventlet stuff https://review.opendev.org/#/q/94b9dc32ec1f52a582adbd97fe2847f7c87d6c17 as we are going to drop Eventlet, else we will loose time to back and forth with interweaved things between oslo.service, oslo.log, and oslo.concurrency. Keeping this eventlet debug feature is a blocker for the whole eventlet removal, as it is not compatible with the new eventlet asyncio hub. I\u0027d rather suggest to spend time on placing fastener lock like proposed by Sean.","commit_id":"f1dde6e3f15d46d3fd89fdbcdd4d4f802cd94ccc"},{"author":{"_account_id":11604,"name":"sean mooney","email":"smooney@redhat.com","username":"sean-k-mooney"},"change_message_id":"42ea98f464e3891e14d6a45c8f78dff2335e8509","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":2,"id":"50ff7698_cc0c9f9b","in_reply_to":"71f919d3_c6c7985b","updated":"2024-07-12 10:27:09.000000000","message":"yep that what i also concluded on irc\n\nwe need to effectivly port some of the lock form oslo.concurrancy\n\nbasiclaly we need a fiar external lock liek the lockutils.lock provides\nhttps://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L253-L331\n\n\nit does that by first taking an internal fiar lock ( a fateners reader writer lock) followed by a file system lock in the form of a fasteners interproces lock\n\nhttps://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L107-L147\nhttps://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L205-L208\nhttps://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L104\n\nthat is eventlet safe and allows safe sycronisation via a shared lock between eventlets.\n\nwe cant depend on oslo.concurrance but we can depend on fasteners and replace the pip mutex with this instead.\n\n\nlong term moving lockutils to oslo.utils might be the better move but porting the 100-200 line so of need code directly to oslo.log should be quite doable\n\nif we want to go that route","commit_id":"f1dde6e3f15d46d3fd89fdbcdd4d4f802cd94ccc"},{"author":{"_account_id":9816,"name":"Takashi Kajinami","email":"kajinamit@oss.nttdata.com","username":"kajinamit"},"change_message_id":"333dbbf6e234bf8687c4b0ea0c6cc48ed866d3e8","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":2,"id":"29587393_1373af3b","in_reply_to":"a3c8e672_1691a483","updated":"2024-07-16 13:21:10.000000000","message":"Let\u0027s make small steps. We can eventually revert https://review.opendev.org/c/openstack/oslo.log/+/852443 but I\u0027d suggest we first revert this specific change to unblock bump.","commit_id":"f1dde6e3f15d46d3fd89fdbcdd4d4f802cd94ccc"},{"author":{"_account_id":9816,"name":"Takashi Kajinami","email":"kajinamit@oss.nttdata.com","username":"kajinamit"},"change_message_id":"c226d5a6479992a92d025816c405881b38fe6787","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":2,"id":"71f919d3_c6c7985b","in_reply_to":"f82f8259_ec83a3f3","updated":"2024-07-12 09:54:23.000000000","message":"Probably we should have reverted the whole 94b9dc32ec1f52a582adbd97fe2847f7c87d6c17 instead of removing the debug flag...","commit_id":"f1dde6e3f15d46d3fd89fdbcdd4d4f802cd94ccc"}]}