)]}'
{"specs/kilo/parallel-periodic-task.rst":[{"author":{"_account_id":2472,"name":"Doug Hellmann","email":"dhellmann@redhat.com","username":"doug-hellmann"},"change_message_id":"dea25f0cfb1928f0f9948470686dab347b8d2023","unresolved":false,"context_lines":[{"line_number":2,"context_line":"Support for running periodic tasks in a separate thread"},{"line_number":3,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":4,"context_line":""},{"line_number":5,"context_line":"https://blueprints.launchpad.net/oslo-incubator/+spec/parallel-periodic-task"},{"line_number":6,"context_line":""},{"line_number":7,"context_line":"This blueprint enables developers to request running designated periodic tasks"},{"line_number":8,"context_line":"on a thread group in a separate thread."}],"source_content_type":"text/x-rst","patch_set":1,"id":"5a890539_71c6f069","line":5,"updated":"2014-11-13 20:16:59.000000000","message":"This will need to go into oslo.service, so it should be part of that LP project when it has been created. I can help you retarget the blueprint if you don\u0027t have whatever permissions are needed to do that.","commit_id":"ccb09785ccbbbef4d2a8c6607a21815ee2ebe527"},{"author":{"_account_id":10239,"name":"Dmitry Tantsur","email":"dtantsur@protonmail.com","username":"dtantsur"},"change_message_id":"209734f28a2d4aecd922ce6155915ff47e0504d0","unresolved":false,"context_lines":[{"line_number":2,"context_line":"Support for running periodic tasks in a separate thread"},{"line_number":3,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":4,"context_line":""},{"line_number":5,"context_line":"https://blueprints.launchpad.net/oslo-incubator/+spec/parallel-periodic-task"},{"line_number":6,"context_line":""},{"line_number":7,"context_line":"This blueprint enables developers to request running designated periodic tasks"},{"line_number":8,"context_line":"on a thread group in a separate thread."}],"source_content_type":"text/x-rst","patch_set":1,"id":"5a890539_e3f75b57","line":5,"in_reply_to":"5a890539_71c6f069","updated":"2014-11-14 13:03:27.000000000","message":"yeah, I temporary targeted to oslo-incubator. Would be great if you could retarget it after the project creation.","commit_id":"ccb09785ccbbbef4d2a8c6607a21815ee2ebe527"},{"author":{"_account_id":2472,"name":"Doug Hellmann","email":"dhellmann@redhat.com","username":"doug-hellmann"},"change_message_id":"dea25f0cfb1928f0f9948470686dab347b8d2023","unresolved":false,"context_lines":[{"line_number":31,"context_line":""},{"line_number":32,"context_line":"* Add ``thread_group`` argument to the ``run_periodic_tasks`` method of"},{"line_number":33,"context_line":"  ``PeriodicTasks`` class with default value of None. If present, it should be"},{"line_number":34,"context_line":"  a ``ThreadGroup`` on which to run parallel tasks."},{"line_number":35,"context_line":""},{"line_number":36,"context_line":"* Modify ``run_periodic_tasks`` so that parallel tasks are executed on a"},{"line_number":37,"context_line":"  thread group, while the other executed as before."}],"source_content_type":"text/x-rst","patch_set":1,"id":"5a890539_b17bf8f3","line":34,"updated":"2014-11-13 20:16:59.000000000","message":"Who is responsible for creating and managing the ThreadGroup? Is this a real thread, or an eventlet thread? Does it need to be a \"group\" if there\u0027s only one function being called? Is a group used to handle threads dying?","commit_id":"ccb09785ccbbbef4d2a8c6607a21815ee2ebe527"},{"author":{"_account_id":10239,"name":"Dmitry Tantsur","email":"dtantsur@protonmail.com","username":"dtantsur"},"change_message_id":"209734f28a2d4aecd922ce6155915ff47e0504d0","unresolved":false,"context_lines":[{"line_number":31,"context_line":""},{"line_number":32,"context_line":"* Add ``thread_group`` argument to the ``run_periodic_tasks`` method of"},{"line_number":33,"context_line":"  ``PeriodicTasks`` class with default value of None. If present, it should be"},{"line_number":34,"context_line":"  a ``ThreadGroup`` on which to run parallel tasks."},{"line_number":35,"context_line":""},{"line_number":36,"context_line":"* Modify ``run_periodic_tasks`` so that parallel tasks are executed on a"},{"line_number":37,"context_line":"  thread group, while the other executed as before."}],"source_content_type":"text/x-rst","patch_set":1,"id":"5a890539_e353db36","line":34,"in_reply_to":"5a890539_b17bf8f3","updated":"2014-11-14 13:03:27.000000000","message":"In Ironic we already use a thread group to schedule run_periodic_task call, so I think it\u0027s reasonable to reuse it. Creating something separate will make harder to reason about number of threads to execute at max.\n\n\u003e Is a group used to handle threads dying?\n\nSorry, I didn\u0027t get the question. Could you elaborate?","commit_id":"ccb09785ccbbbef4d2a8c6607a21815ee2ebe527"},{"author":{"_account_id":2472,"name":"Doug Hellmann","email":"dhellmann@redhat.com","username":"doug-hellmann"},"change_message_id":"dea25f0cfb1928f0f9948470686dab347b8d2023","unresolved":false,"context_lines":[{"line_number":38,"context_line":""},{"line_number":39,"context_line":"* Ensure that a parallel task is never run again while the previous run"},{"line_number":40,"context_line":"  didn\u0027t finish. Add a new dictionary on ``Performance`` class to track this."},{"line_number":41,"context_line":"  Issue a warning if this situation happens."},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"* Issue a warning, if a parallel task is encountered, but no ``thread_group``"},{"line_number":44,"context_line":"  is provided."}],"source_content_type":"text/x-rst","patch_set":1,"id":"5a890539_316f082d","line":41,"updated":"2014-11-13 20:16:59.000000000","message":"If the thread just loops calling the single parallel function at its period, we don\u0027t need to protect it, right?","commit_id":"ccb09785ccbbbef4d2a8c6607a21815ee2ebe527"},{"author":{"_account_id":10239,"name":"Dmitry Tantsur","email":"dtantsur@protonmail.com","username":"dtantsur"},"change_message_id":"209734f28a2d4aecd922ce6155915ff47e0504d0","unresolved":false,"context_lines":[{"line_number":38,"context_line":""},{"line_number":39,"context_line":"* Ensure that a parallel task is never run again while the previous run"},{"line_number":40,"context_line":"  didn\u0027t finish. Add a new dictionary on ``Performance`` class to track this."},{"line_number":41,"context_line":"  Issue a warning if this situation happens."},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"* Issue a warning, if a parallel task is encountered, but no ``thread_group``"},{"line_number":44,"context_line":"  is provided."}],"source_content_type":"text/x-rst","patch_set":1,"id":"5a890539_031b8781","line":41,"in_reply_to":"5a890539_316f082d","updated":"2014-11-14 13:03:27.000000000","message":"What if this function takes longer to execute (e.g. for some transient reasons) than the period? I don\u0027t think we want several instances of the same call to be executed in parallel.","commit_id":"ccb09785ccbbbef4d2a8c6607a21815ee2ebe527"},{"author":{"_account_id":2472,"name":"Doug Hellmann","email":"dhellmann@redhat.com","username":"doug-hellmann"},"change_message_id":"dea25f0cfb1928f0f9948470686dab347b8d2023","unresolved":false,"context_lines":[{"line_number":41,"context_line":"  Issue a warning if this situation happens."},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"* Issue a warning, if a parallel task is encountered, but no ``thread_group``"},{"line_number":44,"context_line":"  is provided."},{"line_number":45,"context_line":""},{"line_number":46,"context_line":"Alternatives"},{"line_number":47,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"5a890539_5153b46a","line":44,"updated":"2014-11-13 20:16:59.000000000","message":"How is the warning issued? LOG.warn() implies this is something the deployer can fix, but it really isn\u0027t. Should we use LOG.debug()? Should the function be invoked in some other way to ensure it is run, or should we abort the process with an error that the setup is not coded correctly?","commit_id":"ccb09785ccbbbef4d2a8c6607a21815ee2ebe527"},{"author":{"_account_id":10239,"name":"Dmitry Tantsur","email":"dtantsur@protonmail.com","username":"dtantsur"},"change_message_id":"209734f28a2d4aecd922ce6155915ff47e0504d0","unresolved":false,"context_lines":[{"line_number":41,"context_line":"  Issue a warning if this situation happens."},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"* Issue a warning, if a parallel task is encountered, but no ``thread_group``"},{"line_number":44,"context_line":"  is provided."},{"line_number":45,"context_line":""},{"line_number":46,"context_line":"Alternatives"},{"line_number":47,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"5a890539_1e075e1a","line":44,"in_reply_to":"5a890539_5153b46a","updated":"2014-11-14 13:03:27.000000000","message":"My alternative is to just use greenthread.spawn_n if the group is not available. What do you think?","commit_id":"ccb09785ccbbbef4d2a8c6607a21815ee2ebe527"},{"author":{"_account_id":2472,"name":"Doug Hellmann","email":"dhellmann@redhat.com","username":"doug-hellmann"},"change_message_id":"dea25f0cfb1928f0f9948470686dab347b8d2023","unresolved":false,"context_lines":[{"line_number":116,"context_line":"Documentation Impact"},{"line_number":117,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":118,"context_line":""},{"line_number":119,"context_line":"Only docstrings update."},{"line_number":120,"context_line":""},{"line_number":121,"context_line":"Dependencies"},{"line_number":122,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":1,"id":"5a890539_71fa7044","line":119,"updated":"2014-11-13 20:16:59.000000000","message":"It would be good to have an example added to the documentation showing how to set up a parallel task.","commit_id":"ccb09785ccbbbef4d2a8c6607a21815ee2ebe527"},{"author":{"_account_id":10239,"name":"Dmitry Tantsur","email":"dtantsur@protonmail.com","username":"dtantsur"},"change_message_id":"209734f28a2d4aecd922ce6155915ff47e0504d0","unresolved":false,"context_lines":[{"line_number":116,"context_line":"Documentation Impact"},{"line_number":117,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":118,"context_line":""},{"line_number":119,"context_line":"Only docstrings update."},{"line_number":120,"context_line":""},{"line_number":121,"context_line":"Dependencies"},{"line_number":122,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":1,"id":"5a890539_5e0d6636","line":119,"in_reply_to":"5a890539_71fa7044","updated":"2014-11-14 13:03:27.000000000","message":"Done","commit_id":"ccb09785ccbbbef4d2a8c6607a21815ee2ebe527"},{"author":{"_account_id":2472,"name":"Doug Hellmann","email":"dhellmann@redhat.com","username":"doug-hellmann"},"change_message_id":"c19e0a276332e8736f7af30f3533ef376f68fb79","unresolved":false,"context_lines":[{"line_number":33,"context_line":""},{"line_number":34,"context_line":"* Add ``thread_group`` argument to the ``run_periodic_tasks`` method of"},{"line_number":35,"context_line":"  ``PeriodicTasks`` class with default value of None. If present, it should be"},{"line_number":36,"context_line":"  a ``ThreadGroup`` on which to run parallel tasks. If it is not set"},{"line_number":37,"context_line":"  ``greenthread.spawn_n`` will be used."},{"line_number":38,"context_line":""},{"line_number":39,"context_line":"* Modify ``run_periodic_tasks`` so that parallel tasks are executed on a"}],"source_content_type":"text/x-rst","patch_set":2,"id":"5a890539_333bdeec","line":36,"updated":"2014-11-19 21:29:12.000000000","message":"Restating my question from the earlier draft: I don\u0027t understand what a ThreadGroup is. Is it a collection of multiple threads? If we only have one function to call do we need more than one thread?","commit_id":"b0dad981c9b4b24f1741cf552453f77a1385e6a7"},{"author":{"_account_id":10239,"name":"Dmitry Tantsur","email":"dtantsur@protonmail.com","username":"dtantsur"},"change_message_id":"5c61cf4005b15426718c4f7f0df146c64acf10ac","unresolved":false,"context_lines":[{"line_number":33,"context_line":""},{"line_number":34,"context_line":"* Add ``thread_group`` argument to the ``run_periodic_tasks`` method of"},{"line_number":35,"context_line":"  ``PeriodicTasks`` class with default value of None. If present, it should be"},{"line_number":36,"context_line":"  a ``ThreadGroup`` on which to run parallel tasks. If it is not set"},{"line_number":37,"context_line":"  ``greenthread.spawn_n`` will be used."},{"line_number":38,"context_line":""},{"line_number":39,"context_line":"* Modify ``run_periodic_tasks`` so that parallel tasks are executed on a"}],"source_content_type":"text/x-rst","patch_set":2,"id":"5a890539_d94ab9ab","line":36,"in_reply_to":"5a890539_333bdeec","updated":"2014-11-19 22:42:43.000000000","message":"1. ThreadGroup is openstack.common.thread_group.ThreadGroup (essentially a thread pool), which I expect will get into the same oslo.service\n\n2. If you have only one parallel task, you can live without it and pass None. Ironic is likely to have multiple parallel tasks, so it makes sense to run them on a group. Just using spawn_n may lead to too high concurrency, though it\u0027s arguable.\n\nWhy it is called a group, not a pool - use git blame on oslo-incubator to figure out :)","commit_id":"b0dad981c9b4b24f1741cf552453f77a1385e6a7"},{"author":{"_account_id":2472,"name":"Doug Hellmann","email":"dhellmann@redhat.com","username":"doug-hellmann"},"change_message_id":"c19e0a276332e8736f7af30f3533ef376f68fb79","unresolved":false,"context_lines":[{"line_number":41,"context_line":""},{"line_number":42,"context_line":"* Ensure that a parallel task is never run again while the previous run"},{"line_number":43,"context_line":"  didn\u0027t finish. Add a new dictionary on ``PeriodicTasks`` class to track this."},{"line_number":44,"context_line":"  Issue a warning if this situation happens."},{"line_number":45,"context_line":""},{"line_number":46,"context_line":"  This may be the case if a parallel task suddenly takes more then it\u0027s period."},{"line_number":47,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"5a890539_4e0efd35","line":44,"updated":"2014-11-19 21:29:12.000000000","message":"Carrying over the discussion from the earlier patch: If the function in the thread is responsible for timing the background tasks, we will never have more than one execute in parallel. If we don\u0027t want them to run in parallel, we\u0027re already going to have the situation where a call that takes a long time will prevent the \"next\" execution from starting \"on time\" so I think we can ignore that concern.\n\nWe need to be explicit about the warning, too. Do we expect this to be something a deployer can deal with (by reconfiguring the period, or by providing more resources to the service)? If so, a LOG.warn() call may be appropriate. If not, we should use LOG.debug().","commit_id":"b0dad981c9b4b24f1741cf552453f77a1385e6a7"},{"author":{"_account_id":10239,"name":"Dmitry Tantsur","email":"dtantsur@protonmail.com","username":"dtantsur"},"change_message_id":"5c61cf4005b15426718c4f7f0df146c64acf10ac","unresolved":false,"context_lines":[{"line_number":41,"context_line":""},{"line_number":42,"context_line":"* Ensure that a parallel task is never run again while the previous run"},{"line_number":43,"context_line":"  didn\u0027t finish. Add a new dictionary on ``PeriodicTasks`` class to track this."},{"line_number":44,"context_line":"  Issue a warning if this situation happens."},{"line_number":45,"context_line":""},{"line_number":46,"context_line":"  This may be the case if a parallel task suddenly takes more then it\u0027s period."},{"line_number":47,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"5a890539_d97d59b3","line":44,"in_reply_to":"5a890539_4e0efd35","updated":"2014-11-19 22:42:43.000000000","message":"\u003e we will never have more than one execute in parallel\n\nFor one particular task - yes. Other tasks of course can run in parallel with it as usual, only this task is affected.\n\n\u003e Do we expect this to be something a deployer can deal with\n\nAt least [s]he has to be aware. In our case, it may be BMC taking too much time to respond - check the node? It may be too many BMC\u0027s handled by a conductor - add more conductors? Networking issue - debug it?\n\nSo I think it should be a warning.","commit_id":"b0dad981c9b4b24f1741cf552453f77a1385e6a7"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"aa65806989d4d2f70a99c1467bf0b81c3a485f82","unresolved":false,"context_lines":[{"line_number":43,"context_line":"  didn\u0027t finish. Add a new dictionary on ``PeriodicTasks`` class to track this."},{"line_number":44,"context_line":"  Issue a warning if this situation happens."},{"line_number":45,"context_line":""},{"line_number":46,"context_line":"  This may be the case if a parallel task suddenly takes more then it\u0027s period."},{"line_number":47,"context_line":""},{"line_number":48,"context_line":"Alternatives"},{"line_number":49,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":2,"id":"5a890539_1ae5bcdb","line":46,"updated":"2014-11-22 00:19:45.000000000","message":"I question that a dictionary is needed here.\n\nLet me propose an alternative implementation.\n\nLets say we have no ``thread_group`` but instead use a executor which returns one of these objects; https://docs.python.org/3/library/concurrent.futures.html#future-objects \n\nThen u can attach a callback to the method https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Future.add_done_callback that resubmits it for execution. That way u never have to worry about this kind of warning or debug or whatever output; the future when done will reschedule itself for the next run in the future when its done; seems more elegant (assuming it works).\n\nAlso seems more inline with things potentially coming down the pipes:\n\nhttp://bugs.python.org/issue17956  (\u0027add ScheduledExecutor\u0027)","commit_id":"b0dad981c9b4b24f1741cf552453f77a1385e6a7"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"de83220f720b770857dde66397d95fd02c6aed90","unresolved":false,"context_lines":[{"line_number":43,"context_line":"  didn\u0027t finish. Add a new dictionary on ``PeriodicTasks`` class to track this."},{"line_number":44,"context_line":"  Issue a warning if this situation happens."},{"line_number":45,"context_line":""},{"line_number":46,"context_line":"  This may be the case if a parallel task suddenly takes more then it\u0027s period."},{"line_number":47,"context_line":""},{"line_number":48,"context_line":"Alternatives"},{"line_number":49,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":2,"id":"5a890539_9af9ac50","line":46,"in_reply_to":"5a890539_1ae5bcdb","updated":"2014-11-22 00:44:32.000000000","message":"For example:\n\nSomething I whipped up that could be useful to show the concept.\n\nhttp://paste.ubuntu.com/9160239/","commit_id":"b0dad981c9b4b24f1741cf552453f77a1385e6a7"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"804188a9c4eda6996077776c9469da2c91a35f8e","unresolved":false,"context_lines":[{"line_number":43,"context_line":"  didn\u0027t finish. Add a new dictionary on ``PeriodicTasks`` class to track this."},{"line_number":44,"context_line":"  Issue a warning if this situation happens."},{"line_number":45,"context_line":""},{"line_number":46,"context_line":"  This may be the case if a parallel task suddenly takes more then it\u0027s period."},{"line_number":47,"context_line":""},{"line_number":48,"context_line":"Alternatives"},{"line_number":49,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":2,"id":"5a890539_462354be","line":46,"in_reply_to":"5a890539_7a49c95f","updated":"2014-11-27 18:38:38.000000000","message":"Possibly, maybe something more optimized could be done ;)\n\nI did just create the above example in like an hour, ha.\n\nMight be useful to see or look at whats done in:\n\nhttp://bugs.python.org/file30233/scheduled-3.diff\n\nThat could have some insight into how to make a good scheduled feature that doesn\u0027t suck so much...","commit_id":"b0dad981c9b4b24f1741cf552453f77a1385e6a7"},{"author":{"_account_id":10239,"name":"Dmitry Tantsur","email":"dtantsur@protonmail.com","username":"dtantsur"},"change_message_id":"5c6596a3146b727fec228a3e4fdf20f9ec629efd","unresolved":false,"context_lines":[{"line_number":43,"context_line":"  didn\u0027t finish. Add a new dictionary on ``PeriodicTasks`` class to track this."},{"line_number":44,"context_line":"  Issue a warning if this situation happens."},{"line_number":45,"context_line":""},{"line_number":46,"context_line":"  This may be the case if a parallel task suddenly takes more then it\u0027s period."},{"line_number":47,"context_line":""},{"line_number":48,"context_line":"Alternatives"},{"line_number":49,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":2,"id":"5a890539_7a49c95f","line":46,"in_reply_to":"5a890539_9af9ac50","updated":"2014-11-27 15:04:50.000000000","message":"one problem I see here is that you have a green thread with an infinite loop with small sleeps. it\u0027s more or less ok for native threads, but for green thread it may seriously affect performance of the whole service.","commit_id":"b0dad981c9b4b24f1741cf552453f77a1385e6a7"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"aa65806989d4d2f70a99c1467bf0b81c3a485f82","unresolved":false,"context_lines":[{"line_number":53,"context_line":""},{"line_number":54,"context_line":"* Finally, every project could just go into a thread for their tasks directly."},{"line_number":55,"context_line":"  Still, I think anyone using oslo.service could benifit from this feature."},{"line_number":56,"context_line":""},{"line_number":57,"context_line":"Impact on Existing APIs"},{"line_number":58,"context_line":"-----------------------"},{"line_number":59,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"5a890539_7ac88055","line":56,"updated":"2014-11-22 00:19:45.000000000","message":"I\u0027d be really nice to avoid adding more things that tie themselves to eventlet. I\u0027d almost rather see this be ``executor`` instead of ``thread_group`` and then use one of the standard https://docs.python.org/3/library/concurrent.futures.html#executor-objects (or its backport) instead of using yet another concept strongly tied to eventlet (aka threadgroups). What do u think about just doing that (if eventlet is monkey patching the threading module, then the ThreadPoolExecutor that is in standard python should just use greenthreads...)?","commit_id":"b0dad981c9b4b24f1741cf552453f77a1385e6a7"},{"author":{"_account_id":10239,"name":"Dmitry Tantsur","email":"dtantsur@protonmail.com","username":"dtantsur"},"change_message_id":"5c6596a3146b727fec228a3e4fdf20f9ec629efd","unresolved":false,"context_lines":[{"line_number":53,"context_line":""},{"line_number":54,"context_line":"* Finally, every project could just go into a thread for their tasks directly."},{"line_number":55,"context_line":"  Still, I think anyone using oslo.service could benifit from this feature."},{"line_number":56,"context_line":""},{"line_number":57,"context_line":"Impact on Existing APIs"},{"line_number":58,"context_line":"-----------------------"},{"line_number":59,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"5a890539_fa077916","line":56,"in_reply_to":"5a890539_7ac88055","updated":"2014-11-27 15:04:50.000000000","message":"\u003e or its backport\n\ncould you point me to this backport?","commit_id":"b0dad981c9b4b24f1741cf552453f77a1385e6a7"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"804188a9c4eda6996077776c9469da2c91a35f8e","unresolved":false,"context_lines":[{"line_number":53,"context_line":""},{"line_number":54,"context_line":"* Finally, every project could just go into a thread for their tasks directly."},{"line_number":55,"context_line":"  Still, I think anyone using oslo.service could benifit from this feature."},{"line_number":56,"context_line":""},{"line_number":57,"context_line":"Impact on Existing APIs"},{"line_number":58,"context_line":"-----------------------"},{"line_number":59,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"5a890539_a6531838","line":56,"in_reply_to":"5a890539_fa077916","updated":"2014-11-27 18:38:38.000000000","message":"https://pypi.python.org/pypi/futures","commit_id":"b0dad981c9b4b24f1741cf552453f77a1385e6a7"},{"author":{"_account_id":4190,"name":"lifeless","email":"robertc@robertcollins.net","username":"lifeless"},"change_message_id":"2d140dc2ee53ea5d985b87a05cf4423adc402d3d","unresolved":false,"context_lines":[{"line_number":1,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":2,"context_line":"Support for running periodic tasks in a separate thread"},{"line_number":3,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":4,"context_line":""},{"line_number":5,"context_line":"https://blueprints.launchpad.net/oslo-incubator/+spec/parallel-periodic-task"},{"line_number":6,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"fa32b979_bee9e480","line":3,"updated":"2015-06-19 02:37:59.000000000","message":"sadface nit: this needs to be a liberty spec now I believe.","commit_id":"8069bd6f3bdbe1433b9493bafebfc39e44aa2379"},{"author":{"_account_id":4190,"name":"lifeless","email":"robertc@robertcollins.net","username":"lifeless"},"change_message_id":"2d140dc2ee53ea5d985b87a05cf4423adc402d3d","unresolved":false,"context_lines":[{"line_number":4,"context_line":""},{"line_number":5,"context_line":"https://blueprints.launchpad.net/oslo-incubator/+spec/parallel-periodic-task"},{"line_number":6,"context_line":""},{"line_number":7,"context_line":"TODO: retarget to oslo.service once it exists"},{"line_number":8,"context_line":""},{"line_number":9,"context_line":"This blueprint enables developers to request running designated periodic tasks"},{"line_number":10,"context_line":"on a thread group in a separate thread."}],"source_content_type":"text/x-rst","patch_set":3,"id":"fa32b979_3e8dd4d2","line":7,"updated":"2015-06-19 02:37:59.000000000","message":"It now exists :)","commit_id":"8069bd6f3bdbe1433b9493bafebfc39e44aa2379"},{"author":{"_account_id":4190,"name":"lifeless","email":"robertc@robertcollins.net","username":"lifeless"},"change_message_id":"2d140dc2ee53ea5d985b87a05cf4423adc402d3d","unresolved":false,"context_lines":[{"line_number":28,"context_line":"[1]. Changes are proposed to the yet-to-be-created oslo.service project."},{"line_number":29,"context_line":""},{"line_number":30,"context_line":"In this spec we introduce notion of *futures* and *executor* that corresponds"},{"line_number":31,"context_line":"to appropriate notions of Python 3.2 library or PyPI futures_ project."},{"line_number":32,"context_line":""},{"line_number":33,"context_line":".. _futures: https://pypi.python.org/pypi/futures"},{"line_number":34,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"fa32b979_5e7a18be","line":31,"updated":"2015-06-19 02:37:59.000000000","message":"I think the executor is not needed here. Executors are intended to permit different implementations of concurrency, but we\u0027re currently a monoculture and AIUI the existing periodic job implementation assumes eventlet. So we should just use a thread pool here rather than making it pluggable - I worry that folk will end up with hard to support combinations if we give an implementation hook thats not really relevant to this particular change. IMBW though - just a thought.","commit_id":"8069bd6f3bdbe1433b9493bafebfc39e44aa2379"},{"author":{"_account_id":2472,"name":"Doug Hellmann","email":"dhellmann@redhat.com","username":"doug-hellmann"},"change_message_id":"2ca11bf77e99ac5f762eb46d00a29735da7ad764","unresolved":false,"context_lines":[{"line_number":30,"context_line":"In this spec we introduce notion of *futures* and *executor* that corresponds"},{"line_number":31,"context_line":"to appropriate notions of Python 3.2 library or PyPI futures_ project."},{"line_number":32,"context_line":""},{"line_number":33,"context_line":".. _futures: https://pypi.python.org/pypi/futures"},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"* Add ``parallel`` argument (boolean, defaults to False) to the"},{"line_number":36,"context_line":"  ``@periodic_task`` decorator. If set to True, the task will be run in"}],"source_content_type":"text/x-rst","patch_set":3,"id":"9a0c5dc1_6ce293d7","line":33,"updated":"2015-05-12 17:05:01.000000000","message":"There may be some overlap with the futurist lib described in https://review.openstack.org/179890 as well","commit_id":"8069bd6f3bdbe1433b9493bafebfc39e44aa2379"},{"author":{"_account_id":10239,"name":"Dmitry Tantsur","email":"dtantsur@protonmail.com","username":"dtantsur"},"change_message_id":"3eb08fdac4b1373927764733607804c497d462ce","unresolved":false,"context_lines":[{"line_number":30,"context_line":"In this spec we introduce notion of *futures* and *executor* that corresponds"},{"line_number":31,"context_line":"to appropriate notions of Python 3.2 library or PyPI futures_ project."},{"line_number":32,"context_line":""},{"line_number":33,"context_line":".. _futures: https://pypi.python.org/pypi/futures"},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"* Add ``parallel`` argument (boolean, defaults to False) to the"},{"line_number":36,"context_line":"  ``@periodic_task`` decorator. If set to True, the task will be run in"}],"source_content_type":"text/x-rst","patch_set":3,"id":"7a016987_24d243d3","line":33,"in_reply_to":"7a016987_0e83e805","updated":"2015-05-15 17:00:43.000000000","message":"futurist.periodic sounds good. do you have some specs process for futurist? or any other development convention? :)","commit_id":"8069bd6f3bdbe1433b9493bafebfc39e44aa2379"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"58181fe9dcfe9f8a2d33a9abbb07d11e63e13bf7","unresolved":false,"context_lines":[{"line_number":30,"context_line":"In this spec we introduce notion of *futures* and *executor* that corresponds"},{"line_number":31,"context_line":"to appropriate notions of Python 3.2 library or PyPI futures_ project."},{"line_number":32,"context_line":""},{"line_number":33,"context_line":".. _futures: https://pypi.python.org/pypi/futures"},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"* Add ``parallel`` argument (boolean, defaults to False) to the"},{"line_number":36,"context_line":"  ``@periodic_task`` decorator. If set to True, the task will be run in"}],"source_content_type":"text/x-rst","patch_set":3,"id":"7a016987_7cd21302","line":33,"in_reply_to":"7a016987_24d243d3","updated":"2015-05-15 18:24:20.000000000","message":"https://review.openstack.org/#/c/179890/\n\nhttps://review.openstack.org/#/c/182808/\n\nand a few more...","commit_id":"8069bd6f3bdbe1433b9493bafebfc39e44aa2379"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"0a1c94ca9857475ca03e56d889262ac2527738a3","unresolved":false,"context_lines":[{"line_number":30,"context_line":"In this spec we introduce notion of *futures* and *executor* that corresponds"},{"line_number":31,"context_line":"to appropriate notions of Python 3.2 library or PyPI futures_ project."},{"line_number":32,"context_line":""},{"line_number":33,"context_line":".. _futures: https://pypi.python.org/pypi/futures"},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"* Add ``parallel`` argument (boolean, defaults to False) to the"},{"line_number":36,"context_line":"  ``@periodic_task`` decorator. If set to True, the task will be run in"}],"source_content_type":"text/x-rst","patch_set":3,"id":"7a016987_0e83e805","line":33,"in_reply_to":"7a016987_ce162555","updated":"2015-05-15 16:10:28.000000000","message":"That\u0027d be nice, I can extract some of my prototype code and that could become part of futurist (or a submodule of futurist named something like \u0027futurist.periodic\u0027)","commit_id":"8069bd6f3bdbe1433b9493bafebfc39e44aa2379"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"eb87f20cfe65b3850d14e4d55638e16597b5ff0f","unresolved":false,"context_lines":[{"line_number":30,"context_line":"In this spec we introduce notion of *futures* and *executor* that corresponds"},{"line_number":31,"context_line":"to appropriate notions of Python 3.2 library or PyPI futures_ project."},{"line_number":32,"context_line":""},{"line_number":33,"context_line":".. _futures: https://pypi.python.org/pypi/futures"},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"* Add ``parallel`` argument (boolean, defaults to False) to the"},{"line_number":36,"context_line":"  ``@periodic_task`` decorator. If set to True, the task will be run in"}],"source_content_type":"text/x-rst","patch_set":3,"id":"9a0c5dc1_e5dd7171","line":33,"in_reply_to":"9a0c5dc1_6ce293d7","updated":"2015-05-12 20:02:07.000000000","message":"Possibly.","commit_id":"8069bd6f3bdbe1433b9493bafebfc39e44aa2379"},{"author":{"_account_id":10239,"name":"Dmitry Tantsur","email":"dtantsur@protonmail.com","username":"dtantsur"},"change_message_id":"1daf838c04d95e2cbc9a1aba276ad2806b5aa517","unresolved":false,"context_lines":[{"line_number":30,"context_line":"In this spec we introduce notion of *futures* and *executor* that corresponds"},{"line_number":31,"context_line":"to appropriate notions of Python 3.2 library or PyPI futures_ project."},{"line_number":32,"context_line":""},{"line_number":33,"context_line":".. _futures: https://pypi.python.org/pypi/futures"},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"* Add ``parallel`` argument (boolean, defaults to False) to the"},{"line_number":36,"context_line":"  ``@periodic_task`` decorator. If set to True, the task will be run in"}],"source_content_type":"text/x-rst","patch_set":3,"id":"7a016987_ce162555","line":33,"in_reply_to":"9a0c5dc1_e5dd7171","updated":"2015-05-14 13:41:10.000000000","message":"Do you think we should retarget the spec to futurist instead and deprecate periodic_tasks in oslo.service?","commit_id":"8069bd6f3bdbe1433b9493bafebfc39e44aa2379"},{"author":{"_account_id":2472,"name":"Doug Hellmann","email":"dhellmann@redhat.com","username":"doug-hellmann"},"change_message_id":"2ca11bf77e99ac5f762eb46d00a29735da7ad764","unresolved":false,"context_lines":[{"line_number":34,"context_line":""},{"line_number":35,"context_line":"* Add ``parallel`` argument (boolean, defaults to False) to the"},{"line_number":36,"context_line":"  ``@periodic_task`` decorator. If set to True, the task will be run in"},{"line_number":37,"context_line":"  a separate green thread."},{"line_number":38,"context_line":""},{"line_number":39,"context_line":"* Add ``executor`` argument to ``PeriodicTasks.run_periodic_tasks``."},{"line_number":40,"context_line":"  It should by an object conforming to `executor interface"}],"source_content_type":"text/x-rst","patch_set":3,"id":"9a0c5dc1_ec6a031a","line":37,"updated":"2015-05-12 17:05:01.000000000","message":"So there will be one \"main\" periodic thread running several tasks, and a separate thread for each task where the flag is set? What happens if all of the tasks for a project are marked as parallel, do we have an empty set of periodic tasks running no-ops in that main loop?","commit_id":"8069bd6f3bdbe1433b9493bafebfc39e44aa2379"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"eb87f20cfe65b3850d14e4d55638e16597b5ff0f","unresolved":false,"context_lines":[{"line_number":34,"context_line":""},{"line_number":35,"context_line":"* Add ``parallel`` argument (boolean, defaults to False) to the"},{"line_number":36,"context_line":"  ``@periodic_task`` decorator. If set to True, the task will be run in"},{"line_number":37,"context_line":"  a separate green thread."},{"line_number":38,"context_line":""},{"line_number":39,"context_line":"* Add ``executor`` argument to ``PeriodicTasks.run_periodic_tasks``."},{"line_number":40,"context_line":"  It should by an object conforming to `executor interface"}],"source_content_type":"text/x-rst","patch_set":3,"id":"9a0c5dc1_0581a54a","line":37,"in_reply_to":"9a0c5dc1_ec6a031a","updated":"2015-05-12 20:02:07.000000000","message":"One way to do this is done/prototyped at:\n\nhttps://review.openstack.org/#/c/158092/7/taskflow/types/periodic.py\n\nA schedule needs to be maintained (that code uses a heapq for that), and that schedule needs to be modified when a parallel action occurs/triggers, then when that parallel action is done, it re-inserts itself.\n\nYou should be able to download/checkout that review and mess around with it relatively easily:\n\nHere\u0027s something to try/mess around with...\n\nhttp://paste.openstack.org/show/221119/\n\nIt runs things in parallel (depending on how many workers you let it use...), with output like the following: http://paste.openstack.org/show/221120/","commit_id":"8069bd6f3bdbe1433b9493bafebfc39e44aa2379"},{"author":{"_account_id":2472,"name":"Doug Hellmann","email":"dhellmann@redhat.com","username":"doug-hellmann"},"change_message_id":"2ca11bf77e99ac5f762eb46d00a29735da7ad764","unresolved":false,"context_lines":[{"line_number":40,"context_line":"  It should by an object conforming to `executor interface"},{"line_number":41,"context_line":"  \u003chttps://docs.python.org/3/library/concurrent.futures.html#executor-objects\u003e`_."},{"line_number":42,"context_line":"  An exception is raised if ``executor`` is None and parallel periodic tasks"},{"line_number":43,"context_line":"  are present."},{"line_number":44,"context_line":""},{"line_number":45,"context_line":"* The first time ``run_periodic_tasks`` encounteres a parallel periodic task,"},{"line_number":46,"context_line":"  it schedules the task for running on executor and attaches a callback"}],"source_content_type":"text/x-rst","patch_set":3,"id":"9a0c5dc1_ecffa332","line":43,"updated":"2015-05-12 17:05:01.000000000","message":"Are there example implementations? Could we just provide a valid default instead of forcing developers to do that?","commit_id":"8069bd6f3bdbe1433b9493bafebfc39e44aa2379"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"e0ead6d6681a9634977abd108041a42b48ce01e2","unresolved":false,"context_lines":[{"line_number":40,"context_line":"  It should by an object conforming to `executor interface"},{"line_number":41,"context_line":"  \u003chttps://docs.python.org/3/library/concurrent.futures.html#executor-objects\u003e`_."},{"line_number":42,"context_line":"  An exception is raised if ``executor`` is None and parallel periodic tasks"},{"line_number":43,"context_line":"  are present."},{"line_number":44,"context_line":""},{"line_number":45,"context_line":"* The first time ``run_periodic_tasks`` encounteres a parallel periodic task,"},{"line_number":46,"context_line":"  it schedules the task for running on executor and attaches a callback"}],"source_content_type":"text/x-rst","patch_set":3,"id":"9a0c5dc1_8500f5ab","line":43,"in_reply_to":"9a0c5dc1_ecffa332","updated":"2015-05-12 20:03:48.000000000","message":"Possibly, see above.","commit_id":"8069bd6f3bdbe1433b9493bafebfc39e44aa2379"},{"author":{"_account_id":4190,"name":"lifeless","email":"robertc@robertcollins.net","username":"lifeless"},"change_message_id":"2d140dc2ee53ea5d985b87a05cf4423adc402d3d","unresolved":false,"context_lines":[{"line_number":54,"context_line":"  would be a bit more code for Ironic, but otherwise looks just as well."},{"line_number":55,"context_line":""},{"line_number":56,"context_line":"* Finally, every project could just go into a thread for their tasks directly."},{"line_number":57,"context_line":"  Still, I think anyone using oslo.service could benifit from this feature."},{"line_number":58,"context_line":""},{"line_number":59,"context_line":"Impact on Existing APIs"},{"line_number":60,"context_line":"-----------------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"fa32b979_beb7a483","line":57,"updated":"2015-06-19 02:37:59.000000000","message":"Typo: benefit","commit_id":"8069bd6f3bdbe1433b9493bafebfc39e44aa2379"}]}