)]}'
{"taskflow/jobs/backends/impl_zookeeper.py":[{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"2fe2ce5069e2d2ea93d123590f3f8cbbc24c2646","unresolved":false,"context_lines":[{"line_number":574,"context_line":"            raise excp.JobFailure(\"Failed to do post-connection\""},{"line_number":575,"context_line":"                                  \" initialization\", e)"},{"line_number":576,"context_line":""},{"line_number":577,"context_line":"    def clear(self):"},{"line_number":578,"context_line":"        self._clear()"}],"source_content_type":"text/x-python","patch_set":1,"id":"9ad9bd40_ef1d0d75","line":577,"updated":"2014-05-01 02:07:36.000000000","message":"So a question here, do u want to actually go to zookeeper and remove the jobs in the zookeeper storage? Clearing the local state will clear it from this object, but not from that backend storage (and if someone does an iterjobs with ensure_fresh then this data will be readded). I guess this becomes a question of what do u want clear to be :-)","commit_id":"2f56211269679d7be8928cfabb948c2d1ffb95f7"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"95d9f68d1da695879de125293c6092844aa3a30a","unresolved":false,"context_lines":[{"line_number":574,"context_line":"            raise excp.JobFailure(\"Failed to do post-connection\""},{"line_number":575,"context_line":"                                  \" initialization\", e)"},{"line_number":576,"context_line":""},{"line_number":577,"context_line":"    def clear(self):"},{"line_number":578,"context_line":"        self._clear()"}],"source_content_type":"text/x-python","patch_set":1,"id":"9ad9bd40_b6308202","line":577,"in_reply_to":"9ad9bd40_15776d3b","updated":"2014-05-04 02:12:51.000000000","message":"I also just found https://issues.apache.org/jira/browse/ZOOKEEPER-1665 \u0026 https://github.com/python-zk/kazoo/issues/195 which if implemented would I think make for a clear that would functionally work correctly without race conditions.","commit_id":"2f56211269679d7be8928cfabb948c2d1ffb95f7"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"b75de4b626225520d9a94442e1f3e02adbb086ab","unresolved":false,"context_lines":[{"line_number":574,"context_line":"            raise excp.JobFailure(\"Failed to do post-connection\""},{"line_number":575,"context_line":"                                  \" initialization\", e)"},{"line_number":576,"context_line":""},{"line_number":577,"context_line":"    def clear(self):"},{"line_number":578,"context_line":"        self._clear()"}],"source_content_type":"text/x-python","patch_set":1,"id":"9ad9bd40_15776d3b","line":577,"in_reply_to":"9ad9bd40_3551d1d8","updated":"2014-05-02 21:55:41.000000000","message":"Yes, u would want something like a mutex, but being that jobboard users can be distributed across machines it will be non-trival to have such a mutex. U are right that it would not r/w lock access, but then a question is do u really want to r/w lock access, or is it fine not to? If u really want to r/w lock access, then any entity posting on the jobboard would need to acquire such a lock before doing work (which means this lock becomes a limiting factor).\n\nKazoo does have recipes[1] we can use to help here, but we just have to be careful with what enabling such recipes does (creating a single lock file would mean a global mutex for posting entities).\n\n1. https://github.com/python-zk/kazoo/tree/master/kazoo/recipe\n\nCan we maybe figure out why a clear() is needed and determine if it really is, and maybe for that use-case it is quite possible to do not require a lock (if clear is done when there is no possibility of others working on it \u0027ever again\u0027, then u don\u0027t need a lock)","commit_id":"2f56211269679d7be8928cfabb948c2d1ffb95f7"},{"author":{"_account_id":8122,"name":"Cyril Roelandt","email":"cyril@redhat.com","username":"cyril.roelandt.enovance"},"change_message_id":"9b43b85bc779ef9ec91d86ec2c2edf910dcbab59","unresolved":false,"context_lines":[{"line_number":574,"context_line":"            raise excp.JobFailure(\"Failed to do post-connection\""},{"line_number":575,"context_line":"                                  \" initialization\", e)"},{"line_number":576,"context_line":""},{"line_number":577,"context_line":"    def clear(self):"},{"line_number":578,"context_line":"        self._clear()"}],"source_content_type":"text/x-python","patch_set":1,"id":"9ad9bd40_3551d1d8","line":577,"in_reply_to":"9ad9bd40_529cabc0","updated":"2014-05-02 21:45:10.000000000","message":"Being a C programmer, what I really feel like doing is:\n\npthread_mutex_lock(...);\n/* remove all jobs */\npthread_mutex_unlock(...);\n\nI don\u0027t really know the internals of zookeeper/kazoo, so I think I might have a hard time implementing this :) In the meantime, do you think there is a way for me to do that in code that uses taskflow? I can use claim/consume to \"remove\" a job, but this would not lock R/W access to the board.","commit_id":"2f56211269679d7be8928cfabb948c2d1ffb95f7"},{"author":{"_account_id":8122,"name":"Cyril Roelandt","email":"cyril@redhat.com","username":"cyril.roelandt.enovance"},"change_message_id":"b1a3c0e67e0ebf5bc5711435d7be05439f94c282","unresolved":false,"context_lines":[{"line_number":574,"context_line":"            raise excp.JobFailure(\"Failed to do post-connection\""},{"line_number":575,"context_line":"                                  \" initialization\", e)"},{"line_number":576,"context_line":""},{"line_number":577,"context_line":"    def clear(self):"},{"line_number":578,"context_line":"        self._clear()"}],"source_content_type":"text/x-python","patch_set":1,"id":"9ad9bd40_feed8bb8","line":577,"in_reply_to":"9ad9bd40_ef1d0d75","updated":"2014-05-02 20:36:18.000000000","message":"Oh, I really want to erase everything, so obviously this patch does not do what I need. \n\nWhat would be a good way of doing that ? I could just do \"self.claim(); self.consume()\" but that does not look so good. I guess something like:\n\nfor job in self.iterjobs():\n    self._remove_job(job.path)\n\nWould that be enough ?","commit_id":"2f56211269679d7be8928cfabb948c2d1ffb95f7"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"7fe76c2dc596b316f9128cea0c8b43751fde2abe","unresolved":false,"context_lines":[{"line_number":574,"context_line":"            raise excp.JobFailure(\"Failed to do post-connection\""},{"line_number":575,"context_line":"                                  \" initialization\", e)"},{"line_number":576,"context_line":""},{"line_number":577,"context_line":"    def clear(self):"},{"line_number":578,"context_line":"        self._clear()"}],"source_content_type":"text/x-python","patch_set":1,"id":"9ad9bd40_529cabc0","line":577,"in_reply_to":"9ad9bd40_feed8bb8","updated":"2014-05-02 20:57:54.000000000","message":"So there is going to be an interesting question here. Doing what u suggested here will cause a single user of the library to remove all jobs, this will clear the zookeeper backend for the jobs that 1 user is aware of. The problem becomes that what is stopping some other user of a jobboard from at the same time posting a job that this user doing the clear() will not be aware of. This would end up with jobs that are not cleared.\n\nIn this case, if u really want to clear it and also stop other entities from putting jobs on at the same time (which I think is what u want). This will mean using something like https://github.com/python-zk/kazoo/blob/master/kazoo/client.py#L1208 (but the question is what happens if another worker is posting a job while that delete + recursive is occurring, does the delete fail?). Would that mean this delete would need to be retried (and how many retries would be needed?)\n\nThere might be some kind of path stored in zookeeper that is used (and checked by others) to see if a clearing is occurring (a \".clear\" file in the jobboards directory for example) and on posting this file \u0027lock\u0027 would be acquired (of course this decreases scalability since now this is a new lock point).\n\nSo I guess maybe the question is whats the intended goal :)","commit_id":"2f56211269679d7be8928cfabb948c2d1ffb95f7"}]}