)]}'
{"doc/source/specs/keep_db_consistency.rst":[{"author":{"_account_id":13070,"name":"Eran Gampel","email":"eran@gampel.net","username":"Gampel"},"change_message_id":"c979fdf88c30515b37d7eab9e3be1d5d5848bbf3","unresolved":false,"context_lines":[{"line_number":17,"context_line":"network topology of OpenStack clouds. It is a reliable relational data store"},{"line_number":18,"context_line":"and can be considered as the master DB in Dragonflow distributed architecture."},{"line_number":19,"context_line":"As a result, the question becomes how to synchronize the slave DB (Dragonflow"},{"line_number":20,"context_line":"distributed NoSQL data store) to the master DB (Neutron data store). In this"},{"line_number":21,"context_line":"design, we will first propose several potential solutions and discuss the"},{"line_number":22,"context_line":"pros and cons. Finally, we will conclude which way to go."},{"line_number":23,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"da6ed579_7d2b5195","line":20,"range":{"start_line":20,"start_character":30,"end_line":20,"end_character":32},"updated":"2016-01-16 07:51:56.000000000","message":"from","commit_id":"9ef5ef28184b3611952476eb74c97d2d2ba1a60a"},{"author":{"_account_id":13070,"name":"Eran Gampel","email":"eran@gampel.net","username":"Gampel"},"change_message_id":"c979fdf88c30515b37d7eab9e3be1d5d5848bbf3","unresolved":false,"context_lines":[{"line_number":22,"context_line":"pros and cons. Finally, we will conclude which way to go."},{"line_number":23,"context_line":""},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"Problem Description"},{"line_number":26,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":27,"context_line":""},{"line_number":28,"context_line":"Currently, Dragonflow architecture has two type of data stores. One is Neutron"}],"source_content_type":"text/x-rst","patch_set":2,"id":"da6ed579_0072eaad","line":25,"range":{"start_line":25,"start_character":0,"end_line":25,"end_character":7},"updated":"2016-01-16 07:51:56.000000000","message":"I think that we should add the fact that currently we write to the DF DB under the SQL DB lock is problematic for performance  and can cause the DB to hang in case of timeout on connecting to the DF DB","commit_id":"9ef5ef28184b3611952476eb74c97d2d2ba1a60a"},{"author":{"_account_id":13070,"name":"Eran Gampel","email":"eran@gampel.net","username":"Gampel"},"change_message_id":"c979fdf88c30515b37d7eab9e3be1d5d5848bbf3","unresolved":false,"context_lines":[{"line_number":68,"context_line":""},{"line_number":69,"context_line":"To solve the problems discussed above, the general idea is to introduce a"},{"line_number":70,"context_line":"full-sync mechanism for Neutron plugin layer and introduce a virtual"},{"line_number":71,"context_line":"transaction mechanism for DB-API layer."},{"line_number":72,"context_line":""},{"line_number":73,"context_line":"Full-Sync Mechanism"},{"line_number":74,"context_line":"-------------------"}],"source_content_type":"text/x-rst","patch_set":2,"id":"da6ed579_40d46246","line":71,"range":{"start_line":71,"start_character":33,"end_line":71,"end_character":38},"updated":"2016-01-16 07:51:56.000000000","message":"I think that we shloul say that this  solution  is eventually consistent","commit_id":"9ef5ef28184b3611952476eb74c97d2d2ba1a60a"},{"author":{"_account_id":13070,"name":"Eran Gampel","email":"eran@gampel.net","username":"Gampel"},"change_message_id":"c979fdf88c30515b37d7eab9e3be1d5d5848bbf3","unresolved":false,"context_lines":[{"line_number":67,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":68,"context_line":""},{"line_number":69,"context_line":"To solve the problems discussed above, the general idea is to introduce a"},{"line_number":70,"context_line":"full-sync mechanism for Neutron plugin layer and introduce a virtual"},{"line_number":71,"context_line":"transaction mechanism for DB-API layer."},{"line_number":72,"context_line":""},{"line_number":73,"context_line":"Full-Sync Mechanism"},{"line_number":74,"context_line":"-------------------"}],"source_content_type":"text/x-rst","patch_set":2,"id":"da6ed579_20e84e16","line":71,"range":{"start_line":70,"start_character":60,"end_line":71,"end_character":11},"updated":"2016-01-16 07:51:56.000000000","message":"I think that we should add few words about the virtual transaction requirements to decouple the Neutron DB transaction from the DF DB potentially using role forward mechanizem  and keeping the Neutron SQL DB as the master.","commit_id":"9ef5ef28184b3611952476eb74c97d2d2ba1a60a"},{"author":{"_account_id":18668,"name":"ding bo","email":"longddropt@163.com","username":"longddropt"},"change_message_id":"5eee04d7932efa747923826a7f577aa40103a9e3","unresolved":false,"context_lines":[{"line_number":67,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":68,"context_line":""},{"line_number":69,"context_line":"To solve the problems discussed above, the general idea is to introduce a"},{"line_number":70,"context_line":"full-sync mechanism for Neutron plugin layer and introduce a virtual"},{"line_number":71,"context_line":"transaction mechanism for DB-API layer."},{"line_number":72,"context_line":""},{"line_number":73,"context_line":"Full-Sync Mechanism"},{"line_number":74,"context_line":"-------------------"}],"source_content_type":"text/x-rst","patch_set":2,"id":"ba63e191_41ff8301","line":71,"range":{"start_line":70,"start_character":60,"end_line":71,"end_character":11},"in_reply_to":"da6ed579_20e84e16","updated":"2016-01-20 01:59:10.000000000","message":"What does \"role forward mechanizem \" mean?","commit_id":"9ef5ef28184b3611952476eb74c97d2d2ba1a60a"},{"author":{"_account_id":12257,"name":"Ayal Baron","email":"ayal.baron@huawei.com","username":"AyalBaron"},"change_message_id":"412065d7b79134376198fb5962d27fd89f75d277","unresolved":false,"context_lines":[{"line_number":87,"context_line":"To achieve this, the DB-API layer should provide a global lock mechanism."},{"line_number":88,"context_line":"The pseudo-code is as follows:"},{"line_number":89,"context_line":""},{"line_number":90,"context_line":"    If df_plugin.db.driver.locked(‘global_key’, ’key’):"},{"line_number":91,"context_line":"        Run compare-db to identify the difference between two DBs."},{"line_number":92,"context_line":"        Run update-df-db to make it be synchronized."},{"line_number":93,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"da6ed579_805495ca","line":90,"updated":"2016-01-16 14:57:38.000000000","message":"1. What happens if the host that took the lock crashed?\n2. If there is an on-going sync then why not just move on? (what\u0027s the point in multiple neutron servers running the sync sequentially?)","commit_id":"9ef5ef28184b3611952476eb74c97d2d2ba1a60a"},{"author":{"_account_id":12257,"name":"Ayal Baron","email":"ayal.baron@huawei.com","username":"AyalBaron"},"change_message_id":"412065d7b79134376198fb5962d27fd89f75d277","unresolved":false,"context_lines":[{"line_number":103,"context_line":"        return True"},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"In addition, the DB backend driver should provide a compare-and-swap function"},{"line_number":106,"context_line":"to update the global lock."},{"line_number":107,"context_line":""},{"line_number":108,"context_line":"Virtual Transaction Mechanism for DB-API"},{"line_number":109,"context_line":"----------------------------------------"}],"source_content_type":"text/x-rst","patch_set":2,"id":"da6ed579_c0f84dc7","line":106,"updated":"2016-01-16 14:57:38.000000000","message":"My understanding from our discussions in the meetup were that the consensus is to take a lock in the Neutron (master) db that all the neutron servers are synchronized on and not on the distributed db (that may or may not support lock/compare and swap)","commit_id":"9ef5ef28184b3611952476eb74c97d2d2ba1a60a"},{"author":{"_account_id":13070,"name":"Eran Gampel","email":"eran@gampel.net","username":"Gampel"},"change_message_id":"c979fdf88c30515b37d7eab9e3be1d5d5848bbf3","unresolved":false,"context_lines":[{"line_number":122,"context_line":"        with df_context.virtual_transaction of Dragonflow DB:"},{"line_number":123,"context_line":"            with context.transaction of Neutron DB:"},{"line_number":124,"context_line":"                call Neutron DB API"},{"line_number":125,"context_line":"    \t\tcall Dragonflow DB API"},{"line_number":126,"context_line":""},{"line_number":127,"context_line":"Pros:"},{"line_number":128,"context_line":"It simplifies the NB-API layer. The usage of transaction of DB-API is the same"}],"source_content_type":"text/x-rst","patch_set":2,"id":"da6ed579_8051fada","line":125,"range":{"start_line":125,"start_character":0,"end_line":125,"end_character":6},"updated":"2016-01-16 07:51:56.000000000","message":"Please remove","commit_id":"9ef5ef28184b3611952476eb74c97d2d2ba1a60a"},{"author":{"_account_id":12257,"name":"Ayal Baron","email":"ayal.baron@huawei.com","username":"AyalBaron"},"change_message_id":"412065d7b79134376198fb5962d27fd89f75d277","unresolved":false,"context_lines":[{"line_number":169,"context_line":"DB backends and simplify the architecture."},{"line_number":170,"context_line":""},{"line_number":171,"context_line":"In DB-API:"},{"line_number":172,"context_line":"    def obtain_lock(self, session_id, table, key):"},{"line_number":173,"context_line":"        int total_wait_time \u003d 0"},{"line_number":174,"context_line":"        while(self.compare_and_swap(table, key, new_val\u003dsession_id, old_val\u003d0)):"},{"line_number":175,"context_line":"            time.sleep(duration)"}],"source_content_type":"text/x-rst","patch_set":2,"id":"da6ed579_8019f524","line":172,"updated":"2016-01-16 14:57:38.000000000","message":"this should probably be a lease in case of host crash (and then we need to start thinking about fencing mechanisms in case of lease timeout)","commit_id":"9ef5ef28184b3611952476eb74c97d2d2ba1a60a"},{"author":{"_account_id":12257,"name":"Ayal Baron","email":"ayal.baron@huawei.com","username":"AyalBaron"},"change_message_id":"412065d7b79134376198fb5962d27fd89f75d277","unresolved":false,"context_lines":[{"line_number":204,"context_line":"    with df_context.virtual_transaction.precommit of Dragonflow DB:"},{"line_number":205,"context_line":"        with context.transaction of Neutron DB:"},{"line_number":206,"context_line":"            call Neutron DB API"},{"line_number":207,"context_line":"            call Dragonflow DB API"},{"line_number":208,"context_line":""},{"line_number":209,"context_line":"    with df_context.virtual_transaction.postcommit of Dragonflow DB:"},{"line_number":210,"context_line":"        pass"}],"source_content_type":"text/x-rst","patch_set":2,"id":"da6ed579_40b9bdeb","line":207,"updated":"2016-01-16 14:57:38.000000000","message":"the whole point of defining neutron DB as the master and creating the queue is to avoid putting the dragonflow db api calls inside the neutron DB transactions.  This might kill the Neutron DB performance.","commit_id":"9ef5ef28184b3611952476eb74c97d2d2ba1a60a"},{"author":{"_account_id":13070,"name":"Eran Gampel","email":"eran@gampel.net","username":"Gampel"},"change_message_id":"c979fdf88c30515b37d7eab9e3be1d5d5848bbf3","unresolved":false,"context_lines":[{"line_number":237,"context_line":"Neutron DB or the existing Dragonflow DB backends. No matter what we use as the"},{"line_number":238,"context_line":"ActionQueue, the backend should provide a general transaction mechanism."},{"line_number":239,"context_line":""},{"line_number":240,"context_line":"Pros: "},{"line_number":241,"context_line":"The implementation is easy to achieve and understand."},{"line_number":242,"context_line":""},{"line_number":243,"context_line":"Cons:"}],"source_content_type":"text/x-rst","patch_set":2,"id":"da6ed579_a0c3de89","line":240,"range":{"start_line":240,"start_character":4,"end_line":240,"end_character":5},"updated":"2016-01-16 07:51:56.000000000","message":"Please remove trailing white space","commit_id":"9ef5ef28184b3611952476eb74c97d2d2ba1a60a"},{"author":{"_account_id":20229,"name":"Omer Anson","email":"omer.anson@toganetworks.com","username":"oanson"},"change_message_id":"db6886fca33f9d87eea642a37797707f6a1afde2","unresolved":false,"context_lines":[{"line_number":46,"context_line":"transaction is committed. We have to set up a rollback mechanism to make sure"},{"line_number":47,"context_line":"the two DB is tied up. This phenomenon is discovered when Dragonflow is"},{"line_number":48,"context_line":"disconnected after Neutron DB transaction is committed."},{"line_number":49,"context_line":""},{"line_number":50,"context_line":"* Scenario 2, in Bug [2]:"},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"    with transaction of Neutron DB:"}],"source_content_type":"text/x-rst","patch_set":8,"id":"7a5de9d1_7c040f17","line":49,"updated":"2016-01-31 14:39:32.000000000","message":"The Neutron DB is master - errors in the DragonFlow DB shouldn\u0027t affect the data therein. That is something the user set up using Neutron API, which is legal (otherwise Neutron would have rejected it). We cannot undo it on a whim.\n\nI think a more correct wording is that the Dragonflow has to be re-synced with the Neutron DB.","commit_id":"edb25323a1ab7cb331b052bbca70099a9d4424f0"},{"author":{"_account_id":7805,"name":"Li Ma","email":"skywalker.nick@gmail.com","username":"Nick"},"change_message_id":"fcc0d5e469370e1332537579e3fcaa6bdc1aee8c","unresolved":false,"context_lines":[{"line_number":46,"context_line":"transaction is committed. We have to set up a rollback mechanism to make sure"},{"line_number":47,"context_line":"the two DB is tied up. This phenomenon is discovered when Dragonflow is"},{"line_number":48,"context_line":"disconnected after Neutron DB transaction is committed."},{"line_number":49,"context_line":""},{"line_number":50,"context_line":"* Scenario 2, in Bug [2]:"},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"    with transaction of Neutron DB:"}],"source_content_type":"text/x-rst","patch_set":8,"id":"7a5de9d1_178614db","line":49,"in_reply_to":"7a5de9d1_7c040f17","updated":"2016-01-31 15:31:58.000000000","message":"+1.","commit_id":"edb25323a1ab7cb331b052bbca70099a9d4424f0"},{"author":{"_account_id":20229,"name":"Omer Anson","email":"omer.anson@toganetworks.com","username":"oanson"},"change_message_id":"db6886fca33f9d87eea642a37797707f6a1afde2","unresolved":false,"context_lines":[{"line_number":83,"context_line":"corresponding network table \u0027lswitch\u0027, which increases the frequency"},{"line_number":84,"context_line":"of concurrent read/write operations on the same key. In order to reduce"},{"line_number":85,"context_line":"the possibility of race condition, we need to refactor the db schema."},{"line_number":86,"context_line":""},{"line_number":87,"context_line":"Full-Sync Mechanism"},{"line_number":88,"context_line":"-------------------"},{"line_number":89,"context_line":""}],"source_content_type":"text/x-rst","patch_set":8,"id":"7a5de9d1_374f906c","line":86,"updated":"2016-01-31 14:39:32.000000000","message":"Perhaps also add a post-update verification stage, that verifies that the information in the distributed NoDB (DF DB) is the same as the NeutronDB?","commit_id":"edb25323a1ab7cb331b052bbca70099a9d4424f0"},{"author":{"_account_id":7805,"name":"Li Ma","email":"skywalker.nick@gmail.com","username":"Nick"},"change_message_id":"fcc0d5e469370e1332537579e3fcaa6bdc1aee8c","unresolved":false,"context_lines":[{"line_number":83,"context_line":"corresponding network table \u0027lswitch\u0027, which increases the frequency"},{"line_number":84,"context_line":"of concurrent read/write operations on the same key. In order to reduce"},{"line_number":85,"context_line":"the possibility of race condition, we need to refactor the db schema."},{"line_number":86,"context_line":""},{"line_number":87,"context_line":"Full-Sync Mechanism"},{"line_number":88,"context_line":"-------------------"},{"line_number":89,"context_line":""}],"source_content_type":"text/x-rst","patch_set":8,"id":"7a5de9d1_57800cc0","line":86,"in_reply_to":"7a5de9d1_374f906c","updated":"2016-01-31 15:31:58.000000000","message":"The compare-db function is for verification. We don\u0027t need to set a post-update stage. In the spec, we run compare-db periodically. If we verify that the two DB have the same data, we don\u0027t need to sync data.","commit_id":"edb25323a1ab7cb331b052bbca70099a9d4424f0"},{"author":{"_account_id":18668,"name":"ding bo","email":"longddropt@163.com","username":"longddropt"},"change_message_id":"6a28e09fb6a5578b7e481b6b1bd9c45ceba7546f","unresolved":false,"context_lines":[{"line_number":104,"context_line":"--------------------------"},{"line_number":105,"context_line":""},{"line_number":106,"context_line":"To solve these problems, a SQL-based distributed lock mechanism is introduced."},{"line_number":107,"context_line":"A carefully-designed SQL transaction is capable of being an external atomic"},{"line_number":108,"context_line":"lock to protect all the dependent database operations both for Neutron DB and"},{"line_number":109,"context_line":"DF distributed NoSQL server in a given API context. This can greatly reduces"},{"line_number":110,"context_line":"the complexity of introducing other sophisticated sychoronization mechanism"},{"line_number":111,"context_line":"from scratch."}],"source_content_type":"text/x-rst","patch_set":8,"id":"5a5ae5dd_3cd5d742","line":108,"range":{"start_line":107,"start_character":0,"end_line":108,"end_character":5},"updated":"2016-02-03 09:24:18.000000000","message":"Sorry, I still cannot understand this logic. Transaction itself cannot be lock. We can have two process executing in two transactions in which manipulating the same data. Unless we set the transaction isolation level to \"serials\", other isolation levels such as \"read uncommitted\", \"read committed\" and \"repeatable read\" cannot prevent concurrency. And what is the meaning of  \"Release the distributed lock in DF DB\"(line 313)?  I did not see where we lock up the lock in DF DB. Please let me know if I have something misunderstood.","commit_id":"edb25323a1ab7cb331b052bbca70099a9d4424f0"},{"author":{"_account_id":20229,"name":"Omer Anson","email":"omer.anson@toganetworks.com","username":"oanson"},"change_message_id":"db6886fca33f9d87eea642a37797707f6a1afde2","unresolved":false,"context_lines":[{"line_number":131,"context_line":"            Release the distributed lock in DF DB."},{"line_number":132,"context_line":"        Else"},{"line_number":133,"context_line":"            Resync database."},{"line_number":134,"context_line":""},{"line_number":135,"context_line":"* When creating the distributed lock, it starts a DB transaction and updates"},{"line_number":136,"context_line":"Neutron DB and DF DB in the same transaction to prevent from inconsistency. As"},{"line_number":137,"context_line":"a result, the DB operations are both conducted or failed."}],"source_content_type":"text/x-rst","patch_set":8,"id":"7a5de9d1_37d5501b","line":134,"updated":"2016-01-31 14:39:32.000000000","message":"In this algorithm the DF DB indeed doesn\u0027t force rollback of the Neutron API.\n\nNote that when the NeutronDB transaction is rolled back, the DF DB updates don\u0027t need to take place. This is not evident from the pseudo-code above.","commit_id":"edb25323a1ab7cb331b052bbca70099a9d4424f0"},{"author":{"_account_id":18668,"name":"ding bo","email":"longddropt@163.com","username":"longddropt"},"change_message_id":"7f9a0f2895db1e85648d4cb6acac641655f7656b","unresolved":false,"context_lines":[{"line_number":132,"context_line":"        Else"},{"line_number":133,"context_line":"            Resync database."},{"line_number":134,"context_line":""},{"line_number":135,"context_line":"* When creating the distributed lock, it starts a DB transaction and updates"},{"line_number":136,"context_line":"Neutron DB and DF DB in the same transaction to prevent from inconsistency. As"},{"line_number":137,"context_line":"a result, the DB operations are both conducted or failed."},{"line_number":138,"context_line":""},{"line_number":139,"context_line":"* If some exception happens during Neutron DB operations, the Neutron DB will"},{"line_number":140,"context_line":"be rolled back."}],"source_content_type":"text/x-rst","patch_set":8,"id":"7a5de9d1_a4cef72a","line":137,"range":{"start_line":135,"start_character":0,"end_line":137,"end_character":56},"updated":"2016-01-30 13:06:49.000000000","message":"Since the neutron DB updating and DF DB updating are in the same transaction. We can only use lock in neutron DB. The code looks something like this:\nstart transaction:\n    select * from  Lock_Table where lock_id\u003dapi_id for update\n    do neutron DB update\n    do df DB update\nSince \"select for update\" can prevent concurrency from other processes or threads, So my question is: why do we want the lock-id and the lock in DF DB?","commit_id":"edb25323a1ab7cb331b052bbca70099a9d4424f0"},{"author":{"_account_id":7805,"name":"Li Ma","email":"skywalker.nick@gmail.com","username":"Nick"},"change_message_id":"7d33d135ec19c4b26db6883280abbde34a45c01f","unresolved":false,"context_lines":[{"line_number":132,"context_line":"        Else"},{"line_number":133,"context_line":"            Resync database."},{"line_number":134,"context_line":""},{"line_number":135,"context_line":"* When creating the distributed lock, it starts a DB transaction and updates"},{"line_number":136,"context_line":"Neutron DB and DF DB in the same transaction to prevent from inconsistency. As"},{"line_number":137,"context_line":"a result, the DB operations are both conducted or failed."},{"line_number":138,"context_line":""},{"line_number":139,"context_line":"* If some exception happens during Neutron DB operations, the Neutron DB will"},{"line_number":140,"context_line":"be rolled back."}],"source_content_type":"text/x-rst","patch_set":8,"id":"7a5de9d1_01c32458","line":137,"range":{"start_line":135,"start_character":0,"end_line":137,"end_character":56},"in_reply_to":"7a5de9d1_175db478","updated":"2016-02-01 08:08:48.000000000","message":"Updates: if you use select-for-update as the lock, you have to introduce DB-retry mechanism for deadlocked transactions.\n\nBut as Omer suggested, you need to break the logic of the two DB operations. Neutron DB need to be rolled back due to exceptions but DF DB don\u0027t do it in the same way. So, we still need a same lock id to protect both of them separately.","commit_id":"edb25323a1ab7cb331b052bbca70099a9d4424f0"},{"author":{"_account_id":7805,"name":"Li Ma","email":"skywalker.nick@gmail.com","username":"Nick"},"change_message_id":"fcc0d5e469370e1332537579e3fcaa6bdc1aee8c","unresolved":false,"context_lines":[{"line_number":132,"context_line":"        Else"},{"line_number":133,"context_line":"            Resync database."},{"line_number":134,"context_line":""},{"line_number":135,"context_line":"* When creating the distributed lock, it starts a DB transaction and updates"},{"line_number":136,"context_line":"Neutron DB and DF DB in the same transaction to prevent from inconsistency. As"},{"line_number":137,"context_line":"a result, the DB operations are both conducted or failed."},{"line_number":138,"context_line":""},{"line_number":139,"context_line":"* If some exception happens during Neutron DB operations, the Neutron DB will"},{"line_number":140,"context_line":"be rolled back."}],"source_content_type":"text/x-rst","patch_set":8,"id":"7a5de9d1_777508f8","line":137,"range":{"start_line":135,"start_character":0,"end_line":137,"end_character":56},"in_reply_to":"7a5de9d1_97b9a417","updated":"2016-01-31 15:31:58.000000000","message":"Yes, you are right.","commit_id":"edb25323a1ab7cb331b052bbca70099a9d4424f0"},{"author":{"_account_id":7805,"name":"Li Ma","email":"skywalker.nick@gmail.com","username":"Nick"},"change_message_id":"fcc0d5e469370e1332537579e3fcaa6bdc1aee8c","unresolved":false,"context_lines":[{"line_number":132,"context_line":"        Else"},{"line_number":133,"context_line":"            Resync database."},{"line_number":134,"context_line":""},{"line_number":135,"context_line":"* When creating the distributed lock, it starts a DB transaction and updates"},{"line_number":136,"context_line":"Neutron DB and DF DB in the same transaction to prevent from inconsistency. As"},{"line_number":137,"context_line":"a result, the DB operations are both conducted or failed."},{"line_number":138,"context_line":""},{"line_number":139,"context_line":"* If some exception happens during Neutron DB operations, the Neutron DB will"},{"line_number":140,"context_line":"be rolled back."}],"source_content_type":"text/x-rst","patch_set":8,"id":"7a5de9d1_175db478","line":137,"range":{"start_line":135,"start_character":0,"end_line":137,"end_character":56},"in_reply_to":"7a5de9d1_a4cef72a","updated":"2016-01-31 15:31:58.000000000","message":"\u0027select for update\u0027 cannot work in clustering deployment. In Neutron, we fixed almost all the \u0027select-for-update\u0027 issues in the previous releases. We will pass \u0027lock-id\u0027 to the following DF DB operations and compare it with the one in DF DB to make sure they are in the same API session.","commit_id":"edb25323a1ab7cb331b052bbca70099a9d4424f0"},{"author":{"_account_id":20229,"name":"Omer Anson","email":"omer.anson@toganetworks.com","username":"oanson"},"change_message_id":"db6886fca33f9d87eea642a37797707f6a1afde2","unresolved":false,"context_lines":[{"line_number":132,"context_line":"        Else"},{"line_number":133,"context_line":"            Resync database."},{"line_number":134,"context_line":""},{"line_number":135,"context_line":"* When creating the distributed lock, it starts a DB transaction and updates"},{"line_number":136,"context_line":"Neutron DB and DF DB in the same transaction to prevent from inconsistency. As"},{"line_number":137,"context_line":"a result, the DB operations are both conducted or failed."},{"line_number":138,"context_line":""},{"line_number":139,"context_line":"* If some exception happens during Neutron DB operations, the Neutron DB will"},{"line_number":140,"context_line":"be rolled back."}],"source_content_type":"text/x-rst","patch_set":8,"id":"7a5de9d1_97b9a417","line":137,"range":{"start_line":135,"start_character":0,"end_line":137,"end_character":56},"in_reply_to":"7a5de9d1_a4cef72a","updated":"2016-01-31 14:39:32.000000000","message":"I don\u0027t think we should fail the Neutron DB transaction. If the NeutronDB update succeeds, the DF DB (which is the slave) should allow it. DF should recover gracefully from any errors on the DF side, using e.g. a full re-sync.","commit_id":"edb25323a1ab7cb331b052bbca70099a9d4424f0"},{"author":{"_account_id":20229,"name":"Omer Anson","email":"omer.anson@toganetworks.com","username":"oanson"},"change_message_id":"db6886fca33f9d87eea642a37797707f6a1afde2","unresolved":false,"context_lines":[{"line_number":140,"context_line":"be rolled back."},{"line_number":141,"context_line":""},{"line_number":142,"context_line":"* If some exception happens during DF DB operations, the Neutron DB doesn\u0027t"},{"line_number":143,"context_line":"need to be rolled back. Instead, it will invoke full-sync or partial-sync from"},{"line_number":144,"context_line":"Neutron DB to DF DB because the DF DB has dirty data."},{"line_number":145,"context_line":""},{"line_number":146,"context_line":"* The DF DB operations are queued in the function calling chain, and they are"}],"source_content_type":"text/x-rst","patch_set":8,"id":"7a5de9d1_9750c4d5","line":143,"updated":"2016-01-31 14:39:32.000000000","message":"I\u0027d go as far as saying it *shouldn\u0027t* be rolled back.","commit_id":"edb25323a1ab7cb331b052bbca70099a9d4424f0"},{"author":{"_account_id":7805,"name":"Li Ma","email":"skywalker.nick@gmail.com","username":"Nick"},"change_message_id":"fcc0d5e469370e1332537579e3fcaa6bdc1aee8c","unresolved":false,"context_lines":[{"line_number":140,"context_line":"be rolled back."},{"line_number":141,"context_line":""},{"line_number":142,"context_line":"* If some exception happens during DF DB operations, the Neutron DB doesn\u0027t"},{"line_number":143,"context_line":"need to be rolled back. Instead, it will invoke full-sync or partial-sync from"},{"line_number":144,"context_line":"Neutron DB to DF DB because the DF DB has dirty data."},{"line_number":145,"context_line":""},{"line_number":146,"context_line":"* The DF DB operations are queued in the function calling chain, and they are"}],"source_content_type":"text/x-rst","patch_set":8,"id":"7a5de9d1_97720410","line":143,"in_reply_to":"7a5de9d1_9750c4d5","updated":"2016-01-31 15:31:58.000000000","message":"Yes.","commit_id":"edb25323a1ab7cb331b052bbca70099a9d4424f0"},{"author":{"_account_id":18668,"name":"ding bo","email":"longddropt@163.com","username":"longddropt"},"change_message_id":"90a0ce16ccfbc73864468c8b957d1e1730d74eca","unresolved":false,"context_lines":[{"line_number":17,"context_line":"network topology of OpenStack clouds. It is a reliable relational data store"},{"line_number":18,"context_line":"and can be considered as the master DB in Dragonflow distributed architecture."},{"line_number":19,"context_line":"As a result, the question becomes how to synchronize the slave DB (Dragonflow"},{"line_number":20,"context_line":"distributed NoSQL data store) to the master DB (Neutron data store). In this"},{"line_number":21,"context_line":"design, we will propose a simple and feasible distributed lock solution and"},{"line_number":22,"context_line":"improve it later according to evaluation in production."},{"line_number":23,"context_line":""}],"source_content_type":"text/x-rst","patch_set":11,"id":"5a5ae5dd_1619aa88","line":20,"range":{"start_line":20,"start_character":30,"end_line":20,"end_character":32},"updated":"2016-02-04 03:28:12.000000000","message":"from","commit_id":"4f8b6775458411e64771425839b018a7c8d8e92c"},{"author":{"_account_id":18668,"name":"ding bo","email":"longddropt@163.com","username":"longddropt"},"change_message_id":"ced2dce108815ac4ec4a5f1b5487c4dce39d0617","unresolved":false,"context_lines":[{"line_number":132,"context_line":"        Else"},{"line_number":133,"context_line":"            Resync database."},{"line_number":134,"context_line":""},{"line_number":135,"context_line":"* When creating the distributed lock, it starts a DB transaction and updates"},{"line_number":136,"context_line":"Neutron DB and DF DB in the same transaction to prevent from inconsistency. As"},{"line_number":137,"context_line":"a result, the DB operations are both conducted or failed."},{"line_number":138,"context_line":""},{"line_number":139,"context_line":"* If some exception happens during Neutron DB operations, the Neutron DB will"}],"source_content_type":"text/x-rst","patch_set":11,"id":"5a5ae5dd_9c7c0b22","line":136,"range":{"start_line":135,"start_character":2,"end_line":136,"end_character":74},"updated":"2016-02-03 09:27:34.000000000","message":"Sorry, I still cannot understand this logic. Transaction itself cannot be lock. We can have two process executing in two transactions in which manipulating the same data. Unless we set the transaction isolation level to \"serials\", other isolation levels such as \"read uncommitted\", \"read committed\" and \"repeatable read\" cannot prevent concurrency. And what is the meaning of  \"Release the distributed lock in DF DB\"(line 313)?  I did not see where we lock up the lock in DF DB. Please let me know if I have something misunderstood.","commit_id":"4f8b6775458411e64771425839b018a7c8d8e92c"},{"author":{"_account_id":7805,"name":"Li Ma","email":"skywalker.nick@gmail.com","username":"Nick"},"change_message_id":"393e3c53cbda1b4d1da5ebd6564e7e8c56b74cae","unresolved":false,"context_lines":[{"line_number":132,"context_line":"        Else"},{"line_number":133,"context_line":"            Resync database."},{"line_number":134,"context_line":""},{"line_number":135,"context_line":"* When creating the distributed lock, it starts a DB transaction and updates"},{"line_number":136,"context_line":"Neutron DB and DF DB in the same transaction to prevent from inconsistency. As"},{"line_number":137,"context_line":"a result, the DB operations are both conducted or failed."},{"line_number":138,"context_line":""},{"line_number":139,"context_line":"* If some exception happens during Neutron DB operations, the Neutron DB will"}],"source_content_type":"text/x-rst","patch_set":11,"id":"5a5ae5dd_cf912c54","line":136,"range":{"start_line":135,"start_character":2,"end_line":136,"end_character":74},"in_reply_to":"5a5ae5dd_9c4059a7","updated":"2016-02-04 06:44:54.000000000","message":"So you wanna implement your version control mechanism and store the the latest versions into both DF DB and Neutron DB. As you mentioned that PUB/SUB needs it, I think it\u0027s worth discussing.","commit_id":"4f8b6775458411e64771425839b018a7c8d8e92c"},{"author":{"_account_id":7805,"name":"Li Ma","email":"skywalker.nick@gmail.com","username":"Nick"},"change_message_id":"393e3c53cbda1b4d1da5ebd6564e7e8c56b74cae","unresolved":false,"context_lines":[{"line_number":132,"context_line":"        Else"},{"line_number":133,"context_line":"            Resync database."},{"line_number":134,"context_line":""},{"line_number":135,"context_line":"* When creating the distributed lock, it starts a DB transaction and updates"},{"line_number":136,"context_line":"Neutron DB and DF DB in the same transaction to prevent from inconsistency. As"},{"line_number":137,"context_line":"a result, the DB operations are both conducted or failed."},{"line_number":138,"context_line":""},{"line_number":139,"context_line":"* If some exception happens during Neutron DB operations, the Neutron DB will"}],"source_content_type":"text/x-rst","patch_set":11,"id":"5a5ae5dd_4f09bcdc","line":136,"range":{"start_line":135,"start_character":2,"end_line":136,"end_character":74},"in_reply_to":"5a5ae5dd_9c7c0b22","updated":"2016-02-04 06:44:54.000000000","message":"1. If we take advantage of \u0027query.with_for_update\u0027 and DB retry, we can implement this lock by SQL transaction.\nODL journal uses such a technique.\nhttps://github.com/openstack/networking-odl/commit/78f656d95cf031772a315e6d9b1c95e57eaf9a8a\n\n2. Thanks for your question about key operations. Actually, I\u0027m thinking of simplifying the procedure. I will update it soon.","commit_id":"4f8b6775458411e64771425839b018a7c8d8e92c"},{"author":{"_account_id":18668,"name":"ding bo","email":"longddropt@163.com","username":"longddropt"},"change_message_id":"90a0ce16ccfbc73864468c8b957d1e1730d74eca","unresolved":false,"context_lines":[{"line_number":132,"context_line":"        Else"},{"line_number":133,"context_line":"            Resync database."},{"line_number":134,"context_line":""},{"line_number":135,"context_line":"* When creating the distributed lock, it starts a DB transaction and updates"},{"line_number":136,"context_line":"Neutron DB and DF DB in the same transaction to prevent from inconsistency. As"},{"line_number":137,"context_line":"a result, the DB operations are both conducted or failed."},{"line_number":138,"context_line":""},{"line_number":139,"context_line":"* If some exception happens during Neutron DB operations, the Neutron DB will"}],"source_content_type":"text/x-rst","patch_set":11,"id":"5a5ae5dd_9c4059a7","line":136,"range":{"start_line":135,"start_character":2,"end_line":136,"end_character":74},"in_reply_to":"5a5ae5dd_9c7c0b22","updated":"2016-02-04 03:28:12.000000000","message":"What I think is to add a version number(sync anchor) for every key in DF DB table which is also needed by pub-sub mechanism. And we store this version number in a new neutron DB table.The method goes like this:\nIn Neutron plugin:\n    When an API is processing:\n        select version number corresponding to the key for update  \n\n        Start Neutron DB transaction.\n            Do Neutron DB operations.\n            If exception ocuurs:\n                roll back\n\n        retry calling get_key() until return version number\u003dcurrent number \n        call set_key()with new version number.\n        Emit messages via PUB/SUB with new version number.\n        update the version number to new version number in neutron DB.\n        If exception ocuurs:\n            Resync database.","commit_id":"4f8b6775458411e64771425839b018a7c8d8e92c"},{"author":{"_account_id":18668,"name":"ding bo","email":"longddropt@163.com","username":"longddropt"},"change_message_id":"d5ce88cd6795062cfa35af8490de46e240d3f527","unresolved":false,"context_lines":[{"line_number":132,"context_line":"        Else"},{"line_number":133,"context_line":"            Resync database."},{"line_number":134,"context_line":""},{"line_number":135,"context_line":"* When creating the distributed lock, it starts a DB transaction and updates"},{"line_number":136,"context_line":"Neutron DB and DF DB in the same transaction to prevent from inconsistency. As"},{"line_number":137,"context_line":"a result, the DB operations are both conducted or failed."},{"line_number":138,"context_line":""},{"line_number":139,"context_line":"* If some exception happens during Neutron DB operations, the Neutron DB will"}],"source_content_type":"text/x-rst","patch_set":11,"id":"5a5ae5dd_bb851108","line":136,"range":{"start_line":135,"start_character":2,"end_line":136,"end_character":74},"in_reply_to":"5a5ae5dd_cf912c54","updated":"2016-02-05 02:48:59.000000000","message":"Yes","commit_id":"4f8b6775458411e64771425839b018a7c8d8e92c"},{"author":{"_account_id":20229,"name":"Omer Anson","email":"omer.anson@toganetworks.com","username":"oanson"},"change_message_id":"ee0c39b567e782dddf344e5f6ff2e49bf9eedfb8","unresolved":false,"context_lines":[{"line_number":82,"context_line":"corresponding network table \u0027lswitch\u0027, which increases the frequency"},{"line_number":83,"context_line":"of concurrent read/write operations on the same key. In order to reduce"},{"line_number":84,"context_line":"the possibility of race condition, we need to refactor the db schema."},{"line_number":85,"context_line":""},{"line_number":86,"context_line":"Full-Sync Mechanism"},{"line_number":87,"context_line":"-------------------"},{"line_number":88,"context_line":""}],"source_content_type":"text/x-rst","patch_set":14,"id":"5a5ae5dd_719e6808","line":85,"updated":"2016-02-07 06:20:44.000000000","message":"On an unrelated note, this will also make pub/sub subnet events more concise.","commit_id":"6be20c69692ba212527e81d57ce75a01c1e2c53f"},{"author":{"_account_id":20229,"name":"Omer Anson","email":"omer.anson@toganetworks.com","username":"oanson"},"change_message_id":"ee0c39b567e782dddf344e5f6ff2e49bf9eedfb8","unresolved":false,"context_lines":[{"line_number":117,"context_line":"        Generate a lock-id for the current API context."},{"line_number":118,"context_line":"        Create a distributed lock with lock-id in Neutron DB."},{"line_number":119,"context_line":""},{"line_number":120,"context_line":"        Start Neutron DB transaction (wrapped by DB retry)"},{"line_number":121,"context_line":"            Query the lock-id for update."},{"line_number":122,"context_line":"            If the lock-id exists in Neutron DB"},{"line_number":123,"context_line":"                Do Neutron DB operations."},{"line_number":124,"context_line":"                Backup DF DB objects for restore."},{"line_number":125,"context_line":"                Do DF DB operations."},{"line_number":126,"context_line":"                Emit messages via PUB/SUB."},{"line_number":127,"context_line":"            Else"},{"line_number":128,"context_line":"                Rollback DF DB by restoring the modified objects."},{"line_number":129,"context_line":"                Rollback Neutron DB transaction."},{"line_number":130,"context_line":""},{"line_number":131,"context_line":"* When creating the distributed lock, it starts a DB transaction and inserts"},{"line_number":132,"context_line":"a lock record into Neutron DB according to the current tenant. If some record"}],"source_content_type":"text/x-rst","patch_set":14,"id":"5a5ae5dd_f19258f5","line":129,"range":{"start_line":120,"start_character":0,"end_line":129,"end_character":48},"updated":"2016-02-07 06:20:44.000000000","message":"With this algorithm, it is still possible that we roll back the neutron db for an error in the DF db. This should be avoided. An error in the DF DB should cause a full-sync, not a Neutron DB rollback.","commit_id":"6be20c69692ba212527e81d57ce75a01c1e2c53f"},{"author":{"_account_id":20229,"name":"Omer Anson","email":"omer.anson@toganetworks.com","username":"oanson"},"change_message_id":"ee0c39b567e782dddf344e5f6ff2e49bf9eedfb8","unresolved":false,"context_lines":[{"line_number":142,"context_line":"* For DF DB rollback, we first backup all the related objects from DF DB."},{"line_number":143,"context_line":"If some exceptions happen, the object will be restored by backup."},{"line_number":144,"context_line":""},{"line_number":145,"context_line":"Phase 2: Journal Mechanism"},{"line_number":146,"context_line":"--------------------------"},{"line_number":147,"context_line":""},{"line_number":148,"context_line":"The drawback of Phase 1 is obvious. If the exceptions happen during DF DB"}],"source_content_type":"text/x-rst","patch_set":14,"id":"5a5ae5dd_d1695cfa","line":145,"updated":"2016-02-07 06:20:44.000000000","message":"Separation in phases here means implementation phase? i.e. first implement phase 1, then phase 2?","commit_id":"6be20c69692ba212527e81d57ce75a01c1e2c53f"},{"author":{"_account_id":18668,"name":"ding bo","email":"longddropt@163.com","username":"longddropt"},"change_message_id":"2cddc8afa47a6ec3dde3ace291ad00aae59aab81","unresolved":false,"context_lines":[{"line_number":162,"context_line":""},{"line_number":163,"context_line":"In Journal Thread running at the background:"},{"line_number":164,"context_line":"    Get each DF DB operation in a loop,"},{"line_number":165,"context_line":"       according to each session-id and the timestamp."},{"line_number":166,"context_line":"        Do it."},{"line_number":167,"context_line":"        Emit messages via PUB/SUB."},{"line_number":168,"context_line":""}],"source_content_type":"text/x-rst","patch_set":14,"id":"5a5ae5dd_7bf52755","line":165,"range":{"start_line":165,"start_character":7,"end_line":165,"end_character":13},"updated":"2016-02-05 02:53:16.000000000","message":"adjust alignment.","commit_id":"6be20c69692ba212527e81d57ce75a01c1e2c53f"},{"author":{"_account_id":18668,"name":"ding bo","email":"longddropt@163.com","username":"longddropt"},"change_message_id":"2cddc8afa47a6ec3dde3ace291ad00aae59aab81","unresolved":false,"context_lines":[{"line_number":169,"context_line":"    If some exception happens during the loop:"},{"line_number":170,"context_line":"        Resync the database."},{"line_number":171,"context_line":""},{"line_number":172,"context_line":"The design of such a journal has been implemented in ODL plugin [4]."},{"line_number":173,"context_line":""},{"line_number":174,"context_line":"Data Model Impact"},{"line_number":175,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":14,"id":"5a5ae5dd_7e899574","line":172,"range":{"start_line":172,"start_character":0,"end_line":172,"end_character":63},"updated":"2016-02-05 02:53:16.000000000","message":"I think this method is better. And this solution is very similar to the one I have mentioned before: use rabbitmq to make the DF DB operations serial with one exception: rabbitmq uses queue, and journal uses DB table.And rabbitmq solution has a advantage: it can has topics to make every topic a dedicated thread to consume it which is a advantage in multi-processor. And I think we also should add the version number field in the journal db column. Any way. we should keep tracking of the version of nosql objects.","commit_id":"6be20c69692ba212527e81d57ce75a01c1e2c53f"},{"author":{"_account_id":18668,"name":"ding bo","email":"longddropt@163.com","username":"longddropt"},"change_message_id":"551b431921f70efbd191adeafa84c42ad153bd03","unresolved":false,"context_lines":[{"line_number":87,"context_line":"        Run compare-db to identify the difference between two DBs."},{"line_number":88,"context_line":"        Run update-df-db to make it be synchronized."},{"line_number":89,"context_line":""},{"line_number":90,"context_line":"    Run a df-db-health-checker greenthread or external process:"},{"line_number":91,"context_line":"    (this should be an option for df-plugin):"},{"line_number":92,"context_line":"        Periodically run compare-db and call update-df-db if necessary."},{"line_number":93,"context_line":""}],"source_content_type":"text/x-rst","patch_set":20,"id":"bae84128_1e069a51","line":90,"range":{"start_line":90,"start_character":4,"end_line":90,"end_character":57},"updated":"2016-02-24 02:45:28.000000000","message":"I dont think background greenthread helps a lot. If we depend on it to completely recover the inconsistency, we should inevitably lock the whole neutron db, this is an unbearable overhead when neutron server is running.","commit_id":"47aabb03c4f89909fae7616e600c156591b83a6c"},{"author":{"_account_id":7805,"name":"Li Ma","email":"skywalker.nick@gmail.com","username":"Nick"},"change_message_id":"61b7d6c8b3fb0d4099640ffea37225ae1eae6aff","unresolved":false,"context_lines":[{"line_number":87,"context_line":"        Run compare-db to identify the difference between two DBs."},{"line_number":88,"context_line":"        Run update-df-db to make it be synchronized."},{"line_number":89,"context_line":""},{"line_number":90,"context_line":"    Run a df-db-health-checker greenthread or external process:"},{"line_number":91,"context_line":"    (this should be an option for df-plugin):"},{"line_number":92,"context_line":"        Periodically run compare-db and call update-df-db if necessary."},{"line_number":93,"context_line":""}],"source_content_type":"text/x-rst","patch_set":20,"id":"bae84128_830b6e7c","line":90,"range":{"start_line":90,"start_character":4,"end_line":90,"end_character":57},"in_reply_to":"bae84128_1e069a51","updated":"2016-02-24 05:41:19.000000000","message":"we sync neutron objects one by one, according to its version.","commit_id":"47aabb03c4f89909fae7616e600c156591b83a6c"},{"author":{"_account_id":18668,"name":"ding bo","email":"longddropt@163.com","username":"longddropt"},"change_message_id":"551b431921f70efbd191adeafa84c42ad153bd03","unresolved":false,"context_lines":[{"line_number":115,"context_line":""},{"line_number":116,"context_line":"        Start Neutron DB transaction for updating version."},{"line_number":117,"context_line":"            Read the version from Neutron DB."},{"line_number":118,"context_line":"            Increment the version of the corresponding Neutron object."},{"line_number":119,"context_line":""},{"line_number":120,"context_line":"        Acquire the distributed lock for the Neutron object."},{"line_number":121,"context_line":"            Compare the version from Neutron DB and the version from DF DB."}],"source_content_type":"text/x-rst","patch_set":20,"id":"bae84128_792c2099","line":118,"range":{"start_line":118,"start_character":12,"end_line":118,"end_character":21},"updated":"2016-02-24 02:45:28.000000000","message":"increase","commit_id":"47aabb03c4f89909fae7616e600c156591b83a6c"},{"author":{"_account_id":18668,"name":"ding bo","email":"longddropt@163.com","username":"longddropt"},"change_message_id":"551b431921f70efbd191adeafa84c42ad153bd03","unresolved":false,"context_lines":[{"line_number":123,"context_line":"                Do DF DB operations."},{"line_number":124,"context_line":"                Emit messages via PUB/SUB."},{"line_number":125,"context_line":"            Else:"},{"line_number":126,"context_line":"                raise BadVersionException."},{"line_number":127,"context_line":"        Release the distributed lock."},{"line_number":128,"context_line":""},{"line_number":129,"context_line":"* When creating the distributed lock, it starts a DB transaction and inserts"}],"source_content_type":"text/x-rst","patch_set":20,"id":"bae84128_d9a44c09","line":126,"range":{"start_line":126,"start_character":16,"end_line":126,"end_character":41},"updated":"2016-02-24 02:45:28.000000000","message":"Then how we deal with this case? As I mentioned before, I dont think we could rely on background greenthread to completely recover the consistency of neutron db and df db","commit_id":"47aabb03c4f89909fae7616e600c156591b83a6c"},{"author":{"_account_id":7805,"name":"Li Ma","email":"skywalker.nick@gmail.com","username":"Nick"},"change_message_id":"61b7d6c8b3fb0d4099640ffea37225ae1eae6aff","unresolved":false,"context_lines":[{"line_number":123,"context_line":"                Do DF DB operations."},{"line_number":124,"context_line":"                Emit messages via PUB/SUB."},{"line_number":125,"context_line":"            Else:"},{"line_number":126,"context_line":"                raise BadVersionException."},{"line_number":127,"context_line":"        Release the distributed lock."},{"line_number":128,"context_line":""},{"line_number":129,"context_line":"* When creating the distributed lock, it starts a DB transaction and inserts"}],"source_content_type":"text/x-rst","patch_set":20,"id":"bae84128_c329f6e7","line":126,"range":{"start_line":126,"start_character":16,"end_line":126,"end_character":41},"in_reply_to":"bae84128_d9a44c09","updated":"2016-02-24 05:41:19.000000000","message":"issue a resync on this object only.","commit_id":"47aabb03c4f89909fae7616e600c156591b83a6c"},{"author":{"_account_id":18668,"name":"ding bo","email":"longddropt@163.com","username":"longddropt"},"change_message_id":"551b431921f70efbd191adeafa84c42ad153bd03","unresolved":false,"context_lines":[{"line_number":139,"context_line":"exception happens, it will retry it to make sure it will be committed later."},{"line_number":140,"context_line":""},{"line_number":141,"context_line":"* Potential Issue: When concurrent write operations on a certain key happen,"},{"line_number":142,"context_line":"due to the inconsistency window of DF DB. I introduce a restore operation for"},{"line_number":143,"context_line":"DF DB. As long as the Neutron DB is committed, it can safely update the object"},{"line_number":144,"context_line":"in DF DB no matter what the version in DF DB is. However, it may not be"},{"line_number":145,"context_line":"effective, because the update on DF DB is always delayed because the previous"}],"source_content_type":"text/x-rst","patch_set":20,"id":"bae84128_391578e4","line":142,"range":{"start_line":142,"start_character":41,"end_line":142,"end_character":73},"updated":"2016-02-24 02:45:28.000000000","message":"Can you explain more details?","commit_id":"47aabb03c4f89909fae7616e600c156591b83a6c"},{"author":{"_account_id":7805,"name":"Li Ma","email":"skywalker.nick@gmail.com","username":"Nick"},"change_message_id":"61b7d6c8b3fb0d4099640ffea37225ae1eae6aff","unresolved":false,"context_lines":[{"line_number":139,"context_line":"exception happens, it will retry it to make sure it will be committed later."},{"line_number":140,"context_line":""},{"line_number":141,"context_line":"* Potential Issue: When concurrent write operations on a certain key happen,"},{"line_number":142,"context_line":"due to the inconsistency window of DF DB. I introduce a restore operation for"},{"line_number":143,"context_line":"DF DB. As long as the Neutron DB is committed, it can safely update the object"},{"line_number":144,"context_line":"in DF DB no matter what the version in DF DB is. However, it may not be"},{"line_number":145,"context_line":"effective, because the update on DF DB is always delayed because the previous"}],"source_content_type":"text/x-rst","patch_set":20,"id":"bae84128_a36b121d","line":142,"range":{"start_line":142,"start_character":41,"end_line":142,"end_character":73},"in_reply_to":"bae84128_391578e4","updated":"2016-02-24 05:41:19.000000000","message":"i will update it asap. it should be removed.","commit_id":"47aabb03c4f89909fae7616e600c156591b83a6c"},{"author":{"_account_id":20229,"name":"Omer Anson","email":"omer.anson@toganetworks.com","username":"oanson"},"change_message_id":"e47b57a2eca66bf0c6b68c40109b4b704cefa63e","unresolved":false,"context_lines":[{"line_number":111,"context_line":"In Neutron plugin:"},{"line_number":112,"context_line":"    When an API is processing:"},{"line_number":113,"context_line":"        Start Neutron DB transaction for network operations."},{"line_number":114,"context_line":"            Do Neutron DB operations."},{"line_number":115,"context_line":""},{"line_number":116,"context_line":"        Acquire the distributed lock for the Neutron object."},{"line_number":117,"context_line":"            Start Neutron DB transaction for updating version."}],"source_content_type":"text/x-rst","patch_set":23,"id":"9aed3d3a_ae0fd42c","line":114,"updated":"2016-02-28 11:48:39.000000000","message":"I think the version bump should be done here, on the first transaction. Otherwise, there may be a race:\nwith lock (1):\n    update 1\nwith lock (2):\n    update 2\nwith distributed lock (2):\n    update and get version 1\nwith distributed lock (1):\n    update and get version 2\n\nThe generated versions will be reversed.\n\nBy generating the version on the original Neutron DB transaction, we only have one NeutronDB transaction, which is better.\n\nThe race may be unavoidable, but if we can reduce the SQL transactions, I think we should.\n\nOr have I missed something?","commit_id":"8e916ebcb2e8d81627f5f9c4c9064afb461c5b80"},{"author":{"_account_id":7805,"name":"Li Ma","email":"skywalker.nick@gmail.com","username":"Nick"},"change_message_id":"a3d02e4dbcecf432d534da20ef66220a3aa5d056","unresolved":false,"context_lines":[{"line_number":111,"context_line":"In Neutron plugin:"},{"line_number":112,"context_line":"    When an API is processing:"},{"line_number":113,"context_line":"        Start Neutron DB transaction for network operations."},{"line_number":114,"context_line":"            Do Neutron DB operations."},{"line_number":115,"context_line":""},{"line_number":116,"context_line":"        Acquire the distributed lock for the Neutron object."},{"line_number":117,"context_line":"            Start Neutron DB transaction for updating version."}],"source_content_type":"text/x-rst","patch_set":23,"id":"5aef4532_9c6e8fff","line":114,"in_reply_to":"9aed3d3a_ae0fd42c","updated":"2016-03-09 06:09:12.000000000","message":"api 1: do update in neutron db\napi 2: do update in neutron db\n\nwith distributed lock(api-2):\n    update neutron db from 0 to 1\n    get version 1 and update df db to 1\n\nwith distributed lock(api-1):\n    update neutron db from 1 to 2\n    get version 2 and update df db to 2\n\nActually, the logic is reversed due to the separate transactions, but it seems that \nit is self-healing because there\u0027s only one distributed lock to protect the whole context.","commit_id":"8e916ebcb2e8d81627f5f9c4c9064afb461c5b80"},{"author":{"_account_id":20229,"name":"Omer Anson","email":"omer.anson@toganetworks.com","username":"oanson"},"change_message_id":"e47b57a2eca66bf0c6b68c40109b4b704cefa63e","unresolved":false,"context_lines":[{"line_number":140,"context_line":"row inside the transaction will continue to not return the new value under"},{"line_number":141,"context_line":"Neutron default transaction isolation level (REPEATABLE). To deal with this"},{"line_number":142,"context_line":"problem, we just re-issue a new transaction for re-read the lock row and"},{"line_number":143,"context_line":"update it."},{"line_number":144,"context_line":""},{"line_number":145,"context_line":"* If MySQL clustering involves, only one lock transaction will be committed"},{"line_number":146,"context_line":"and others will be deadlocked. Here we introduce DB retry mechanism provided"}],"source_content_type":"text/x-rst","patch_set":23,"id":"9aed3d3a_ee471c49","line":143,"updated":"2016-02-28 11:48:39.000000000","message":"Isn\u0027t there a danger of version mismatch? e.g. first transaction gets second transaction\u0027s version, and vice versa, as per the race I mentioned above?","commit_id":"8e916ebcb2e8d81627f5f9c4c9064afb461c5b80"},{"author":{"_account_id":20229,"name":"Omer Anson","email":"omer.anson@toganetworks.com","username":"oanson"},"change_message_id":"4d1f48eca557f2675f4311fc70ebe0a0890d4857","unresolved":false,"context_lines":[{"line_number":126,"context_line":"                raise BadVersionException."},{"line_number":127,"context_line":"        Release the distributed lock."},{"line_number":128,"context_line":""},{"line_number":129,"context_line":"* When creating the distributed lock, it starts a DB transaction and inserts"},{"line_number":130,"context_line":"a lock record into Neutron DB according to the current object."},{"line_number":131,"context_line":""},{"line_number":132,"context_line":"* When acquiring the distributed lock, it first issue SELECT-FOR-UPDATE to"}],"source_content_type":"text/x-rst","patch_set":24,"id":"7af24918_2a50f15d","line":129,"updated":"2016-03-02 11:31:58.000000000","message":"I don\u0027t think I understand - this is the process for aquiring a distributed lock?","commit_id":"b99380e27d42ec5636ac3c32ad3fe0bb4827cde8"},{"author":{"_account_id":7805,"name":"Li Ma","email":"skywalker.nick@gmail.com","username":"Nick"},"change_message_id":"b2e6b0c6150c7c30666330405920053c39c9c414","unresolved":false,"context_lines":[{"line_number":126,"context_line":"                raise BadVersionException."},{"line_number":127,"context_line":"        Release the distributed lock."},{"line_number":128,"context_line":""},{"line_number":129,"context_line":"* When creating the distributed lock, it starts a DB transaction and inserts"},{"line_number":130,"context_line":"a lock record into Neutron DB according to the current object."},{"line_number":131,"context_line":""},{"line_number":132,"context_line":"* When acquiring the distributed lock, it first issue SELECT-FOR-UPDATE to"}],"source_content_type":"text/x-rst","patch_set":24,"id":"5aef4532_9bbc146e","line":129,"in_reply_to":"7af24918_2a50f15d","updated":"2016-03-08 03:18:16.000000000","message":"creates a lock record, not acquiring it. this operation only happens when we create a neutron object.","commit_id":"b99380e27d42ec5636ac3c32ad3fe0bb4827cde8"},{"author":{"_account_id":20229,"name":"Omer Anson","email":"omer.anson@toganetworks.com","username":"oanson"},"change_message_id":"c29fd303b966c252d64b83d18716a352f82296fa","unresolved":false,"context_lines":[{"line_number":114,"context_line":"            Do Neutron DB operations."},{"line_number":115,"context_line":""},{"line_number":116,"context_line":"        Acquire the distributed lock for the Neutron object."},{"line_number":117,"context_line":"            Start Neutron DB transaction for updating version."},{"line_number":118,"context_line":"                Read the version from Neutron DB."},{"line_number":119,"context_line":"                Increase the version of the corresponding Neutron object."},{"line_number":120,"context_line":""},{"line_number":121,"context_line":"            Compare the version from Neutron DB and the version from DF DB."},{"line_number":122,"context_line":"            If they are equal:"}],"source_content_type":"text/x-rst","patch_set":25,"id":"5aef4532_926ea594","line":119,"range":{"start_line":117,"start_character":0,"end_line":119,"end_character":73},"updated":"2016-03-08 11:55:23.000000000","message":"This operation is done with both locks, and separately from the transaction which actually makes the change. Is this intentional? (Re: my comment on patch-set 23)","commit_id":"2460af85d36f5ecb516e65676ace7abf909872cd"},{"author":{"_account_id":7805,"name":"Li Ma","email":"skywalker.nick@gmail.com","username":"Nick"},"change_message_id":"ea70e4b482549779093dde53127780254a19baa2","unresolved":false,"context_lines":[{"line_number":114,"context_line":"            Do Neutron DB operations."},{"line_number":115,"context_line":""},{"line_number":116,"context_line":"        Acquire the distributed lock for the Neutron object."},{"line_number":117,"context_line":"            Start Neutron DB transaction for updating version."},{"line_number":118,"context_line":"                Read the version from Neutron DB."},{"line_number":119,"context_line":"                Increase the version of the corresponding Neutron object."},{"line_number":120,"context_line":""},{"line_number":121,"context_line":"            Compare the version from Neutron DB and the version from DF DB."},{"line_number":122,"context_line":"            If they are equal:"}],"source_content_type":"text/x-rst","patch_set":25,"id":"5aef4532_3f4b9f5a","line":119,"range":{"start_line":117,"start_character":0,"end_line":119,"end_character":73},"in_reply_to":"5aef4532_2662c5ad","updated":"2016-03-10 11:33:17.000000000","message":"SELECT-FOR-UPDATE lock primitive provided by SQL backend leads to deadlock exception in the clustering deployment. You need to catch and retry it to let the transaction continue to work. This mechanism is implemented in oslo.db.\n\nHowever, you cannot retry a part of transaction (subtransaction, or something like it), which makes its parent in error state. If we retry the whole transaction in neutron core plugin, the logic is also wrong. For example, we call core plugin api from another core plugin api, like create_subnet:\n\ncall create_subnet:\n    start neutron transaction 1:\n        db.create_subnet\n        start neutron transaction 1-1 for versioning\n            db.increate_version\n        call create_port for dhcp:\n            start neutron transaction 1-2 for dhcp\n                ...\n\n(The subtransaction generally means it belongs to its parent and make itself as a part of the parent transaction) See [1].\n\nIf we do retry on the same parent transaction 1, if deadlock happens, the transactions (1, 1-1, 1-2...) will be all rolled back and the records about new dhcp port are all removed, but how about DF DB records created by create_port? We have to delete them from DF DB manually at the plugin level. And finally, we retry the whole logic. But, there are lots of nested calling on core plugin api, which makes the internal logic very complicated.\n\nThe way that can solve the problem you mentioned is not using db-retry, which, however, leads to incompatibility of mysql clustering. This is also, what I said in the spec, the first step on db consistency. If we wanna make it perfect, we may discuss about how to get rid of the neutron db or using an external distributed lock like openstack/tooz.\n\n[1] http://docs.sqlalchemy.org/en/latest/orm/session_transaction.html","commit_id":"2460af85d36f5ecb516e65676ace7abf909872cd"},{"author":{"_account_id":7805,"name":"Li Ma","email":"skywalker.nick@gmail.com","username":"Nick"},"change_message_id":"3fb31c4e1249676c127dc90aa902ba39d28e59d2","unresolved":false,"context_lines":[{"line_number":114,"context_line":"            Do Neutron DB operations."},{"line_number":115,"context_line":""},{"line_number":116,"context_line":"        Acquire the distributed lock for the Neutron object."},{"line_number":117,"context_line":"            Start Neutron DB transaction for updating version."},{"line_number":118,"context_line":"                Read the version from Neutron DB."},{"line_number":119,"context_line":"                Increase the version of the corresponding Neutron object."},{"line_number":120,"context_line":""},{"line_number":121,"context_line":"            Compare the version from Neutron DB and the version from DF DB."},{"line_number":122,"context_line":"            If they are equal:"}],"source_content_type":"text/x-rst","patch_set":25,"id":"5aef4532_b718ea85","line":119,"range":{"start_line":117,"start_character":0,"end_line":119,"end_character":73},"in_reply_to":"5aef4532_926ea594","updated":"2016-03-08 15:01:03.000000000","message":"The intentional behavior is based on L37-38 [1].\n[1] https://review.openstack.org/#/c/282290/24/dragonflow/db/neutron/versionedobjects_db.py\n\nIf we do not use db-retry of oslo.db, it doesn\u0027t work in mysql clustering. So we have to separate the transactions. Do you have some suggestions? I\u0027ll try to understand the potential race you noticed tomorrow. Hopefully I will find out and deal with it.","commit_id":"2460af85d36f5ecb516e65676ace7abf909872cd"},{"author":{"_account_id":20229,"name":"Omer Anson","email":"omer.anson@toganetworks.com","username":"oanson"},"change_message_id":"7f7921fc8e8361ba43b2473b1c1147d68993b8e3","unresolved":false,"context_lines":[{"line_number":114,"context_line":"            Do Neutron DB operations."},{"line_number":115,"context_line":""},{"line_number":116,"context_line":"        Acquire the distributed lock for the Neutron object."},{"line_number":117,"context_line":"            Start Neutron DB transaction for updating version."},{"line_number":118,"context_line":"                Read the version from Neutron DB."},{"line_number":119,"context_line":"                Increase the version of the corresponding Neutron object."},{"line_number":120,"context_line":""},{"line_number":121,"context_line":"            Compare the version from Neutron DB and the version from DF DB."},{"line_number":122,"context_line":"            If they are equal:"}],"source_content_type":"text/x-rst","patch_set":25,"id":"5aef4532_2662c5ad","line":119,"range":{"start_line":117,"start_character":0,"end_line":119,"end_character":73},"in_reply_to":"5aef4532_9be859b6","updated":"2016-03-09 07:19:24.000000000","message":"The race condition is theoretical, and may appear only in very specific scenarios (or not at all). I haven\u0027t tried to actually create it.\n\nWhat about using the same transaction, and not sub-transactions? What do you mean by \u0027burst its parent\u0027?","commit_id":"2460af85d36f5ecb516e65676ace7abf909872cd"},{"author":{"_account_id":7805,"name":"Li Ma","email":"skywalker.nick@gmail.com","username":"Nick"},"change_message_id":"35d5161ade26b3c727d93acf091de9caea0896aa","unresolved":false,"context_lines":[{"line_number":114,"context_line":"            Do Neutron DB operations."},{"line_number":115,"context_line":""},{"line_number":116,"context_line":"        Acquire the distributed lock for the Neutron object."},{"line_number":117,"context_line":"            Start Neutron DB transaction for updating version."},{"line_number":118,"context_line":"                Read the version from Neutron DB."},{"line_number":119,"context_line":"                Increase the version of the corresponding Neutron object."},{"line_number":120,"context_line":""},{"line_number":121,"context_line":"            Compare the version from Neutron DB and the version from DF DB."},{"line_number":122,"context_line":"            If they are equal:"}],"source_content_type":"text/x-rst","patch_set":25,"id":"5aef4532_9be859b6","line":119,"range":{"start_line":117,"start_character":0,"end_line":119,"end_character":73},"in_reply_to":"5aef4532_b718ea85","updated":"2016-03-09 06:07:50.000000000","message":"I concurrently run 100+ create_subnet calls and I cannot identify the race during testing.\n\nIf we move version bump into the first transaction, a InvalidRequestError will be raised by sqlalchemy. It is because you cannot create a new transaction inside the existed transaction without enabling subtransanction. However if we continue to make it by enabling subtransaction, the select-for-update lock with oslo.db retry will burst its parent.","commit_id":"2460af85d36f5ecb516e65676ace7abf909872cd"},{"author":{"_account_id":11364,"name":"Fei Rao","email":"milo.frao@gmail.com","username":"milo.frao"},"change_message_id":"00c8eddacd2921654fcb3a34c3f9c97da4db9f66","unresolved":false,"context_lines":[{"line_number":116,"context_line":""},{"line_number":117,"context_line":"When creating a neutron object::"},{"line_number":118,"context_line":""},{"line_number":119,"context_line":"    def create_object(context, obj):"},{"line_number":120,"context_line":"        with db_api.autonested_transaction(context.session):"},{"line_number":121,"context_line":"            obj_db \u003d super(Plugin, self).create_object(context, obj)"},{"line_number":122,"context_line":"        lock_db.create_lock(context, obj_db[\u0027id\u0027])"},{"line_number":123,"context_line":"        nb_lock \u003d lock_db.DFDBLock(context, obj_db[\u0027id\u0027])"},{"line_number":124,"context_line":"        with nb_lock:"},{"line_number":125,"context_line":"            self.nb_api.create_object(name\u003dobj_db[\u0027id\u0027],"},{"line_number":126,"context_line":"                                      topic\u003dobj_db[\u0027tenant_id\u0027],"},{"line_number":127,"context_line":"                                      obj_db)"},{"line_number":128,"context_line":""},{"line_number":129,"context_line":"When updating a neutron object::"},{"line_number":130,"context_line":""}],"source_content_type":"text/x-rst","patch_set":27,"id":"5aef4532_32ca766f","line":127,"range":{"start_line":119,"start_character":0,"end_line":127,"end_character":45},"updated":"2016-03-11 09:19:10.000000000","message":"Here, first create Neutron DB without the lock, and then create DF DB under a lock, They are in different transaction, Can this ensure the two db keep consistent?","commit_id":"c6107ab81f1236622d57271334007d3d533f0aef"},{"author":{"_account_id":7805,"name":"Li Ma","email":"skywalker.nick@gmail.com","username":"Nick"},"change_message_id":"212902f5e646e7f500605dbb3c54dcde3b9db194","unresolved":false,"context_lines":[{"line_number":116,"context_line":""},{"line_number":117,"context_line":"When creating a neutron object::"},{"line_number":118,"context_line":""},{"line_number":119,"context_line":"    def create_object(context, obj):"},{"line_number":120,"context_line":"        with db_api.autonested_transaction(context.session):"},{"line_number":121,"context_line":"            obj_db \u003d super(Plugin, self).create_object(context, obj)"},{"line_number":122,"context_line":"        lock_db.create_lock(context, obj_db[\u0027id\u0027])"},{"line_number":123,"context_line":"        nb_lock \u003d lock_db.DFDBLock(context, obj_db[\u0027id\u0027])"},{"line_number":124,"context_line":"        with nb_lock:"},{"line_number":125,"context_line":"            self.nb_api.create_object(name\u003dobj_db[\u0027id\u0027],"},{"line_number":126,"context_line":"                                      topic\u003dobj_db[\u0027tenant_id\u0027],"},{"line_number":127,"context_line":"                                      obj_db)"},{"line_number":128,"context_line":""},{"line_number":129,"context_line":"When updating a neutron object::"},{"line_number":130,"context_line":""}],"source_content_type":"text/x-rst","patch_set":27,"id":"5aef4532_46aca335","line":127,"range":{"start_line":119,"start_character":0,"end_line":127,"end_character":45},"in_reply_to":"5aef4532_32ca766f","updated":"2016-03-11 09:48:48.000000000","message":"I\u0027m updating this spec now. Thanks for your reminding. I\u0027m testing this implementation.","commit_id":"c6107ab81f1236622d57271334007d3d533f0aef"},{"author":{"_account_id":20229,"name":"Omer Anson","email":"omer.anson@toganetworks.com","username":"oanson"},"change_message_id":"cb9a7d21e4c3580b0259edb1d619acae664d66ea","unresolved":false,"context_lines":[{"line_number":179,"context_line":"atomic operations on a set of read/write operations, we can refactor"},{"line_number":180,"context_line":"the current SQL-based implementation."},{"line_number":181,"context_line":""},{"line_number":182,"context_line":"4. REMOVE Neutron DB. As a result, we don\u0027t need to bother the consistency"},{"line_number":183,"context_line":"of two distinct databases. We only need to make sure a set of read/write"},{"line_number":184,"context_line":"operations of DF DB is atomic to prevent from race due to concurrency."},{"line_number":185,"context_line":"This solution is appealing but not feasible if we cannot solve the"}],"source_content_type":"text/x-rst","patch_set":28,"id":"3afc51ec_0d55684f","line":182,"updated":"2016-03-13 20:24:37.000000000","message":"+1","commit_id":"c22d1d9a50edbda3c114a4c07d4cb05555a3ff5f"},{"author":{"_account_id":18668,"name":"ding bo","email":"longddropt@163.com","username":"longddropt"},"change_message_id":"8352bb87a225a2d1f5e6fc1d76c883bc39bd9fc3","unresolved":false,"context_lines":[{"line_number":100,"context_line":"and commits the transaction. If exception happens, it will re-try it for"},{"line_number":101,"context_line":"several times. If the lock has been obtained, it will wait and re-try."},{"line_number":102,"context_line":""},{"line_number":103,"context_line":"* If MySQL clustering involves, only one lock transaction will be committed"},{"line_number":104,"context_line":"and others will be deadlocked. Here we introduce DB retry mechanism."},{"line_number":105,"context_line":"If deadlock exception happens, it will retry it to make sure it will be"},{"line_number":106,"context_line":"committed later."},{"line_number":107,"context_line":""}],"source_content_type":"text/x-rst","patch_set":30,"id":"fa0719c6_ae2f4ff5","line":104,"range":{"start_line":103,"start_character":2,"end_line":104,"end_character":29},"updated":"2016-03-20 14:48:27.000000000","message":"or all will fail","commit_id":"06828cb600bf89dd2322fe2768d4746666cc7566"},{"author":{"_account_id":18668,"name":"ding bo","email":"longddropt@163.com","username":"longddropt"},"change_message_id":"8352bb87a225a2d1f5e6fc1d76c883bc39bd9fc3","unresolved":false,"context_lines":[{"line_number":105,"context_line":"If deadlock exception happens, it will retry it to make sure it will be"},{"line_number":106,"context_line":"committed later."},{"line_number":107,"context_line":""},{"line_number":108,"context_line":"* Potential Issue: When concurrent write operations on a certain key happen,"},{"line_number":109,"context_line":"due to the inconsistency window of DF DB. If the update on DF DB is always"},{"line_number":110,"context_line":"delayed because the previous operations have already been delayed."},{"line_number":111,"context_line":"The root cause is that Neutron DB is strongly consistent but DF DB is"},{"line_number":112,"context_line":"eventually consistent. We cannot guarantee the updates on DF DB is committed."},{"line_number":113,"context_line":""}],"source_content_type":"text/x-rst","patch_set":30,"id":"fa0719c6_2eadff31","line":110,"range":{"start_line":108,"start_character":18,"end_line":110,"end_character":66},"updated":"2016-03-20 14:48:27.000000000","message":"What does this mean? I think we at least need strong consistency for write operation even sacrificing some availability. So if write operation for DF DB successfully returned, it is guaranteed that the value is will be indeed stored in the DF DB. Moreover, since only one thread could acquire the distributed lock at one time, there is no possible concurrent writing operation will occur.","commit_id":"06828cb600bf89dd2322fe2768d4746666cc7566"}]}