)]}'
{"specs/in_progress/container_sharding.rst":[{"author":{"_account_id":16206,"name":"chenhuayi","email":"chenhuayi@huawei.com","username":"chenhuayi2"},"change_message_id":"214d5aedcde5666f90f3646142948562526291e6","unresolved":false,"context_lines":[{"line_number":71,"context_line":"important, a prefix tree is a good solution, but we need something even simpler"},{"line_number":72,"context_line":"Enter the pivot tree."},{"line_number":73,"context_line":""},{"line_number":74,"context_line":"The basic and simplified idea is rather simple. Firstly, to enable container sharding pass in a \"X-Container-Sharding: On\" X-Header via either PUT or POST::"},{"line_number":75,"context_line":""},{"line_number":76,"context_line":"  curl -i -H \u0027X-Auth-Token: \u003ctoken\u003e\u0027 -H \u0027X-Container-Sharding: On\u0027 \u003curl\u003e/\u003caccount\u003e/\u003ccontainer\u003e -X PUT"},{"line_number":77,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_0bb812c7","line":74,"updated":"2015-08-31 11:18:14.000000000","message":"1 extra\u0027:\u0027\n2 this line should be break.","commit_id":"07fecc4d3de447e152fb74b9f0cb3022ecb362e5"},{"author":{"_account_id":16206,"name":"chenhuayi","email":"chenhuayi@huawei.com","username":"chenhuayi2"},"change_message_id":"214d5aedcde5666f90f3646142948562526291e6","unresolved":false,"context_lines":[{"line_number":75,"context_line":""},{"line_number":76,"context_line":"  curl -i -H \u0027X-Auth-Token: \u003ctoken\u003e\u0027 -H \u0027X-Container-Sharding: On\u0027 \u003curl\u003e/\u003caccount\u003e/\u003ccontainer\u003e -X PUT"},{"line_number":77,"context_line":""},{"line_number":78,"context_line":"Once enabled when a container gets too full, say at 1 million objects. A pivot point is found (the middle item) which will be used to split the container. This will create 2 new containers each holding 1/2 the objects. The original container will now hold a reference to it\u0027s pivot, and not hold objects anymore."},{"line_number":79,"context_line":""},{"line_number":80,"context_line":"All sharded containers exist in it\u0027s own account namespace based off the users account. Meaning they will only"},{"line_number":81,"context_line":"ever see 1 container, this we call the root container. The sharded namespace is::"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_abd006f7","line":78,"updated":"2015-08-31 11:18:14.000000000","message":"this line should be break.","commit_id":"07fecc4d3de447e152fb74b9f0cb3022ecb362e5"},{"author":{"_account_id":16206,"name":"chenhuayi","email":"chenhuayi@huawei.com","username":"chenhuayi2"},"change_message_id":"214d5aedcde5666f90f3646142948562526291e6","unresolved":false,"context_lines":[{"line_number":83,"context_line":"  .sharded_\u003caccount\u003e/"},{"line_number":84,"context_line":""},{"line_number":85,"context_line":"The root container will contain a full listing of all sharded pivot points, allowing quick access to the nodes"},{"line_number":86,"context_line":"container object data (the leaves of the pivot tree). Pivot points are stored in a new table in the container database called pivot_points. Because of the binary nature of this splitting approach, we don\u0027t need to store a list of all sharded containers, instead we only need to pivot points, from there we determine the container names."},{"line_number":87,"context_line":""},{"line_number":88,"context_line":"The following is an over simplistic picture but might help explain, it assumes for simplicity\u0027s sake all split are made on letters of the alphabet, in reality it\u0027ll be split on actual object names:"},{"line_number":89,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_4bd92a21","line":86,"updated":"2015-08-31 11:18:14.000000000","message":"this line should be break.","commit_id":"07fecc4d3de447e152fb74b9f0cb3022ecb362e5"},{"author":{"_account_id":16206,"name":"chenhuayi","email":"chenhuayi@huawei.com","username":"chenhuayi2"},"change_message_id":"214d5aedcde5666f90f3646142948562526291e6","unresolved":false,"context_lines":[{"line_number":85,"context_line":"The root container will contain a full listing of all sharded pivot points, allowing quick access to the nodes"},{"line_number":86,"context_line":"container object data (the leaves of the pivot tree). Pivot points are stored in a new table in the container database called pivot_points. Because of the binary nature of this splitting approach, we don\u0027t need to store a list of all sharded containers, instead we only need to pivot points, from there we determine the container names."},{"line_number":87,"context_line":""},{"line_number":88,"context_line":"The following is an over simplistic picture but might help explain, it assumes for simplicity\u0027s sake all split are made on letters of the alphabet, in reality it\u0027ll be split on actual object names:"},{"line_number":89,"context_line":""},{"line_number":90,"context_line":".. image:: images/PivotPoints.png"},{"line_number":91,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_cbfbfa79","line":88,"updated":"2015-08-31 11:18:14.000000000","message":"this line should be break.","commit_id":"07fecc4d3de447e152fb74b9f0cb3022ecb362e5"},{"author":{"_account_id":16206,"name":"chenhuayi","email":"chenhuayi@huawei.com","username":"chenhuayi2"},"change_message_id":"214d5aedcde5666f90f3646142948562526291e6","unresolved":false,"context_lines":[{"line_number":89,"context_line":""},{"line_number":90,"context_line":".. image:: images/PivotPoints.png"},{"line_number":91,"context_line":""},{"line_number":92,"context_line":"The picture introduces you to what I am calling the pivot weights (-1, +1) these simply indicate different sides of the pivot. More about this later."},{"line_number":93,"context_line":""},{"line_number":94,"context_line":"Each shard will contain some metadata, which allows the sharder and container-server to easily audit and split:"},{"line_number":95,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_abf7a689","line":92,"updated":"2015-08-31 11:18:14.000000000","message":"this line should be break.","commit_id":"07fecc4d3de447e152fb74b9f0cb3022ecb362e5"},{"author":{"_account_id":13997,"name":"SHIGEMATSU Mitsuhiro","email":"shigematsu.mitsuhiro@lab.ntt.co.jp","username":"pshige"},"change_message_id":"e1c1b207aba2031fb811d1c9b5bcab0a9975820e","unresolved":false,"context_lines":[{"line_number":101,"context_line":""},{"line_number":102,"context_line":"Pivot point"},{"line_number":103,"context_line":"--------------"},{"line_number":104,"context_line":"The Pivot point is the middle object in the container. As Swift is eventually consistant all the containers could be in flux and so they will not have the same pivot point to split on. Because of this something needs  we to make the decision. In the initial version of the POC, it will be the container-sharder making the decision. And to do so is rather simple. It will query each primary copy of the container asking for what they think the pivot point is. The sharder will choose the container with the most objects (or if a tie, the first, though maybe it should check to see if more then 1 agree to a pivot point,"},{"line_number":105,"context_line":"but lets keep things as simple as possible, for now)."},{"line_number":106,"context_line":""},{"line_number":107,"context_line":"There is a new method in container/backend.py  called ``get_possible_pivot_point`` which does exactly what you\u0027d expect, finds the pivot point of the container, it does this via querying the database with::"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_48ce152f","line":104,"updated":"2015-08-31 08:39:14.000000000","message":"s/consistant/consistent/","commit_id":"07fecc4d3de447e152fb74b9f0cb3022ecb362e5"},{"author":{"_account_id":16206,"name":"chenhuayi","email":"chenhuayi@huawei.com","username":"chenhuayi2"},"change_message_id":"214d5aedcde5666f90f3646142948562526291e6","unresolved":false,"context_lines":[{"line_number":101,"context_line":""},{"line_number":102,"context_line":"Pivot point"},{"line_number":103,"context_line":"--------------"},{"line_number":104,"context_line":"The Pivot point is the middle object in the container. As Swift is eventually consistant all the containers could be in flux and so they will not have the same pivot point to split on. Because of this something needs  we to make the decision. In the initial version of the POC, it will be the container-sharder making the decision. And to do so is rather simple. It will query each primary copy of the container asking for what they think the pivot point is. The sharder will choose the container with the most objects (or if a tie, the first, though maybe it should check to see if more then 1 agree to a pivot point,"},{"line_number":105,"context_line":"but lets keep things as simple as possible, for now)."},{"line_number":106,"context_line":""},{"line_number":107,"context_line":"There is a new method in container/backend.py  called ``get_possible_pivot_point`` which does exactly what you\u0027d expect, finds the pivot point of the container, it does this via querying the database with::"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_cb321a5e","line":104,"updated":"2015-08-31 11:18:14.000000000","message":"this line should be break.","commit_id":"07fecc4d3de447e152fb74b9f0cb3022ecb362e5"},{"author":{"_account_id":16206,"name":"chenhuayi","email":"chenhuayi@huawei.com","username":"chenhuayi2"},"change_message_id":"214d5aedcde5666f90f3646142948562526291e6","unresolved":false,"context_lines":[{"line_number":104,"context_line":"The Pivot point is the middle object in the container. As Swift is eventually consistant all the containers could be in flux and so they will not have the same pivot point to split on. Because of this something needs  we to make the decision. In the initial version of the POC, it will be the container-sharder making the decision. And to do so is rather simple. It will query each primary copy of the container asking for what they think the pivot point is. The sharder will choose the container with the most objects (or if a tie, the first, though maybe it should check to see if more then 1 agree to a pivot point,"},{"line_number":105,"context_line":"but lets keep things as simple as possible, for now)."},{"line_number":106,"context_line":""},{"line_number":107,"context_line":"There is a new method in container/backend.py  called ``get_possible_pivot_point`` which does exactly what you\u0027d expect, finds the pivot point of the container, it does this via querying the database with::"},{"line_number":108,"context_line":""},{"line_number":109,"context_line":"  SELECT name"},{"line_number":110,"context_line":"  FROM object"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_eb37de4a","line":107,"updated":"2015-08-31 11:18:14.000000000","message":"this line should be break.","commit_id":"07fecc4d3de447e152fb74b9f0cb3022ecb362e5"},{"author":{"_account_id":16206,"name":"chenhuayi","email":"chenhuayi@huawei.com","username":"chenhuayi2"},"change_message_id":"214d5aedcde5666f90f3646142948562526291e6","unresolved":false,"context_lines":[{"line_number":118,"context_line":"-----------------"},{"line_number":119,"context_line":"As mentioned earlier, the pivot weight (-1, +1) represents each side of the pivot."},{"line_number":120,"context_line":""},{"line_number":121,"context_line":"There is a new method in swift/common/utils.py called ``pivot_to_pivot_container`` which takes in the root container account and container along with the pivot point and weight and will generate the shard containers::"},{"line_number":122,"context_line":""},{"line_number":123,"context_line":"  # from swift.common.utils import pivot_to_pivot_container"},{"line_number":124,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_4b47eabb","line":121,"updated":"2015-08-31 11:18:14.000000000","message":"this line should be break.","commit_id":"07fecc4d3de447e152fb74b9f0cb3022ecb362e5"},{"author":{"_account_id":16206,"name":"chenhuayi","email":"chenhuayi@huawei.com","username":"chenhuayi2"},"change_message_id":"214d5aedcde5666f90f3646142948562526291e6","unresolved":false,"context_lines":[{"line_number":146,"context_line":"  pivot: ruby weight: -1"},{"line_number":147,"context_line":"  pivot: ruby weight: 1"},{"line_number":148,"context_line":""},{"line_number":149,"context_line":"Further, once build up, you can use the tree\u0027s get or __get_item__ method to find where an object (doesn\u0027t need to be a pivot point) should live in the tree. It will return the pivot point + weight, which if we want we can place directly into ``pivot_to_pivot_conatiner``, to determine where it sits::"},{"line_number":150,"context_line":""},{"line_number":151,"context_line":"  # leaf, weight \u003d tree[\u0027bacon\u0027]"},{"line_number":152,"context_line":"  # print \u0027%s/%s\u0027 % (pivot_to_pivot_container(\u0027acct\u0027, \u0027cont\u0027, leaf.key, weight))"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_ab256609","line":149,"updated":"2015-08-31 11:18:14.000000000","message":"this line should be break.","commit_id":"07fecc4d3de447e152fb74b9f0cb3022ecb362e5"},{"author":{"_account_id":16206,"name":"chenhuayi","email":"chenhuayi@huawei.com","username":"chenhuayi2"},"change_message_id":"214d5aedcde5666f90f3646142948562526291e6","unresolved":false,"context_lines":[{"line_number":180,"context_line":""},{"line_number":181,"context_line":"Proxy"},{"line_number":182,"context_line":"^^^^^^^^^"},{"line_number":183,"context_line":"As far as the proxy is concerned nothing has changed. An object will always hashed with the root container, so no movement of object data is required."},{"line_number":184,"context_line":""},{"line_number":185,"context_line":"Object-Server and Object-Updater"},{"line_number":186,"context_line":"^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_6b1bceca","line":183,"updated":"2015-08-31 11:18:14.000000000","message":"this line should be break.","commit_id":"07fecc4d3de447e152fb74b9f0cb3022ecb362e5"},{"author":{"_account_id":16206,"name":"chenhuayi","email":"chenhuayi@huawei.com","username":"chenhuayi2"},"change_message_id":"214d5aedcde5666f90f3646142948562526291e6","unresolved":false,"context_lines":[{"line_number":184,"context_line":""},{"line_number":185,"context_line":"Object-Server and Object-Updater"},{"line_number":186,"context_line":"^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"},{"line_number":187,"context_line":"The Object-server and Object-Updater (async-pending\u0027s) need some more smarts because they need to update the correct shard. In the current POC implementation, these daemons don\u0027t actually need to be shard aware, they just be know what to do if a container server responds with a HTTPMovedPermanently (301), as the following picture demonstrates:"},{"line_number":188,"context_line":""},{"line_number":189,"context_line":".. image:: images/seq_obj_put_delete.png"},{"line_number":190,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_2b8076bf","line":187,"updated":"2015-08-31 11:18:14.000000000","message":"this line should be break.","commit_id":"07fecc4d3de447e152fb74b9f0cb3022ecb362e5"},{"author":{"_account_id":16206,"name":"chenhuayi","email":"chenhuayi@huawei.com","username":"chenhuayi2"},"change_message_id":"214d5aedcde5666f90f3646142948562526291e6","unresolved":false,"context_lines":[{"line_number":188,"context_line":""},{"line_number":189,"context_line":".. image:: images/seq_obj_put_delete.png"},{"line_number":190,"context_line":""},{"line_number":191,"context_line":"They accomplish this by getting the container-server to set the required X-Container-{Host, Device, Partition} headers in the response that the object-{server, updater} require to redirect it\u0027s update."},{"line_number":192,"context_line":""},{"line_number":193,"context_line":"In the case of the Object-Updater, a change to updating algorithm is required in order to stop excessive requests when dealing with sharded containers. Currently, the updater will send replica count updates to the container servers concurrently. Now that it follows 301\u0027s this causes a problem. When dealing with a sharded container, the request will always attempt to update the root container first, which will result in a 301. With the current state of the algorithm it will potentially result in a replica count of 301s, even though we redirect and fire replica count more to the correct container after receiving the first 301. This way we\u0027re adding an additional 2 requests for every async pending update to update sharded containers. To correct this, the updater now fires 1 request and waits (timeout) for the response. If the response isn\u0027t a 301 then it\u0027ll fire the rest (replica count -1)."},{"line_number":194,"context_line":"If the first request is a 400 or 500 series, but still supposed to be 301, then there may be some additional requests, but this, I think, is a good compromise."}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_4b75aaf7","line":191,"updated":"2015-08-31 11:18:14.000000000","message":"this line should be break.","commit_id":"07fecc4d3de447e152fb74b9f0cb3022ecb362e5"},{"author":{"_account_id":16206,"name":"chenhuayi","email":"chenhuayi@huawei.com","username":"chenhuayi2"},"change_message_id":"214d5aedcde5666f90f3646142948562526291e6","unresolved":false,"context_lines":[{"line_number":191,"context_line":"They accomplish this by getting the container-server to set the required X-Container-{Host, Device, Partition} headers in the response that the object-{server, updater} require to redirect it\u0027s update."},{"line_number":192,"context_line":""},{"line_number":193,"context_line":"In the case of the Object-Updater, a change to updating algorithm is required in order to stop excessive requests when dealing with sharded containers. Currently, the updater will send replica count updates to the container servers concurrently. Now that it follows 301\u0027s this causes a problem. When dealing with a sharded container, the request will always attempt to update the root container first, which will result in a 301. With the current state of the algorithm it will potentially result in a replica count of 301s, even though we redirect and fire replica count more to the correct container after receiving the first 301. This way we\u0027re adding an additional 2 requests for every async pending update to update sharded containers. To correct this, the updater now fires 1 request and waits (timeout) for the response. If the response isn\u0027t a 301 then it\u0027ll fire the rest (replica count -1)."},{"line_number":194,"context_line":"If the first request is a 400 or 500 series, but still supposed to be 301, then there may be some additional requests, but this, I think, is a good compromise."},{"line_number":195,"context_line":""},{"line_number":196,"context_line":"Effects to the container path"},{"line_number":197,"context_line":"---------------------------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_6b72ee12","line":194,"updated":"2015-08-31 11:18:14.000000000","message":"this line should be break.","commit_id":"07fecc4d3de447e152fb74b9f0cb3022ecb362e5"},{"author":{"_account_id":16206,"name":"chenhuayi","email":"chenhuayi@huawei.com","username":"chenhuayi2"},"change_message_id":"214d5aedcde5666f90f3646142948562526291e6","unresolved":false,"context_lines":[{"line_number":202,"context_line":""},{"line_number":203,"context_line":"GET/HEAD"},{"line_number":204,"context_line":"^^^^^^^^^"},{"line_number":205,"context_line":"Both GET and HEADs get much more complicated. On HEAD each leaf needs to be visited in order to get the correct object count (and bytes used). It\u0027s either that or we need to make sure the root container\u0027s object count is updated whenever an object it added."},{"line_number":206,"context_line":""},{"line_number":207,"context_line":"On GETs, additional requests will be required to hit leaf nodes to gather and build the object listings."},{"line_number":208,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_cb577a58","line":205,"updated":"2015-08-31 11:18:14.000000000","message":"this line should be break.","commit_id":"07fecc4d3de447e152fb74b9f0cb3022ecb362e5"},{"author":{"_account_id":16206,"name":"chenhuayi","email":"chenhuayi@huawei.com","username":"chenhuayi2"},"change_message_id":"214d5aedcde5666f90f3646142948562526291e6","unresolved":false,"context_lines":[{"line_number":210,"context_line":""},{"line_number":211,"context_line":"In the proxy:"},{"line_number":212,"context_line":"* Pro: this has the advantage of making new requests from the proxy, being able to probe and build a response."},{"line_number":213,"context_line":"* Con: The pivot tree of the root container needs to be passed back to the proxy. Because this tree could grow, even if only passing back the leaves, passing a tree back would have to be in a body (as the last POC using distributed prefix trees implemented) or in multiple headers."},{"line_number":214,"context_line":""},{"line_number":215,"context_line":"In the container-server:"},{"line_number":216,"context_line":"* Pro: No need to pass the root containers pivot nodes (or just leaves) as all the action will happen with access to the root containers broker."}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_6b498eb6","line":213,"updated":"2015-08-31 11:18:14.000000000","message":"this line should be break.","commit_id":"07fecc4d3de447e152fb74b9f0cb3022ecb362e5"},{"author":{"_account_id":16206,"name":"chenhuayi","email":"chenhuayi@huawei.com","username":"chenhuayi2"},"change_message_id":"214d5aedcde5666f90f3646142948562526291e6","unresolved":false,"context_lines":[{"line_number":214,"context_line":""},{"line_number":215,"context_line":"In the container-server:"},{"line_number":216,"context_line":"* Pro: No need to pass the root containers pivot nodes (or just leaves) as all the action will happen with access to the root containers broker."},{"line_number":217,"context_line":"* Con: Because it\u0027s happening on the container-servers, care needs to be given in regard to requests. Is one of the other primaries also generating a list, when the container-server calls other leaf nodes, are we sending out replica count more request (potentially from each primary node). Maybe a container server needs to only attempt to contact a random primary node at a time to reduce requests, or even GET/HEAD from the proxy with 100-continue, and get the proxy to pick only 1 server to continue on."},{"line_number":218,"context_line":""},{"line_number":219,"context_line":"DELETE"},{"line_number":220,"context_line":"^^^^^^^"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_cbce9a2c","line":217,"updated":"2015-08-31 11:18:14.000000000","message":"this line should be break.","commit_id":"07fecc4d3de447e152fb74b9f0cb3022ecb362e5"},{"author":{"_account_id":16206,"name":"chenhuayi","email":"chenhuayi@huawei.com","username":"chenhuayi2"},"change_message_id":"214d5aedcde5666f90f3646142948562526291e6","unresolved":false,"context_lines":[{"line_number":228,"context_line":"Container server and transport changes"},{"line_number":229,"context_line":"-------------------------------------------"},{"line_number":230,"context_line":""},{"line_number":231,"context_line":"If GET, HEAD and DELETEs are managed in the proxy (still undecided) then there needs to be a way to pass back the pivot tree to the proxy, or at least leaf nodes. There is a limit to the size of a header passed back, so it\u0027ll either need to pass it back in the body or in a bunch of headers.  "},{"line_number":232,"context_line":""},{"line_number":233,"context_line":"Replicater changes"},{"line_number":234,"context_line":"--------------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_6be0ae96","line":231,"updated":"2015-08-31 11:18:14.000000000","message":"this line should be break.","commit_id":"07fecc4d3de447e152fb74b9f0cb3022ecb362e5"},{"author":{"_account_id":13997,"name":"SHIGEMATSU Mitsuhiro","email":"shigematsu.mitsuhiro@lab.ntt.co.jp","username":"pshige"},"change_message_id":"e1c1b207aba2031fb811d1c9b5bcab0a9975820e","unresolved":false,"context_lines":[{"line_number":228,"context_line":"Container server and transport changes"},{"line_number":229,"context_line":"-------------------------------------------"},{"line_number":230,"context_line":""},{"line_number":231,"context_line":"If GET, HEAD and DELETEs are managed in the proxy (still undecided) then there needs to be a way to pass back the pivot tree to the proxy, or at least leaf nodes. There is a limit to the size of a header passed back, so it\u0027ll either need to pass it back in the body or in a bunch of headers.  "},{"line_number":232,"context_line":""},{"line_number":233,"context_line":"Replicater changes"},{"line_number":234,"context_line":"--------------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_88f74d58","line":231,"updated":"2015-08-31 08:39:14.000000000","message":"trailing white spaces","commit_id":"07fecc4d3de447e152fb74b9f0cb3022ecb362e5"},{"author":{"_account_id":13997,"name":"SHIGEMATSU Mitsuhiro","email":"shigematsu.mitsuhiro@lab.ntt.co.jp","username":"pshige"},"change_message_id":"e1c1b207aba2031fb811d1c9b5bcab0a9975820e","unresolved":false,"context_lines":[{"line_number":230,"context_line":""},{"line_number":231,"context_line":"If GET, HEAD and DELETEs are managed in the proxy (still undecided) then there needs to be a way to pass back the pivot tree to the proxy, or at least leaf nodes. There is a limit to the size of a header passed back, so it\u0027ll either need to pass it back in the body or in a bunch of headers.  "},{"line_number":232,"context_line":""},{"line_number":233,"context_line":"Replicater changes"},{"line_number":234,"context_line":"--------------------"},{"line_number":235,"context_line":"The container-replicator (and db_replicator as required) will need to be updated to replicate and sync the pivot_points table."},{"line_number":236,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_a8049149","line":233,"updated":"2015-08-31 08:39:14.000000000","message":"s/Replicater/Replicator/","commit_id":"07fecc4d3de447e152fb74b9f0cb3022ecb362e5"},{"author":{"_account_id":16206,"name":"chenhuayi","email":"chenhuayi@huawei.com","username":"chenhuayi2"},"change_message_id":"214d5aedcde5666f90f3646142948562526291e6","unresolved":false,"context_lines":[{"line_number":234,"context_line":"--------------------"},{"line_number":235,"context_line":"The container-replicator (and db_replicator as required) will need to be updated to replicate and sync the pivot_points table."},{"line_number":236,"context_line":""},{"line_number":237,"context_line":"Swift is eventually consistent, meaning at some point we will have an unsharded version of a container replicate with a sharded one, and being eventually consistent, some of the objects in the unsharded might actually exist and need to be merged into a lower down shard. The current thinking is that a sharded container holds all it\u0027s objects into the leaves. Leaving the root and branch containers object table empty, these non-leaves also will not be queried when object listing. So the plan is:"},{"line_number":238,"context_line":""},{"line_number":239,"context_line":"#. Sync the objects from the unsharded container into the objects table of the root/branch container."},{"line_number":240,"context_line":"#. Let the conatiner-sharder replicate the objects down to the correct shard. (noting that dealing with misplaced objects in a shard is apart of the sharder\u0027s job)"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_2bf5d655","line":237,"updated":"2015-08-31 11:18:14.000000000","message":"this line should be break.","commit_id":"07fecc4d3de447e152fb74b9f0cb3022ecb362e5"},{"author":{"_account_id":16206,"name":"chenhuayi","email":"chenhuayi@huawei.com","username":"chenhuayi2"},"change_message_id":"214d5aedcde5666f90f3646142948562526291e6","unresolved":false,"context_lines":[{"line_number":237,"context_line":"Swift is eventually consistent, meaning at some point we will have an unsharded version of a container replicate with a sharded one, and being eventually consistent, some of the objects in the unsharded might actually exist and need to be merged into a lower down shard. The current thinking is that a sharded container holds all it\u0027s objects into the leaves. Leaving the root and branch containers object table empty, these non-leaves also will not be queried when object listing. So the plan is:"},{"line_number":238,"context_line":""},{"line_number":239,"context_line":"#. Sync the objects from the unsharded container into the objects table of the root/branch container."},{"line_number":240,"context_line":"#. Let the conatiner-sharder replicate the objects down to the correct shard. (noting that dealing with misplaced objects in a shard is apart of the sharder\u0027s job)"},{"line_number":241,"context_line":""},{"line_number":242,"context_line":"pending and merge_items"},{"line_number":243,"context_line":"^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_cbe53aa4","line":240,"updated":"2015-08-31 11:18:14.000000000","message":"this line should be break.","commit_id":"07fecc4d3de447e152fb74b9f0cb3022ecb362e5"},{"author":{"_account_id":16206,"name":"chenhuayi","email":"chenhuayi@huawei.com","username":"chenhuayi2"},"change_message_id":"214d5aedcde5666f90f3646142948562526291e6","unresolved":false,"context_lines":[{"line_number":242,"context_line":"pending and merge_items"},{"line_number":243,"context_line":"^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"},{"line_number":244,"context_line":"This version of the POC will take advantage of the last POC\u0027s changes to replication. They will at least suffice while it\u0027s a POC."},{"line_number":245,"context_line":"The merge_items method in the container/backend.py will be modified to be pivot_points aware. That is to say, the list of items passed to it can now contain a mix of objects and pivot_nodes. A new flag will be added to the pending/pickle file format called record_type, which defaults to RECORD_TYPE_OBJECT in existing pickle/pending files when unpickled. Merge_items will sort into 2 different lists based on the record_type, then insert, update, delete the required tables accordingly."},{"line_number":246,"context_line":""},{"line_number":247,"context_line":"Container replication changes"},{"line_number":248,"context_line":"^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_0bbed264","line":245,"updated":"2015-08-31 11:18:14.000000000","message":"this line should be break.","commit_id":"07fecc4d3de447e152fb74b9f0cb3022ecb362e5"},{"author":{"_account_id":16206,"name":"chenhuayi","email":"chenhuayi@huawei.com","username":"chenhuayi2"},"change_message_id":"214d5aedcde5666f90f3646142948562526291e6","unresolved":false,"context_lines":[{"line_number":246,"context_line":""},{"line_number":247,"context_line":"Container replication changes"},{"line_number":248,"context_line":"^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"},{"line_number":249,"context_line":"Because swift is an eventually consistent system, we need to make sure that when container databases are replicated, this doesn\u0027t only replicate items in the objects table, but also the nodes in the pivot_points table. Most of the database replication code is apart of the db_replicator which is a parent and so shared by account and container replication. The current solution in the POC, is to add an _other_items_hook(broker) hook that is over-written in the container replicator to grab the items from the pivot_points table and returned in the items format to be passed into merge_items."},{"line_number":250,"context_line":""},{"line_number":251,"context_line":"There is a caveat however, which is that currently the hook grabs all the objects from the pivot_points table. There is no idea of pointer/sync point. The number of pivot_point should remain fairly small, at least in relation to objects."},{"line_number":252,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_ebb83e77","line":249,"updated":"2015-08-31 11:18:14.000000000","message":"this line should be break.","commit_id":"07fecc4d3de447e152fb74b9f0cb3022ecb362e5"},{"author":{"_account_id":16206,"name":"chenhuayi","email":"chenhuayi@huawei.com","username":"chenhuayi2"},"change_message_id":"214d5aedcde5666f90f3646142948562526291e6","unresolved":false,"context_lines":[{"line_number":248,"context_line":"^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"},{"line_number":249,"context_line":"Because swift is an eventually consistent system, we need to make sure that when container databases are replicated, this doesn\u0027t only replicate items in the objects table, but also the nodes in the pivot_points table. Most of the database replication code is apart of the db_replicator which is a parent and so shared by account and container replication. The current solution in the POC, is to add an _other_items_hook(broker) hook that is over-written in the container replicator to grab the items from the pivot_points table and returned in the items format to be passed into merge_items."},{"line_number":250,"context_line":""},{"line_number":251,"context_line":"There is a caveat however, which is that currently the hook grabs all the objects from the pivot_points table. There is no idea of pointer/sync point. The number of pivot_point should remain fairly small, at least in relation to objects."},{"line_number":252,"context_line":""},{"line_number":253,"context_line":"Container-sharder"},{"line_number":254,"context_line":"-------------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_0b65721c","line":251,"updated":"2015-08-31 11:18:14.000000000","message":"this line should be break.","commit_id":"07fecc4d3de447e152fb74b9f0cb3022ecb362e5"},{"author":{"_account_id":16206,"name":"chenhuayi","email":"chenhuayi@huawei.com","username":"chenhuayi2"},"change_message_id":"214d5aedcde5666f90f3646142948562526291e6","unresolved":false,"context_lines":[{"line_number":252,"context_line":""},{"line_number":253,"context_line":"Container-sharder"},{"line_number":254,"context_line":"-------------------"},{"line_number":255,"context_line":"The container-sharder, will run on all container-server nodes. At an interval it will parse all shared containers, run a quick audit, deal with misplaced objects and then see if the container needs to be sharded on a pivot point."},{"line_number":256,"context_line":""},{"line_number":257,"context_line":"Audit"},{"line_number":258,"context_line":"^^^^^^"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_cb6a1aed","line":255,"updated":"2015-08-31 11:18:14.000000000","message":"this line should be break.","commit_id":"07fecc4d3de447e152fb74b9f0cb3022ecb362e5"},{"author":{"_account_id":16206,"name":"chenhuayi","email":"chenhuayi@huawei.com","username":"chenhuayi2"},"change_message_id":"214d5aedcde5666f90f3646142948562526291e6","unresolved":false,"context_lines":[{"line_number":256,"context_line":""},{"line_number":257,"context_line":"Audit"},{"line_number":258,"context_line":"^^^^^^"},{"line_number":259,"context_line":"The sharder will perform a basic audit of which simply makes sure the current shard\u0027s pivot exists in it\u0027s parent container and in the root\u0027s. The truth table would look something like:"},{"line_number":260,"context_line":""},{"line_number":261,"context_line":"  +----------+------------+------------------------+"},{"line_number":262,"context_line":"  | root ref | parent ref | Outcome                |"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_8b7982ca","line":259,"updated":"2015-08-31 11:18:14.000000000","message":"this line should be break.","commit_id":"07fecc4d3de447e152fb74b9f0cb3022ecb362e5"},{"author":{"_account_id":16206,"name":"chenhuayi","email":"chenhuayi@huawei.com","username":"chenhuayi2"},"change_message_id":"214d5aedcde5666f90f3646142948562526291e6","unresolved":false,"context_lines":[{"line_number":272,"context_line":""},{"line_number":273,"context_line":"Pivot point determination and splitting"},{"line_number":274,"context_line":"^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"},{"line_number":275,"context_line":"The current thought it is for the sharder to ask each primary container node what it thinks the best pivot point is. Then make the decision and tell all nodes what to pivot on. How the sharder will tell the container node to pivot or how it will hasn\u0027t been fully thought out yet. But maybe:"},{"line_number":276,"context_line":""},{"line_number":277,"context_line":" * Using the REPLICATE verb, but with a different action?"},{"line_number":278,"context_line":" * A special request?"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_6b7c2eb5","line":275,"updated":"2015-08-31 11:18:14.000000000","message":"this line should be break.","commit_id":"07fecc4d3de447e152fb74b9f0cb3022ecb362e5"},{"author":{"_account_id":16206,"name":"chenhuayi","email":"chenhuayi@huawei.com","username":"chenhuayi2"},"change_message_id":"214d5aedcde5666f90f3646142948562526291e6","unresolved":false,"context_lines":[{"line_number":279,"context_line":""},{"line_number":280,"context_line":"Misplaced objects"},{"line_number":281,"context_line":"^^^^^^^^^^^^^^^^^^^"},{"line_number":282,"context_line":"A misplaced object is an object that is in the wrong shard. If it\u0027s a branch shard, then anything in the object table is misplaced and needs to be dealt with. On a leaf node, a quick SQL statement should be enough to find out all the objects that are on the wrong side of the pivot."},{"line_number":283,"context_line":""},{"line_number":284,"context_line":"The sharder uses the container-reconciler/replicator\u0027s approach of creating a container database locally in a handoff partition, loading it up, and then using replication to push it over to where it needs to go."},{"line_number":285,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_2b515636","line":282,"updated":"2015-08-31 11:18:14.000000000","message":"this line should be break.","commit_id":"07fecc4d3de447e152fb74b9f0cb3022ecb362e5"},{"author":{"_account_id":16206,"name":"chenhuayi","email":"chenhuayi@huawei.com","username":"chenhuayi2"},"change_message_id":"214d5aedcde5666f90f3646142948562526291e6","unresolved":false,"context_lines":[{"line_number":281,"context_line":"^^^^^^^^^^^^^^^^^^^"},{"line_number":282,"context_line":"A misplaced object is an object that is in the wrong shard. If it\u0027s a branch shard, then anything in the object table is misplaced and needs to be dealt with. On a leaf node, a quick SQL statement should be enough to find out all the objects that are on the wrong side of the pivot."},{"line_number":283,"context_line":""},{"line_number":284,"context_line":"The sharder uses the container-reconciler/replicator\u0027s approach of creating a container database locally in a handoff partition, loading it up, and then using replication to push it over to where it needs to go."},{"line_number":285,"context_line":""},{"line_number":286,"context_line":"Assignee(s)"},{"line_number":287,"context_line":"-----------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_eb46fe79","line":284,"updated":"2015-08-31 11:18:14.000000000","message":"this line should be break.","commit_id":"07fecc4d3de447e152fb74b9f0cb3022ecb362e5"},{"author":{"_account_id":11317,"name":"Eran Rom","email":"eran@itsonlyme.name","username":"eranr"},"change_message_id":"76b008bd0d7ebff9d9232be927da6312d13d419f","unresolved":false,"context_lines":[{"line_number":76,"context_line":""},{"line_number":77,"context_line":"  curl -i -H \u0027X-Auth-Token: \u003ctoken\u003e\u0027 -H \u0027X-Container-Sharding: On\u0027 \u003curl\u003e/\u003caccount\u003e/\u003ccontainer\u003e -X PUT"},{"line_number":78,"context_line":""},{"line_number":79,"context_line":"Once enabled when a container gets too full, say at 1 million objects. A pivot point is found"},{"line_number":80,"context_line":"(the middle item) which will be used to split the container. This will create 2 new containers each"},{"line_number":81,"context_line":"holding 1/2 the objects. The original container will now hold a reference to it\u0027s pivot, and not hold"},{"line_number":82,"context_line":"objects anymore."}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a2fa921_402fdb87","line":79,"updated":"2015-10-11 12:01:57.000000000","message":"Should 1 million be a container server config option?","commit_id":"cd15a3729068a2e77c52adbf65635a3bed03cf1f"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"9c3fb86f7de5dbd1235e054025eb753c69a0c8b6","unresolved":false,"context_lines":[{"line_number":76,"context_line":""},{"line_number":77,"context_line":"  curl -i -H \u0027X-Auth-Token: \u003ctoken\u003e\u0027 -H \u0027X-Container-Sharding: On\u0027 \u003curl\u003e/\u003caccount\u003e/\u003ccontainer\u003e -X PUT"},{"line_number":78,"context_line":""},{"line_number":79,"context_line":"Once enabled when a container gets too full, say at 1 million objects. A pivot point is found"},{"line_number":80,"context_line":"(the middle item) which will be used to split the container. This will create 2 new containers each"},{"line_number":81,"context_line":"holding 1/2 the objects. The original container will now hold a reference to it\u0027s pivot, and not hold"},{"line_number":82,"context_line":"objects anymore."}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a2fa921_fc15219b","line":79,"in_reply_to":"7a2fa921_402fdb87","updated":"2015-10-12 00:49:31.000000000","message":"It is. this first version of the spec is just the idea not a full technical spec. I will update the spec with that I have before summit. But yes it is and will be config configurable in container-server.config","commit_id":"cd15a3729068a2e77c52adbf65635a3bed03cf1f"},{"author":{"_account_id":11317,"name":"Eran Rom","email":"eran@itsonlyme.name","username":"eranr"},"change_message_id":"76b008bd0d7ebff9d9232be927da6312d13d419f","unresolved":false,"context_lines":[{"line_number":79,"context_line":"Once enabled when a container gets too full, say at 1 million objects. A pivot point is found"},{"line_number":80,"context_line":"(the middle item) which will be used to split the container. This will create 2 new containers each"},{"line_number":81,"context_line":"holding 1/2 the objects. The original container will now hold a reference to it\u0027s pivot, and not hold"},{"line_number":82,"context_line":"objects anymore."},{"line_number":83,"context_line":""},{"line_number":84,"context_line":"All sharded containers exist in it\u0027s own account namespace based off the users account. Meaning they will only"},{"line_number":85,"context_line":"ever see 1 container, this we call the root container. The sharded namespace is::"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a2fa921_a0790f84","line":82,"updated":"2015-10-11 12:01:57.000000000","message":"I think that at this point it would be good to explain the pivot concept in more details. It took me a while to understand this from the spec and drawing. Here is my understanding (you may want to consider using it in some way):\nOnce a container is split, two additional containers are created, each of which holding half of the objects so that:\n\n(1) The new containers\u0027 URI is not accessible to the end user (details below)\n\n(2) The object table of the original container is replaced with a pivot table.\n\nThe pivot table has an entry for each split that the container and any of its shards has gone through. Each entry in the pivot table is called a pivot point.\nAs shown next, the information in the pivot table is sufficient for finding the exact shard where an object actually resides.\n\nTo better understand the pivot table consider the following example:\n\nROWID  name\n\n0               L\n\n1               F\n\n2               R\n\nThis table tells us that the original container has gone 3 splits:\nIt was first split into containers holding all objects with names up to \u0027L\u0027 and objects with names from \u0027L\u0027.\nThe container holding all objects with names up to \u0027L\u0027 was then split around \u0027F\u0027, and finally, the hidden container holding all objects with names from \u0027L\u0027 was split around \u0027R\u0027\n\nAll containers that are shards will have a .sharder_\u003caccount name\u003e prefix. Assume that the container whose pivot table above is in account \u0027a\u0027 and is called \u0027c\u0027.\nDuring the first split, the following two containers were created:\n\n.sharded_a/c\u003c-L\n\n.sharded_a/c-\u003eL\n\nDuring the second split (of .sharded_a/c\u003c-L) we got:\n\n.sharded_a/c\u003c-L\u003c-F\n\n.sharded_a/c\u003c-L-\u003eF\n\nand during the third split (of .sharded_a/c-\u003eL) we got:\n\n.sharded_a/c-\u003eL\u003c-R\n\n.sharded_a/c-\u003eL-\u003eR\n\nAs explained in the object path section below the path of the form .sharded_a/c-\u003eL-\u003eR is to be used by the proxy as container URI\u0027s when e.g. the object server  needs to access a shard during a PUT/ DELETE request.","commit_id":"cd15a3729068a2e77c52adbf65635a3bed03cf1f"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"9c3fb86f7de5dbd1235e054025eb753c69a0c8b6","unresolved":false,"context_lines":[{"line_number":79,"context_line":"Once enabled when a container gets too full, say at 1 million objects. A pivot point is found"},{"line_number":80,"context_line":"(the middle item) which will be used to split the container. This will create 2 new containers each"},{"line_number":81,"context_line":"holding 1/2 the objects. The original container will now hold a reference to it\u0027s pivot, and not hold"},{"line_number":82,"context_line":"objects anymore."},{"line_number":83,"context_line":""},{"line_number":84,"context_line":"All sharded containers exist in it\u0027s own account namespace based off the users account. Meaning they will only"},{"line_number":85,"context_line":"ever see 1 container, this we call the root container. The sharded namespace is::"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a2fa921_9c0aed78","line":82,"in_reply_to":"7a2fa921_a0790f84","updated":"2015-10-12 00:49:31.000000000","message":"Nice, I\u0027ll definitely add a section here to describe it better. As I\u0027ve moved along building the pivot tree POC things have changed a little, but not much. Thanks","commit_id":"cd15a3729068a2e77c52adbf65635a3bed03cf1f"},{"author":{"_account_id":11317,"name":"Eran Rom","email":"eran@itsonlyme.name","username":"eranr"},"change_message_id":"76b008bd0d7ebff9d9232be927da6312d13d419f","unresolved":false,"context_lines":[{"line_number":103,"context_line":""},{"line_number":104,"context_line":"* X-Backend-Shard-Account"},{"line_number":105,"context_line":"* X-Backend-Shard-Container"},{"line_number":106,"context_line":"* X-Backend-Shard-Pivot"},{"line_number":107,"context_line":""},{"line_number":108,"context_line":"NOTE: This is the only metadata stored as the root container is only container that holds the users metadata."},{"line_number":109,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a2fa921_96808ff8","line":106,"updated":"2015-10-11 12:01:57.000000000","message":"I guess that this is clear for seasoned container sharding readers. I could use a little elaboration on the essence of the above per shard metadata.","commit_id":"cd15a3729068a2e77c52adbf65635a3bed03cf1f"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"9c3fb86f7de5dbd1235e054025eb753c69a0c8b6","unresolved":false,"context_lines":[{"line_number":103,"context_line":""},{"line_number":104,"context_line":"* X-Backend-Shard-Account"},{"line_number":105,"context_line":"* X-Backend-Shard-Container"},{"line_number":106,"context_line":"* X-Backend-Shard-Pivot"},{"line_number":107,"context_line":""},{"line_number":108,"context_line":"NOTE: This is the only metadata stored as the root container is only container that holds the users metadata."},{"line_number":109,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a2fa921_bc0fa987","line":106,"in_reply_to":"7a2fa921_96808ff8","updated":"2015-10-12 00:49:31.000000000","message":"Will do, thanks :)\n\nI\u0027ve been at this a while, so thanks for pointing out where I\u0027m assuming too much knowledge.","commit_id":"cd15a3729068a2e77c52adbf65635a3bed03cf1f"},{"author":{"_account_id":11317,"name":"Eran Rom","email":"eran@itsonlyme.name","username":"eranr"},"change_message_id":"76b008bd0d7ebff9d9232be927da6312d13d419f","unresolved":false,"context_lines":[{"line_number":109,"context_line":""},{"line_number":110,"context_line":"Pivot point"},{"line_number":111,"context_line":"--------------"},{"line_number":112,"context_line":"The Pivot point is the middle object in the container. As Swift is eventually consistent all the containers"},{"line_number":113,"context_line":"could be in flux and so they will not have the same pivot point to split on. Because of this something needs"},{"line_number":114,"context_line":"we to make the decision. In the initial version of the POC, it will be the container-sharder making the decision."},{"line_number":115,"context_line":"And to do so is rather simple. It will query each primary copy of the container asking for what they think the"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a2fa921_d6912748","line":112,"updated":"2015-10-11 12:01:57.000000000","message":"Perhaps this does not matter, but would the middle object go to the \u0027\u003c-\u0027 or to the \u0027-\u003e\u0027 side?","commit_id":"cd15a3729068a2e77c52adbf65635a3bed03cf1f"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"9c3fb86f7de5dbd1235e054025eb753c69a0c8b6","unresolved":false,"context_lines":[{"line_number":109,"context_line":""},{"line_number":110,"context_line":"Pivot point"},{"line_number":111,"context_line":"--------------"},{"line_number":112,"context_line":"The Pivot point is the middle object in the container. As Swift is eventually consistent all the containers"},{"line_number":113,"context_line":"could be in flux and so they will not have the same pivot point to split on. Because of this something needs"},{"line_number":114,"context_line":"we to make the decision. In the initial version of the POC, it will be the container-sharder making the decision."},{"line_number":115,"context_line":"And to do so is rather simple. It will query each primary copy of the container asking for what they think the"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a2fa921_fc7c41ed","line":112,"in_reply_to":"7a2fa921_d6912748","updated":"2015-10-12 00:49:31.000000000","message":"yeah, I didn\u0027t go into where the middle object goes, cause that will be decided by the implementation. In the POC I chose the left side. I also changed the \u0027\u003c-\u0027 and \u0027-\u003e\u0027 to \u0027.le.\u0027 and \u0027.gt.\u0027 as the arrows in the log get url-encoded and this makes it harder to read:\n\n  .le. - less than or equal\n  .gt. - greater than.\n\nThis will be described in the updated version of the spec. And obviously I\u0027m ok with that being changed. If this POC (pivot based sharding) goes well and turns into the approach we take and thus becoming a feature branch. I\u0027m happy for these to be changed based on others better tagging iteas :)","commit_id":"cd15a3729068a2e77c52adbf65635a3bed03cf1f"},{"author":{"_account_id":11317,"name":"Eran Rom","email":"eran@itsonlyme.name","username":"eranr"},"change_message_id":"76b008bd0d7ebff9d9232be927da6312d13d419f","unresolved":false,"context_lines":[{"line_number":116,"context_line":"pivot point is. The sharder will choose the container with the most objects (or if a tie, the first,"},{"line_number":117,"context_line":"though maybe it should check to see if more then 1 agree to a pivot point, but lets keep things as simple as"},{"line_number":118,"context_line":"possible, for now)."},{"line_number":119,"context_line":""},{"line_number":120,"context_line":"There is a new method in container/backend.py  called ``get_possible_pivot_point`` which does exactly what"},{"line_number":121,"context_line":"you\u0027d expect, finds the pivot point of the container, it does this via querying the database with::"},{"line_number":122,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a2fa921_16da3f17","line":119,"updated":"2015-10-11 12:01:57.000000000","message":"It was hard for me to compile the above sentence. Can you pls confirm the following understanding:\nThe pivot point is the object that appears in the middle of the container listing. As Swift is eventual consistent different container replicas may have different opinions as to where the pivot point is. In the initial version of the POC, the container sharder will make the decision as to where the pivot point is. This is done by picking the container replica with most objects. \n\nRegarding the tie breaking: you may have two replicas that have the same number of objects but still do not agree on the pivot point. So to make things simple I would suggest to pick one randomly.","commit_id":"cd15a3729068a2e77c52adbf65635a3bed03cf1f"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"9c3fb86f7de5dbd1235e054025eb753c69a0c8b6","unresolved":false,"context_lines":[{"line_number":116,"context_line":"pivot point is. The sharder will choose the container with the most objects (or if a tie, the first,"},{"line_number":117,"context_line":"though maybe it should check to see if more then 1 agree to a pivot point, but lets keep things as simple as"},{"line_number":118,"context_line":"possible, for now)."},{"line_number":119,"context_line":""},{"line_number":120,"context_line":"There is a new method in container/backend.py  called ``get_possible_pivot_point`` which does exactly what"},{"line_number":121,"context_line":"you\u0027d expect, finds the pivot point of the container, it does this via querying the database with::"},{"line_number":122,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a2fa921_bc15694a","line":119,"in_reply_to":"7a2fa921_16da3f17","updated":"2015-10-12 00:49:31.000000000","message":"Yeah, you got it. I\u0027ve gone one step further in the POC so far. (I\u0027ve moved forward a little so I can start benchmarking to see how well it goes).\n\nThe sharder runs on the container servers. It iterates through all sharded containers, audits them, deals with misplaced objects (thise replicated to it that should be there, etc).\n\nThe actual sharding in the sharder is a 2 phase process atm:\n1a:Ask all primary nodes what they think the pivot point should be, the one with the most items wins. If more then 1 primary node has the same number of objects then the first one queried wins (node the nodes are randomised before access, so yeah its random). NOTE we also need to get a quorum of responses (replica count / 2 + 1).  \n\n1b: Once we have a pivot point, set sysmeta on all primary nodes of what the pivot point is. Then move to the next container.\nNOTE: We don\u0027t find a pivot point and shard on in the one pass.\n\n2. On the next sharder pass (that is to say when the sharder finds the sysmeta set locallay) it will make sure it still has quorum with the primary nodes that this is correct, if it is it then shards on that point locally and then replicates.\n\nAgain I\u0027ll explain this better in the next version.","commit_id":"cd15a3729068a2e77c52adbf65635a3bed03cf1f"},{"author":{"_account_id":11317,"name":"Eran Rom","email":"eran@itsonlyme.name","username":"eranr"},"change_message_id":"76b008bd0d7ebff9d9232be927da6312d13d419f","unresolved":false,"context_lines":[{"line_number":144,"context_line":"  Account: .sharded_a Shard_cont(\u003c): c\u003c-swift_is_cool Shard_cont(\u003e): c-\u003eswift_is_cool"},{"line_number":145,"context_line":""},{"line_number":146,"context_line":"PivotTree Class"},{"line_number":147,"context_line":"-----------------"},{"line_number":148,"context_line":"This class once loaded up with pivot points, makes it easy iterate through all leaf nodes::"},{"line_number":149,"context_line":""},{"line_number":150,"context_line":"  # from swift.common.utils import PivotTree, pivot_to_pivot_container"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a2fa921_361adbc6","line":147,"updated":"2015-10-11 12:01:57.000000000","message":"So the pivot tree is essentially a data structure for maintaining pivot points (or would it be more accurate to say maintaining the pivot table?)\n\nThe table support the following operations:\n\n1. Adding a pivot point\n\n2. Iterating over the tree leaves\n\nDo you think there is also a need for deletion? what about constructing from file?","commit_id":"cd15a3729068a2e77c52adbf65635a3bed03cf1f"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"9c3fb86f7de5dbd1235e054025eb753c69a0c8b6","unresolved":false,"context_lines":[{"line_number":144,"context_line":"  Account: .sharded_a Shard_cont(\u003c): c\u003c-swift_is_cool Shard_cont(\u003e): c-\u003eswift_is_cool"},{"line_number":145,"context_line":""},{"line_number":146,"context_line":"PivotTree Class"},{"line_number":147,"context_line":"-----------------"},{"line_number":148,"context_line":"This class once loaded up with pivot points, makes it easy iterate through all leaf nodes::"},{"line_number":149,"context_line":""},{"line_number":150,"context_line":"  # from swift.common.utils import PivotTree, pivot_to_pivot_container"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a2fa921_5f39a7d7","line":147,"in_reply_to":"7a2fa921_361adbc6","updated":"2015-10-12 00:49:31.000000000","message":"Yup, but is tree based as sharding continues down from the root tree.\n\nIt doesn\u0027t delete atm, as I haven\u0027t build deleting (shrinking) the number of shard containers.\n\nThis is usually built inside the container server so grabs the list of pivot points directly from the backend.","commit_id":"cd15a3729068a2e77c52adbf65635a3bed03cf1f"},{"author":{"_account_id":11317,"name":"Eran Rom","email":"eran@itsonlyme.name","username":"eranr"},"change_message_id":"76b008bd0d7ebff9d9232be927da6312d13d419f","unresolved":false,"context_lines":[{"line_number":249,"context_line":""},{"line_number":250,"context_line":"* Pro: No need to pass the root containers pivot nodes (or just leaves) as all the action will happen with access to the root containers broker."},{"line_number":251,"context_line":"* Con: Because it\u0027s happening on the container-servers, care needs to be given in regard to requests. Is one of the other primaries also generating a list, when the container-server calls other leaf nodes, are we sending out replica count more request (potentially from each primary node). Maybe a container server needs to only attempt to contact a random primary node at a time to reduce requests, or even GET/HEAD from the proxy with 100-continue, and get the proxy to pick only 1 server to continue on."},{"line_number":252,"context_line":""},{"line_number":253,"context_line":"DELETE"},{"line_number":254,"context_line":"^^^^^^^"},{"line_number":255,"context_line":"Delete has the same options as GET/HEAD above, either it runs in the proxy or the container-server. But the general idea will be:"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a2fa921_6c116251","line":252,"updated":"2015-10-11 12:01:57.000000000","message":"FWIW, I vote for the proxy option.Its not very different from e.g. SLO or even EC if you consider the gathering part...","commit_id":"cd15a3729068a2e77c52adbf65635a3bed03cf1f"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"9c3fb86f7de5dbd1235e054025eb753c69a0c8b6","unresolved":false,"context_lines":[{"line_number":249,"context_line":""},{"line_number":250,"context_line":"* Pro: No need to pass the root containers pivot nodes (or just leaves) as all the action will happen with access to the root containers broker."},{"line_number":251,"context_line":"* Con: Because it\u0027s happening on the container-servers, care needs to be given in regard to requests. Is one of the other primaries also generating a list, when the container-server calls other leaf nodes, are we sending out replica count more request (potentially from each primary node). Maybe a container server needs to only attempt to contact a random primary node at a time to reduce requests, or even GET/HEAD from the proxy with 100-continue, and get the proxy to pick only 1 server to continue on."},{"line_number":252,"context_line":""},{"line_number":253,"context_line":"DELETE"},{"line_number":254,"context_line":"^^^^^^^"},{"line_number":255,"context_line":"Delete has the same options as GET/HEAD above, either it runs in the proxy or the container-server. But the general idea will be:"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a2fa921_9f55fffb","line":252,"in_reply_to":"7a2fa921_6c116251","updated":"2015-10-12 00:49:31.000000000","message":"Yeah, the problem is passing the pivot tree around and bulding the container list. Both options have pro\u0027s and con\u0027s. I still personally on the fence, the last version of the POC was inside the proxy. the currently implementation is in the container server.. and the code is much cleaner, specifically all the format options. But there are caveats. So both options have merit and this I think will become a good point of discussion when we get to the real implementation phase.","commit_id":"cd15a3729068a2e77c52adbf65635a3bed03cf1f"},{"author":{"_account_id":11317,"name":"Eran Rom","email":"eran@itsonlyme.name","username":"eranr"},"change_message_id":"76b008bd0d7ebff9d9232be927da6312d13d419f","unresolved":false,"context_lines":[{"line_number":272,"context_line":""},{"line_number":273,"context_line":"#. Sync the objects from the unsharded container into the objects table of the root/branch container."},{"line_number":274,"context_line":"#. Let the conatiner-sharder replicate the objects down to the correct shard. (noting that dealing with misplaced objects in a shard is apart of the sharder\u0027s job)"},{"line_number":275,"context_line":""},{"line_number":276,"context_line":"pending and merge_items"},{"line_number":277,"context_line":"^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"},{"line_number":278,"context_line":"This version of the POC will take advantage of the last POC\u0027s changes to replication. They will at least suffice while it\u0027s a POC."}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a2fa921_ac58da31","line":275,"updated":"2015-10-11 12:01:57.000000000","message":"Should we also be worried with a state where we have:\n\none replica seeing an unsharded container\n\nOne replica seeing a sharded container into 2 shards\n\nOne replica seeing a sharded container into 3 shards (one of the above shards got \u0027sharded\u0027 once more)\n\nI guess this can happen only if there has been a replica that was down for a long time...","commit_id":"cd15a3729068a2e77c52adbf65635a3bed03cf1f"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"9c3fb86f7de5dbd1235e054025eb753c69a0c8b6","unresolved":false,"context_lines":[{"line_number":272,"context_line":""},{"line_number":273,"context_line":"#. Sync the objects from the unsharded container into the objects table of the root/branch container."},{"line_number":274,"context_line":"#. Let the conatiner-sharder replicate the objects down to the correct shard. (noting that dealing with misplaced objects in a shard is apart of the sharder\u0027s job)"},{"line_number":275,"context_line":""},{"line_number":276,"context_line":"pending and merge_items"},{"line_number":277,"context_line":"^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"},{"line_number":278,"context_line":"This version of the POC will take advantage of the last POC\u0027s changes to replication. They will at least suffice while it\u0027s a POC."}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a2fa921_3f5e2b13","line":275,"in_reply_to":"7a2fa921_ac58da31","updated":"2015-10-12 00:49:31.000000000","message":"Object placement will never change, so the only thing effected by a stale (old view of the container) is container listings, so we\u0027re in the same state as we would normally be in with unsharded containers.\n\nContainer updates and replication to/from an old container is a problem. but not too bad because the container listing will only look for objects in the leaves. These maybe old (as above), but the dealing with misplaced objects in the sharder will mean data replicated into branches will relocate and merge correctly (via timestamps) in the leaves thanks to the sharder, meaning eventually the data _should_ be correct.\n\nI\u0027ll try an make this clearer in writing and images in my next version of the spec.","commit_id":"cd15a3729068a2e77c52adbf65635a3bed03cf1f"},{"author":{"_account_id":11317,"name":"Eran Rom","email":"eran@itsonlyme.name","username":"eranr"},"change_message_id":"76b008bd0d7ebff9d9232be927da6312d13d419f","unresolved":false,"context_lines":[{"line_number":291,"context_line":""},{"line_number":292,"context_line":"There is a caveat however, which is that currently the hook grabs all the objects from the pivot_points table."},{"line_number":293,"context_line":"There is no idea of pointer/sync point. The number of pivot_point should remain fairly small, at least in relation to objects."},{"line_number":294,"context_line":""},{"line_number":295,"context_line":"Container-sharder"},{"line_number":296,"context_line":"-------------------"},{"line_number":297,"context_line":"The container-sharder, will run on all container-server nodes. At an interval it will parse all shared containers,"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a2fa921_4c4ee63b","line":294,"updated":"2015-10-11 12:01:57.000000000","message":"Container sync impact:\nThe existing implementation of container sync looks for containers having the \u0027sync-to\u0027 metadata item. This metadata points to a remote container to be synced with. Such containers dbs are scanned by the container sync daemons which replicate the PUT / DELETE operations to the container pointed by the \u0027sync-to\u0027 metadata item. While at it the daemon maintains pointers that persistently keep the state it reached in replicating.\n\nSeems like that daemon would now need to gather the list and keep global pointers. One way to do so would be to keep a pointer in the pivot table, and another in the db table.\n\nClearly we cannot assume that the remote side also does sharding, nor that the shards are in anyway in sync with the shards of the local container.","commit_id":"cd15a3729068a2e77c52adbf65635a3bed03cf1f"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"9c3fb86f7de5dbd1235e054025eb753c69a0c8b6","unresolved":false,"context_lines":[{"line_number":291,"context_line":""},{"line_number":292,"context_line":"There is a caveat however, which is that currently the hook grabs all the objects from the pivot_points table."},{"line_number":293,"context_line":"There is no idea of pointer/sync point. The number of pivot_point should remain fairly small, at least in relation to objects."},{"line_number":294,"context_line":""},{"line_number":295,"context_line":"Container-sharder"},{"line_number":296,"context_line":"-------------------"},{"line_number":297,"context_line":"The container-sharder, will run on all container-server nodes. At an interval it will parse all shared containers,"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a2fa921_9fa3bf00","line":294,"in_reply_to":"7a2fa921_4c4ee63b","updated":"2015-10-12 00:49:31.000000000","message":"Yeah there will clearly be container sync issues, and would more then happy to talk to you about them and try and nut out a solution with you, will you be at summit?","commit_id":"cd15a3729068a2e77c52adbf65635a3bed03cf1f"},{"author":{"_account_id":11317,"name":"Eran Rom","email":"eran@itsonlyme.name","username":"eranr"},"change_message_id":"76b008bd0d7ebff9d9232be927da6312d13d419f","unresolved":false,"context_lines":[{"line_number":374,"context_line":"Does this introduce any additional security risks, or are there"},{"line_number":375,"context_line":"security-related considerations which should be discussed?"},{"line_number":376,"context_line":""},{"line_number":377,"context_line":"TBA (I\u0027m sure there are, like potential sharded container name collisions)."},{"line_number":378,"context_line":""},{"line_number":379,"context_line":"Testing"},{"line_number":380,"context_line":"-------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a2fa921_cc60b6a7","line":377,"updated":"2015-10-11 12:01:57.000000000","message":"Not sure this is a security issue but should special code needs to be written so as to block access to the .sharded namespace?","commit_id":"cd15a3729068a2e77c52adbf65635a3bed03cf1f"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"9c3fb86f7de5dbd1235e054025eb753c69a0c8b6","unresolved":false,"context_lines":[{"line_number":374,"context_line":"Does this introduce any additional security risks, or are there"},{"line_number":375,"context_line":"security-related considerations which should be discussed?"},{"line_number":376,"context_line":""},{"line_number":377,"context_line":"TBA (I\u0027m sure there are, like potential sharded container name collisions)."},{"line_number":378,"context_line":""},{"line_number":379,"context_line":"Testing"},{"line_number":380,"context_line":"-------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a2fa921_bfa8fbdc","line":377,"in_reply_to":"7a2fa921_cc60b6a7","updated":"2015-10-12 00:49:31.000000000","message":"Yeah, good point. Need to think about this some more :)","commit_id":"cd15a3729068a2e77c52adbf65635a3bed03cf1f"},{"author":{"_account_id":11317,"name":"Eran Rom","email":"eran@itsonlyme.name","username":"eranr"},"change_message_id":"160586fdfabbfabaadf98bafcac3f5738667783b","unresolved":false,"context_lines":[{"line_number":45,"context_line":"the database file. Because of this behaviour, the current suggestion for clusters storing"},{"line_number":46,"context_line":"many objects in a single container is to make sure the container databases are stored on"},{"line_number":47,"context_line":"SSDs, to reduce latency when accessing large database files."},{"line_number":48,"context_line":""},{"line_number":49,"context_line":"In a previous version of this spec, I investigated different approaches we could use"},{"line_number":50,"context_line":"to shard containers. These were:"},{"line_number":51,"context_line":""}],"source_content_type":"text/x-rst","patch_set":7,"id":"1a122d0e_ee3ba3e9","line":48,"updated":"2016-04-25 19:58:54.000000000","message":"Furthermore, experience from existing deployments show that the container dbs size can be highly skewed, causing non uniform capacity requirements amongst the container servers. With container sync this non-uniformity goes beyond capacity as container servers are also responsible for replication.","commit_id":"d1d3750a3ea70d5d46f33c1302c6550f657a43d7"},{"author":{"_account_id":11317,"name":"Eran Rom","email":"eran@itsonlyme.name","username":"eranr"},"change_message_id":"160586fdfabbfabaadf98bafcac3f5738667783b","unresolved":false,"context_lines":[{"line_number":73,"context_line":"\"X-Container-Sharding: On\" X-Header via either PUT or POST::"},{"line_number":74,"context_line":""},{"line_number":75,"context_line":"  curl -i -H \u0027X-Auth-Token: \u003ctoken\u003e\u0027 -H \u0027X-Container-Sharding: On\u0027 \u003curl\u003e/\u003caccount\u003e/\u003ccontainer\u003e -X PUT"},{"line_number":76,"context_line":""},{"line_number":77,"context_line":"Once enabled when a container gets too full, say at 1 million objects. A pivot point is found"},{"line_number":78,"context_line":"(the middle item) which will be used to split the container. This split will create 2 additional containers each"},{"line_number":79,"context_line":"holding 1/2 the objects. The configuration parameter `shard_container_size` determines what size a container can get to before it\u0027s sharded (defaulting to 1 million)."}],"source_content_type":"text/x-rst","patch_set":7,"id":"1a122d0e_ee128361","line":76,"updated":"2016-04-25 19:58:54.000000000","message":"Perhaps this is already the case, but why not have this as a global parameter.\nGiven the simplified version that saves on requests, and given the fact one can control the default \u0027split\u0027 size perhaps we want to allow only to override a global default at the container level?","commit_id":"d1d3750a3ea70d5d46f33c1302c6550f657a43d7"},{"author":{"_account_id":11317,"name":"Eran Rom","email":"eran@itsonlyme.name","username":"eranr"},"change_message_id":"160586fdfabbfabaadf98bafcac3f5738667783b","unresolved":false,"context_lines":[{"line_number":82,"context_line":"ever see 1 container, this we call the root container. The sharded namespace is::"},{"line_number":83,"context_line":""},{"line_number":84,"context_line":"  .sharded_\u003caccount\u003e/"},{"line_number":85,"context_line":""},{"line_number":86,"context_line":"Containers that have been split no longer hold object metadata and so once the new containers are durable can be deleted (except for the root container)."},{"line_number":87,"context_line":"The root container, like any other split container, contains no objects in it\u0027s ``object`` table however it"},{"line_number":88,"context_line":"has a new table to store the pivot/range information. This information can be used to easily and quickly"}],"source_content_type":"text/x-rst","patch_set":7,"id":"1a122d0e_193627e7","line":85,"updated":"2016-04-25 19:58:54.000000000","message":"A remark I once got from Clay, w.r.t. the synced containr folder: Why hide it from the admin. Perhaps make it sharded_\u003caccount\u003e/?","commit_id":"d1d3750a3ea70d5d46f33c1302c6550f657a43d7"},{"author":{"_account_id":11317,"name":"Eran Rom","email":"eran@itsonlyme.name","username":"eranr"},"change_message_id":"160586fdfabbfabaadf98bafcac3f5738667783b","unresolved":false,"context_lines":[{"line_number":146,"context_line":"* X-Container-Sysmeta-Shard-Container - This is the original container."},{"line_number":147,"context_line":"* X-Container-Sysmeta-Shard-Lower - The lower point of the range for this container."},{"line_number":148,"context_line":"* X-Container-Sysmeta-Shard-Upper - The upper point of the range for this container."},{"line_number":149,"context_line":""},{"line_number":150,"context_line":"Pivot point"},{"line_number":151,"context_line":"--------------"},{"line_number":152,"context_line":"The Pivot point is the middle object in the container. As Swift is eventually consistent all the containers"}],"source_content_type":"text/x-rst","patch_set":7,"id":"1a122d0e_5947ff4a","line":149,"updated":"2016-04-25 19:58:54.000000000","message":"Again, I have not looked at the code so this may be redundant, but for container sync we may need more then that...Esp if we see the sync work getting distributed as well.","commit_id":"d1d3750a3ea70d5d46f33c1302c6550f657a43d7"},{"author":{"_account_id":11317,"name":"Eran Rom","email":"eran@itsonlyme.name","username":"eranr"},"change_message_id":"160586fdfabbfabaadf98bafcac3f5738667783b","unresolved":false,"context_lines":[{"line_number":179,"context_line":"The PivotRange class lives in swift.common.utils, and there are some other helper methods there that are used:"},{"line_number":180,"context_line":""},{"line_number":181,"context_line":"* find_pivot_range(item, ranges) - Finds what range from a list of ranges that an item belongs."},{"line_number":182,"context_line":"* pivot_to_pivot_container(account, container, lower\u003dNone, upper\u003dNone, pivot_range\u003dNone) - Given a root container account and container and either lower and upper or just a pivot_range generate the required sharded name."},{"line_number":183,"context_line":""},{"line_number":184,"context_line":"Getting PivotRanges"},{"line_number":185,"context_line":"--------------------"}],"source_content_type":"text/x-rst","patch_set":7,"id":"1a122d0e_39b04343","line":182,"updated":"2016-04-25 19:58:54.000000000","message":"nit: I would split to:\npivot_bounds_to_pivot_container(account, container, lower, upper)\npivot_range_to_pivot_container(account, container, range)","commit_id":"d1d3750a3ea70d5d46f33c1302c6550f657a43d7"},{"author":{"_account_id":11317,"name":"Eran Rom","email":"eran@itsonlyme.name","username":"eranr"},"change_message_id":"160586fdfabbfabaadf98bafcac3f5738667783b","unresolved":false,"context_lines":[{"line_number":210,"context_line":"as the following picture demonstrates:"},{"line_number":211,"context_line":""},{"line_number":212,"context_line":".. image:: images/seq_obj_put_delete.png"},{"line_number":213,"context_line":""},{"line_number":214,"context_line":"This is accomplished by getting the container-server to set the required X-Container-{Host, Device, Partition}"},{"line_number":215,"context_line":"headers in the response that the object-{server, updater} require to redirect it\u0027s update."},{"line_number":216,"context_line":"Only one new host is added to the headers, the container server determines which one by picking the"}],"source_content_type":"text/x-rst","patch_set":7,"id":"1a122d0e_e92f0686","line":213,"updated":"2016-04-25 19:58:54.000000000","message":"I think that with the recent fast-post work this is also true for object POST (when content type is changed).\nI guess that if put / delete can handle 301 then so does post","commit_id":"d1d3750a3ea70d5d46f33c1302c6550f657a43d7"},{"author":{"_account_id":11317,"name":"Eran Rom","email":"eran@itsonlyme.name","username":"eranr"},"change_message_id":"160586fdfabbfabaadf98bafcac3f5738667783b","unresolved":false,"context_lines":[{"line_number":273,"context_line":"--------------------"},{"line_number":274,"context_line":"The container-replicator (and db_replicator as required) has been updated to replicate and sync the pivot_range table."},{"line_number":275,"context_line":""},{"line_number":276,"context_line":"Swift is eventually consistent, meaning at some point we will have an unsharded version of a container replicate with a sharded one, and being eventually consistent, some of the objects in the un-sharded might actually exist and need to be merged into a lower down shard."},{"line_number":277,"context_line":"The current thinking is that a sharded container holds all it\u0027s objects in the leaves. Leaving the root and branch container\u0027s object table empty, these non-leaves also will not be queried when object listing. So the plan is:"},{"line_number":278,"context_line":""},{"line_number":279,"context_line":"#. Sync the objects from the unsharded container into the objects table of the root/branch container."}],"source_content_type":"text/x-rst","patch_set":7,"id":"1a122d0e_29929e4b","line":276,"updated":"2016-04-25 19:58:54.000000000","message":"Not sure the following makes sense:\nIf the replicator \u0027runs into\u0027 two replicas where one is sharded and the other one isn\u0027t, can it just \u0027bail out\u0027, and wait till the sharder does it job on that container?\n\nMore generally, if I kept up so far then there are 3 types of container dbs:\n1. a sharded root container\n2. an \u0027intermediate\u0027 sharded container\n3. a leaf container (or a non sharded container)\nIMHO, each type should have its own replication logic. Thus, we may see the sharder and replicator(s) as a two level eventual consistency processes. At the top level, the sharder makes sure that each container (not container db but rather a logical container) has a consistent \u0027sharding view\u0027, and the replicator makes sure each sharded \u0027level\u0027 is replicated correctly.\n\nWith that idea in mind, one can think of a single process (sharder / replicator) that when seeing two replicas of the same container where one is sharded and the other one is not, it applies the sharding on the non sharded replica, and waits for the next round to do replication.","commit_id":"d1d3750a3ea70d5d46f33c1302c6550f657a43d7"},{"author":{"_account_id":11317,"name":"Eran Rom","email":"eran@itsonlyme.name","username":"eranr"},"change_message_id":"160586fdfabbfabaadf98bafcac3f5738667783b","unresolved":false,"context_lines":[{"line_number":306,"context_line":"-------------------"},{"line_number":307,"context_line":"The container-sharder, will run on all container-server nodes. At an interval it will parse all shared containers,"},{"line_number":308,"context_line":"on each it:"},{"line_number":309,"context_line":""},{"line_number":310,"context_line":"* Audits the container"},{"line_number":311,"context_line":"* Deals with any misplaced items. That is items that should belong in a different range container."},{"line_number":312,"context_line":"* We then check the size of the container, when we do _one_ of the following happens:"}],"source_content_type":"text/x-rst","patch_set":7,"id":"1a122d0e_89e772e3","line":309,"updated":"2016-04-25 19:58:54.000000000","message":"If it parses only sharded containers, how does a container get sharded for the first time?","commit_id":"d1d3750a3ea70d5d46f33c1302c6550f657a43d7"},{"author":{"_account_id":11317,"name":"Eran Rom","email":"eran@itsonlyme.name","username":"eranr"},"change_message_id":"160586fdfabbfabaadf98bafcac3f5738667783b","unresolved":false,"context_lines":[{"line_number":366,"context_line":"  * create new containers locally"},{"line_number":367,"context_line":"  * fill them up while delete the objects locally"},{"line_number":368,"context_line":"  * replicate all the containers out and update the root container of the changes. (Delete old range, and add to 2 new)."},{"line_number":369,"context_line":""},{"line_number":370,"context_line":".. note::"},{"line_number":371,"context_line":""},{"line_number":372,"context_line":"  When deleting objects from the container being split the timestamp used is the same as the existing object but using Timestamp.internal with the offset incremented. Allowing newer versions of the object/metadata to not be squashed."}],"source_content_type":"text/x-rst","patch_set":7,"id":"1a122d0e_247765a0","line":369,"updated":"2016-04-25 19:58:54.000000000","message":"Now that I have a better idea of the sharding process, I ownder if we can get rid of the \u0027intermediate\u0027 nodes. Seems like with a little extra info in the root node, these are not necessary. Not sure about the pros and cons of it...","commit_id":"d1d3750a3ea70d5d46f33c1302c6550f657a43d7"},{"author":{"_account_id":11317,"name":"Eran Rom","email":"eran@itsonlyme.name","username":"eranr"},"change_message_id":"160586fdfabbfabaadf98bafcac3f5738667783b","unresolved":false,"context_lines":[{"line_number":379,"context_line":"Turns out shrinking (merging containers back when they get too small) is even more complicated then sharding (growing)."},{"line_number":380,"context_line":""},{"line_number":381,"context_line":"When sharding, we at least have all the objects that need to shard all on the container server we were on."},{"line_number":382,"context_line":"When shrinking, we need to find a range neighbour that most likely lives somewhere else."},{"line_number":383,"context_line":""},{"line_number":384,"context_line":"So what does the current POC do? At the moment it\u0027s another 2 phase procedure. Although while writing this SPEC update I think this might have to become a 3 phase as we probably need an initial state to do nothing but"},{"line_number":385,"context_line":"let Swift know something will happen."}],"source_content_type":"text/x-rst","patch_set":7,"id":"1a122d0e_44e9c119","line":382,"updated":"2016-04-25 19:58:54.000000000","message":"I wonder if object_count can be used to do this decision while looking only at the root container. I think you mention that these are kept per range.","commit_id":"d1d3750a3ea70d5d46f33c1302c6550f657a43d7"},{"author":{"_account_id":11317,"name":"Eran Rom","email":"eran@itsonlyme.name","username":"eranr"},"change_message_id":"160586fdfabbfabaadf98bafcac3f5738667783b","unresolved":false,"context_lines":[{"line_number":521,"context_line":"This would increment the current value. In this case it would make the new value 101. A \u0027-\u0027 will decrement."},{"line_number":522,"context_line":""},{"line_number":523,"context_line":"The idea behind this is if an Op wants to sacrifice more requests in the cluster with more uptodate stats, we could get the object-updaters and object-servers to send a + or - once an object is added or deleted. The sharder would correct the count if it\u0027s gets slightly out of sync."},{"line_number":524,"context_line":""},{"line_number":525,"context_line":"The merge_items method in the ContainerBroker will merge prefixed requests together (+2 etc) if required."},{"line_number":526,"context_line":""},{"line_number":527,"context_line":"Assignee(s)"}],"source_content_type":"text/x-rst","patch_set":7,"id":"1a122d0e_e49a8d6d","line":524,"updated":"2016-04-25 19:58:54.000000000","message":"If I got it right, then the object-{updaters,servers} anyway send their updated to the root container (getting back 301) and so the idea here might not need to \"sacrifice more requests\". Still, I think  that the suggested coalesced approach is better.","commit_id":"d1d3750a3ea70d5d46f33c1302c6550f657a43d7"}]}