)]}'
{"specs/in_progress/container_sharding.rst":[{"author":{"_account_id":597,"name":"Pete Zaitcev","email":"zaitcev@kotori.zaitcev.us","username":"zaitcev"},"change_message_id":"8cd1a0b2a85b63c7100d1efab8b28d1af79dd567","unresolved":false,"context_lines":[{"line_number":9,"context_line":"  any of the sections in this template.  If you have nothing to say"},{"line_number":10,"context_line":"  for a whole section, just write: \"None\". For help with syntax, see"},{"line_number":11,"context_line":"  http://sphinx-doc.org/rest.html To test out your formatting, see"},{"line_number":12,"context_line":"  http://www.tele3.cz/jbar/rest/rest.html"},{"line_number":13,"context_line":""},{"line_number":14,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":15,"context_line":"Container Sharding"}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a4dcd0f_1f1b8ccb","line":12,"updated":"2015-08-12 18:10:17.000000000","message":"It\u0027s okay to drop this part :-)","commit_id":"494359620f8642bfef96ef11cdede7d3657ce31d"},{"author":{"_account_id":597,"name":"Pete Zaitcev","email":"zaitcev@kotori.zaitcev.us","username":"zaitcev"},"change_message_id":"8cd1a0b2a85b63c7100d1efab8b28d1af79dd567","unresolved":false,"context_lines":[{"line_number":34,"context_line":"contained within. As the amount of objects in a container grows, so does the size of"},{"line_number":35,"context_line":"the database file. Because of this behaviour, the current suggestion for clusters storing"},{"line_number":36,"context_line":"many objects in a single container is to make sure the container databases are stored on"},{"line_number":37,"context_line":"SSDs, to reduce latency when accessing large database files."},{"line_number":38,"context_line":""},{"line_number":39,"context_line":"In a previous version of this spec, I investigated different approaches we could use"},{"line_number":40,"context_line":"to shard containers. These were:"}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a4dcd0f_30fb17c3","line":37,"updated":"2015-08-12 18:10:17.000000000","message":"Do we have any numerical guidance? For old spinning disks I saw suggestions to start app sharding below 100,000 objects per container. Current users are rumored to push at least half-order above 1,000,000 on SSDs. Would mentioning this help to frame the problem? Or TMI?","commit_id":"494359620f8642bfef96ef11cdede7d3657ce31d"},{"author":{"_account_id":597,"name":"Pete Zaitcev","email":"zaitcev@kotori.zaitcev.us","username":"zaitcev"},"change_message_id":"8cd1a0b2a85b63c7100d1efab8b28d1af79dd567","unresolved":false,"context_lines":[{"line_number":125,"context_line":"    fig 2: a very basic example of a DPT, in this case \"obj\" stores a distributed node."},{"line_number":126,"context_line":"    NOTE: For memory efficientcy we only store the next character in the next node of the trie, but above is easier to read."},{"line_number":127,"context_line":""},{"line_number":128,"context_line":"Here the \u0027obj\u0027 node belongs to a distributed node. When an object is attempted to be put into this portion of the trie, a ShardTrieDistrbutedBranchException is raised which contains the distributed node where this node and it\u0027s children are location.  This tells Swift the prefix of the sharded container to continue with. That container\u0027s shard trie continues from this location, where it might contain another distributed node. This way the trie can be sharded indefinitely, and adapt to the users usage of the container."},{"line_number":129,"context_line":""},{"line_number":130,"context_line":"Solution"},{"line_number":131,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a4dcd0f_8fb500ee","line":128,"updated":"2015-08-12 18:10:17.000000000","message":"\"its\" \"are located\"","commit_id":"494359620f8642bfef96ef11cdede7d3657ce31d"},{"author":{"_account_id":597,"name":"Pete Zaitcev","email":"zaitcev@kotori.zaitcev.us","username":"zaitcev"},"change_message_id":"8cd1a0b2a85b63c7100d1efab8b28d1af79dd567","unresolved":false,"context_lines":[{"line_number":134,"context_line":"-----------------"},{"line_number":135,"context_line":"This class makes the trie easy to use, and does all the hard work. It has the obvious methods to add"},{"line_number":136,"context_line":"and remove nodes, in additional it also has the ability to:"},{"line_number":137,"context_line":""},{"line_number":138,"context_line":"* Split and join subtrees, doing what it suggests. When splitting, a distributed node is added to the"},{"line_number":139,"context_line":"   original trie at the place of the split."},{"line_number":140,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a4dcd0f_c2cc3f37","line":137,"updated":"2015-08-12 18:10:17.000000000","message":"Before going to dump/load, explain the initialization from actual trie in shard_nodes table. This is entirely unobvious, because usually dump/load is all a class has to fill or spill the data, but it\u0027s not so in case of ShardTrie -- if I even understand it.","commit_id":"494359620f8642bfef96ef11cdede7d3657ce31d"},{"author":{"_account_id":597,"name":"Pete Zaitcev","email":"zaitcev@kotori.zaitcev.us","username":"zaitcev"},"change_message_id":"8cd1a0b2a85b63c7100d1efab8b28d1af79dd567","unresolved":false,"context_lines":[{"line_number":152,"context_line":"* Various iteration methods, so you can iterate through distributed nodes, data nodes, or both (important nodes)."},{"line_number":153,"context_line":""},{"line_number":154,"context_line":"* trim_trunk: When dealing with a sharded subtrie that has been built from the container broker, this method"},{"line_number":155,"context_line":"   trims the truck so the root_node of the trie is at the correct location (\u0027obj\u0027) in the above example."},{"line_number":156,"context_line":""},{"line_number":157,"context_line":"When you do a get or add operation, a ShardTrieDistrbutedBranchException will be raised if a distributed node is needed to be accessed. This tells you that you need to go and find the subtree at that prefix, to continue getting or adding the object."},{"line_number":158,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a4dcd0f_cff30872","line":155,"updated":"2015-08-12 18:10:17.000000000","message":"\"trunk\"","commit_id":"494359620f8642bfef96ef11cdede7d3657ce31d"},{"author":{"_account_id":597,"name":"Pete Zaitcev","email":"zaitcev@kotori.zaitcev.us","username":"zaitcev"},"change_message_id":"8cd1a0b2a85b63c7100d1efab8b28d1af79dd567","unresolved":false,"context_lines":[{"line_number":172,"context_line":"   naming of sharded account name and container names are defined, currently::"},{"line_number":173,"context_line":""},{"line_number":174,"context_line":"    .sharded_%(account)s"},{"line_number":175,"context_line":"    %(container)s.prefix.%(prefix)"},{"line_number":176,"context_line":""},{"line_number":177,"context_line":"Sharding approach explained"},{"line_number":178,"context_line":"------------------------------"}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a4dcd0f_2f286c99","line":175,"updated":"2015-08-12 18:10:17.000000000","message":"At this point it\u0027s not explained why the extra account is needed (hiding the shards from the account\u0027s owner, perhaps? is this actually a worthy goal, then?)","commit_id":"494359620f8642bfef96ef11cdede7d3657ce31d"},{"author":{"_account_id":597,"name":"Pete Zaitcev","email":"zaitcev@kotori.zaitcev.us","username":"zaitcev"},"change_message_id":"8cd1a0b2a85b63c7100d1efab8b28d1af79dd567","unresolved":false,"context_lines":[{"line_number":194,"context_line":"NOTE2: AT this time, sharding only goes one way, growing. It should be able to shrink but that code doesn\u0027t exist yet."},{"line_number":195,"context_line":""},{"line_number":196,"context_line":"As the actual sharding process is done in the background. The rest of swift is just shard aware. For example, a container GET"},{"line_number":197,"context_line":"(container listing) on a sharded container with get the shard trie of the root, and then reucursively iterate over it following any distributed"},{"line_number":198,"context_line":"nodes. This maintains the object order."},{"line_number":199,"context_line":""},{"line_number":200,"context_line":"A DELETE on the container will attempt to recursively iterate through the distributed nodes and attempt to delete sharded nodes first. Parent"}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a4dcd0f_423fcfb3","line":197,"updated":"2015-08-12 18:10:17.000000000","message":"\"recursively\" - the whole doc might need to be spell-checked","commit_id":"494359620f8642bfef96ef11cdede7d3657ce31d"},{"author":{"_account_id":597,"name":"Pete Zaitcev","email":"zaitcev@kotori.zaitcev.us","username":"zaitcev"},"change_message_id":"8cd1a0b2a85b63c7100d1efab8b28d1af79dd567","unresolved":false,"context_lines":[{"line_number":254,"context_line":""},{"line_number":255,"context_line":"This could have occurred in two different ways:"},{"line_number":256,"context_line":""},{"line_number":257,"context_line":"#. Many objects came in under the \u0027catalog\u0027 prefix, sharding it into it\u0027s own shard, then at a later time there was enough objects"},{"line_number":258,"context_line":"   under \u0027cat\u0027 for shard again; OR"},{"line_number":259,"context_line":""},{"line_number":260,"context_line":"#. Many objects under \u0027cat\u0027 came in causing a shard at the \u0027cat\u0027 node. Then sometime later, enough objects caused the \u0027cat\u0027 shard"}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a4dcd0f_826e37fe","line":257,"updated":"2015-08-12 18:10:17.000000000","message":"\"its\"","commit_id":"494359620f8642bfef96ef11cdede7d3657ce31d"},{"author":{"_account_id":597,"name":"Pete Zaitcev","email":"zaitcev@kotori.zaitcev.us","username":"zaitcev"},"change_message_id":"8cd1a0b2a85b63c7100d1efab8b28d1af79dd567","unresolved":false,"context_lines":[{"line_number":295,"context_line":"  );"},{"line_number":296,"context_line":""},{"line_number":297,"context_line":"This table behaves similar to the object\u0027s table, where every node is timestamped and a node can be marked as deleted before it"},{"line_number":298,"context_line":"has been cleaned up (see the changes to merge_items below)."},{"line_number":299,"context_line":""},{"line_number":300,"context_line":"build_shard_trie method"},{"line_number":301,"context_line":"^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a4dcd0f_62a9ab8f","line":298,"updated":"2015-08-12 18:10:17.000000000","message":"The above does not explain how the table shard_nodes actually represents the trie. I _presume_ the \u0027name\u0027 field is prefix for a trie node, but please spell it out.","commit_id":"494359620f8642bfef96ef11cdede7d3657ce31d"},{"author":{"_account_id":597,"name":"Pete Zaitcev","email":"zaitcev@kotori.zaitcev.us","username":"zaitcev"},"change_message_id":"8cd1a0b2a85b63c7100d1efab8b28d1af79dd567","unresolved":false,"context_lines":[{"line_number":321,"context_line":"Container replication changes"},{"line_number":322,"context_line":"^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"},{"line_number":323,"context_line":"Because swift is an eventually consistent system, we need to make sure that when container databases are replicated, this doesn\u0027t only replicate"},{"line_number":324,"context_line":"items in the objects table, but also the nodes in the shard_nodes table. Most of the database replication code is apart of the db_replicator"},{"line_number":325,"context_line":"which is a parent and so shared by account and container replication. The current solution in the POC, is to add an _other_items_hook(broker) hook that is over-written in the container replicator to grab the items from the shard_nodes table and returned in the items format to be passed into merge_items."},{"line_number":326,"context_line":""},{"line_number":327,"context_line":"There is a caveat however, which is that currently the hook grabs all the objects from the shard_nodes table. There is no idea of pointer/sync"}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a4dcd0f_c2c19fe5","line":324,"updated":"2015-08-12 18:10:17.000000000","message":"\"is a part\"","commit_id":"494359620f8642bfef96ef11cdede7d3657ce31d"},{"author":{"_account_id":597,"name":"Pete Zaitcev","email":"zaitcev@kotori.zaitcev.us","username":"zaitcev"},"change_message_id":"8cd1a0b2a85b63c7100d1efab8b28d1af79dd567","unresolved":false,"context_lines":[{"line_number":358,"context_line":"proxy changes"},{"line_number":359,"context_line":"------------------"},{"line_number":360,"context_line":""},{"line_number":361,"context_line":"The proxy controllers have been modified, making them shard aware. The base controller now has a get_shard_trie method which takes the account and container and returns it\u0027s shardtrie, it does this by building a container server GET request using the before mentioned \u0027?format\u003dtrie\u0027 query."},{"line_number":362,"context_line":""},{"line_number":363,"context_line":"object controller"},{"line_number":364,"context_line":"^^^^^^^^^^^^^^^^^^^^^^^"}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a4dcd0f_62b44b81","line":361,"updated":"2015-08-12 18:10:17.000000000","message":"\"its\"","commit_id":"494359620f8642bfef96ef11cdede7d3657ce31d"},{"author":{"_account_id":597,"name":"Pete Zaitcev","email":"zaitcev@kotori.zaitcev.us","username":"zaitcev"},"change_message_id":"8cd1a0b2a85b63c7100d1efab8b28d1af79dd567","unresolved":false,"context_lines":[{"line_number":374,"context_line":""},{"line_number":375,"context_line":"GET and HEAD are different kettles of fish. If the container is sharded, GETorHEAD_base method needs to be called on all shard containers, and then the responses combined. If sharded there is a fork in the GETorHEAD path that calls GETorHEAD_sharded, which in essence just what I just mentioned, uses the base class\u0027 get_shard_trie method to recursively itereate through the shard containers building up (combining) the response."},{"line_number":376,"context_line":""},{"line_number":377,"context_line":"The limit, marker and end_marker logic has been added to GETorHEAD_sharded.. but this is mainly untested and still needs to be debugged."},{"line_number":378,"context_line":""},{"line_number":379,"context_line":"Container sharder"},{"line_number":380,"context_line":"^^^^^^^^^^^^^^^^^^^^^^"}],"source_content_type":"text/x-rst","patch_set":5,"id":"1a4dcd0f_05455141","line":377,"updated":"2015-08-12 18:10:17.000000000","message":"not sure if random status updates are part of the spec; if the spec is a todo list, then perhaps","commit_id":"494359620f8642bfef96ef11cdede7d3657ce31d"}]}
