)]}'
{"specs/in_progress/expiring_objects_rework.rst":[{"author":{"_account_id":9625,"name":"Thiago da Silva","email":"thiagodasilva@gmail.com","username":"thiago"},"change_message_id":"5e65616595f7d13211458e49748be4087c83ae29","unresolved":false,"context_lines":[{"line_number":34,"context_line":"This will allow for the container replicater to keep the object counts correct."},{"line_number":35,"context_line":""},{"line_number":36,"context_line":"2.) Allow the auditor to delete objects that have expired during its pass."},{"line_number":37,"context_line":"This will allow for the removal of the object expirer daemon."},{"line_number":38,"context_line":""},{"line_number":39,"context_line":"Implementation Plan"},{"line_number":40,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_d98826c9","line":37,"updated":"2015-09-09 19:09:04.000000000","message":"is the auditor faster than the expirer daemon? I guess I\u0027m missing how having the auditor expire objects would fix the use case where the expirer daemon can\u0027t keep up with the incoming PUTs","commit_id":"5f01e0e66a1691778a173d5d0990d84d6c4a29f5"},{"author":{"_account_id":13297,"name":"Alan Erwin","email":"alan.erwin@rackspace.com","username":"aerwin3"},"change_message_id":"d91b9a759d63d5b55fea6ccec1d72cce1df05b04","unresolved":false,"context_lines":[{"line_number":34,"context_line":"This will allow for the container replicater to keep the object counts correct."},{"line_number":35,"context_line":""},{"line_number":36,"context_line":"2.) Allow the auditor to delete objects that have expired during its pass."},{"line_number":37,"context_line":"This will allow for the removal of the object expirer daemon."},{"line_number":38,"context_line":""},{"line_number":39,"context_line":"Implementation Plan"},{"line_number":40,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":1,"id":"da20952f_94f035cb","line":37,"in_reply_to":"da20952f_d98826c9","updated":"2015-09-09 19:29:38.000000000","message":"The auditors are scaled across the object servers unlike the expirer daemon. The idea for keeping up with demand for expiring object is to use the auditor in conjunction with the update to the container db. *Note that the auditor may take time to remove an object that has expired, but the container and account no longer report it.","commit_id":"5f01e0e66a1691778a173d5d0990d84d6c4a29f5"},{"author":{"_account_id":4608,"name":"Kota Tsuyuzaki","email":"bloodeagle40234@gmail.com","username":"tsuyuzaki-kota"},"change_message_id":"d20af015cd78552f5ac78eed7f968bc55acb2f1b","unresolved":false,"context_lines":[{"line_number":9,"context_line":""},{"line_number":10,"context_line":"Problem description"},{"line_number":11,"context_line":"-------------------"},{"line_number":12,"context_line":"The object expirer daemon does not process objects fast enought"},{"line_number":13,"context_line":"when there are a large amount of files that need to expire."},{"line_number":14,"context_line":"This leads to situtions like:"},{"line_number":15,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"ba15a1d1_4aa5c344","line":12,"updated":"2015-09-17 00:54:24.000000000","message":"(Nits) s/enought/enough/","commit_id":"bbd50d7139a10c386a1bfd9885d532f021395b11"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"80879ad123e55146a0f2d79b698ce8a915f8503c","unresolved":false,"context_lines":[{"line_number":37,"context_line":"This will allow for the container replicater to keep the object counts correct."},{"line_number":38,"context_line":""},{"line_number":39,"context_line":"2.) Allow the auditor to delete objects that have expired during its pass."},{"line_number":40,"context_line":"This will allow for the removal of the object expirer daemon."},{"line_number":41,"context_line":""},{"line_number":42,"context_line":"Implementation Plan"},{"line_number":43,"context_line":"-------------------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"9a1a9d01_fbab6cd4","line":40,"updated":"2015-09-28 20:37:23.000000000","message":"I have some vague worry that we won\u0027t really know how quickly the auditor is reaping the object data - at least with the .expiring_objects account I can go peek and get a feel for where I\u0027m at.","commit_id":"bbd50d7139a10c386a1bfd9885d532f021395b11"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"80879ad123e55146a0f2d79b698ce8a915f8503c","unresolved":false,"context_lines":[{"line_number":48,"context_line":"are before the request time will not be returned."},{"line_number":49,"context_line":""},{"line_number":50,"context_line":"The container replicator will remove the object rows from the container databases"},{"line_number":51,"context_line":"when the expire_at and reclaim age have passed."},{"line_number":52,"context_line":""},{"line_number":53,"context_line":"Once the container updater runs and updates that stats for the containers the"},{"line_number":54,"context_line":"objects that are expired will no longer be considered in the bytes_used or the"}],"source_content_type":"text/x-rst","patch_set":4,"id":"9a1a9d01_fb47ec1a","line":51,"updated":"2015-09-28 20:37:23.000000000","message":"it\u0027s not really \"remove\" the rows?  It\u0027s insert a delete row in it\u0027s place.  I\u0027m not really sure reclaim_age even comes into account - it\u0027s a tombstone row inserted at the time of the expire_at column.  Then the tombstone row will get reaped at the reclaim_age as they always do.\n\nFinding rows with an expire_at value \u003c now seems like the tricky bit.  sqlite has no good way to add a new index to an existing table.","commit_id":"bbd50d7139a10c386a1bfd9885d532f021395b11"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"80879ad123e55146a0f2d79b698ce8a915f8503c","unresolved":false,"context_lines":[{"line_number":52,"context_line":""},{"line_number":53,"context_line":"Once the container updater runs and updates that stats for the containers the"},{"line_number":54,"context_line":"objects that are expired will no longer be considered in the bytes_used or the"},{"line_number":55,"context_line":"object counts in the account database."},{"line_number":56,"context_line":""},{"line_number":57,"context_line":"The object auditor as it makes its pass will remove any expired objects."},{"line_number":58,"context_line":"When the object auditor inspects an object\u0027s metadata, if the X-Delete-At is"}],"source_content_type":"text/x-rst","patch_set":4,"id":"9a1a9d01_db73087f","line":55,"updated":"2015-09-28 20:37:23.000000000","message":"that\u0027s really a property of how expire_at rows are processed, the container-updater would presumably just continue to report whatever the container thinks is current.","commit_id":"bbd50d7139a10c386a1bfd9885d532f021395b11"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"80879ad123e55146a0f2d79b698ce8a915f8503c","unresolved":false,"context_lines":[{"line_number":57,"context_line":"The object auditor as it makes its pass will remove any expired objects."},{"line_number":58,"context_line":"When the object auditor inspects an object\u0027s metadata, if the X-Delete-At is"},{"line_number":59,"context_line":"before the current time, the auditor will delete the object. Due to slow auditor"},{"line_number":60,"context_line":"passes, the cluster will have extra data until the objects get processed."},{"line_number":61,"context_line":""},{"line_number":62,"context_line":"Rollout Plan"},{"line_number":63,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"9a1a9d01_7bd75c23","line":60,"updated":"2015-09-28 20:37:23.000000000","message":"I think this will be mostly reasonable if you ensure that you use the zero-byte-file scan (which is already reading object metadata) and not the checksum scan.\n\nAlso be mindful of:\n\nhttps://bugs.launchpad.net/swift/+bug/1183656","commit_id":"bbd50d7139a10c386a1bfd9885d532f021395b11"},{"author":{"_account_id":4608,"name":"Kota Tsuyuzaki","email":"bloodeagle40234@gmail.com","username":"tsuyuzaki-kota"},"change_message_id":"d20af015cd78552f5ac78eed7f968bc55acb2f1b","unresolved":false,"context_lines":[{"line_number":64,"context_line":""},{"line_number":65,"context_line":"When deploying this change the current expirer deamon can contiue to run until"},{"line_number":66,"context_line":"all objects are removed from the \u0027.expiring_objects\u0027 account. Once that is done"},{"line_number":67,"context_line":"the deamon can be stopped."},{"line_number":68,"context_line":""},{"line_number":69,"context_line":"Also, a script for updating the container databases with the \u0027expire_at\u0027 times"},{"line_number":70,"context_line":"for all the objects with will be created."}],"source_content_type":"text/x-rst","patch_set":4,"id":"ba15a1d1_2de3c9a5","line":67,"updated":"2015-09-17 00:54:24.000000000","message":"Do we need a filter to prevent new coming requests with X-Delete-At during this?\n\nIn this description, container database will be updated after this so it seems to mean we have no way to keep the delete-at info during rolling updates.","commit_id":"bbd50d7139a10c386a1bfd9885d532f021395b11"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"80879ad123e55146a0f2d79b698ce8a915f8503c","unresolved":false,"context_lines":[{"line_number":64,"context_line":""},{"line_number":65,"context_line":"When deploying this change the current expirer deamon can contiue to run until"},{"line_number":66,"context_line":"all objects are removed from the \u0027.expiring_objects\u0027 account. Once that is done"},{"line_number":67,"context_line":"the deamon can be stopped."},{"line_number":68,"context_line":""},{"line_number":69,"context_line":"Also, a script for updating the container databases with the \u0027expire_at\u0027 times"},{"line_number":70,"context_line":"for all the objects with will be created."}],"source_content_type":"text/x-rst","patch_set":4,"id":"9a1a9d01_3bafa495","line":67,"in_reply_to":"ba15a1d1_2de3c9a5","updated":"2015-09-28 20:37:23.000000000","message":"hopefully the object server will continue to handle the x-delete-at headers the same way it always does - the proxy will need to send them until it\u0027s sure all of the containers are handling expiring rows on their own - after all the container servers are updated there\u0027s no need for the proxy to continue to send them.","commit_id":"bbd50d7139a10c386a1bfd9885d532f021395b11"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"80879ad123e55146a0f2d79b698ce8a915f8503c","unresolved":false,"context_lines":[{"line_number":67,"context_line":"the deamon can be stopped."},{"line_number":68,"context_line":""},{"line_number":69,"context_line":"Also, a script for updating the container databases with the \u0027expire_at\u0027 times"},{"line_number":70,"context_line":"for all the objects with will be created."},{"line_number":71,"context_line":""},{"line_number":72,"context_line":""},{"line_number":73,"context_line":"Assignee(s)"}],"source_content_type":"text/x-rst","patch_set":4,"id":"9a1a9d01_fbc60c48","line":70,"updated":"2015-09-28 20:37:23.000000000","message":"this sounds interesting... is the expire_at going to be a separate table maybe?  getting into the objects table is going to mean you have to do a delete/insert just like replication - a whole new row - I think there\u0027s some information about the row (etag and size) that maybe won\u0027t be in the .expiring_objects containers...\n\nI think i could imagine the expire_at being a separate table - that actually solves the index problem.  I think sqlite has some ACID semantics that will allow it to eat .pending files and ensure that when rows with a expire_at value want to get inserted into the objects table they will also for sure get a row in the expire_at table... maybe...","commit_id":"bbd50d7139a10c386a1bfd9885d532f021395b11"}]}