)]}'
{"specs/kilo/approved/archival-framework.rst":[{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"f36807f791d803d979e7eb57e4fe59dd35bc3b58","unresolved":false,"context_lines":[{"line_number":39,"context_line":"This small number of bytes per row isn\u0027t much, but over time, and with millions"},{"line_number":40,"context_line":"of records in the database in all of the tables, can make a definite impact on"},{"line_number":41,"context_line":"performance. Generally, the slimmer we can make a row in a table in the"},{"line_number":42,"context_line":"database, the more records can fit into a single page of memory, and thus fewer"},{"line_number":43,"context_line":"reads are needed for seeks and scans in the database."},{"line_number":44,"context_line":""},{"line_number":45,"context_line":"The second and more important problem related to deleted records has to do"}],"source_content_type":"text/x-rst","patch_set":1,"id":"5a890539_6ee3c2a4","line":42,"updated":"2014-11-28 03:29:30.000000000","message":"I\u0027ve seen the instance usage audit take a long time without even approaching millions (this is the thing in horizon that shows you the monthly usage); we have periodic probing jobs that startup vms and delete them (to ensure stuff is still working) at yahoo in our clusters and these boots and deletes happen under a single tenant. That horizon view was getting borked by this (there weren\u0027t alot of active instances, but a crap-ton of deleted ones...). Just a point of reference...","commit_id":"f7f1b3f4f951aed40f7144aa687cb598b8634125"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"06ccb17c6a5f7c04a6b15b42e5492081ed48e6fc","unresolved":false,"context_lines":[{"line_number":106,"context_line":""},{"line_number":107,"context_line":"Proposed change"},{"line_number":108,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":109,"context_line":""},{"line_number":110,"context_line":"We propose to add a generic archival framework to Nova that will allow a"},{"line_number":111,"context_line":"set of relations in the main Nova database to be deleted and stored in a"},{"line_number":112,"context_line":"separate archival storage system."}],"source_content_type":"text/x-rst","patch_set":1,"id":"5a890539_85f0d692","line":109,"updated":"2014-11-27 16:26:38.000000000","message":"I think we need to explicitly kill the shadow tables as part of this.","commit_id":"f7f1b3f4f951aed40f7144aa687cb598b8634125"},{"author":{"_account_id":5371,"name":"Mathew Odden","email":"locke105@gmail.com","username":"locke105"},"change_message_id":"7ed81aa39d858646d4156b99d4f128c2e1726036","unresolved":false,"context_lines":[{"line_number":106,"context_line":""},{"line_number":107,"context_line":"Proposed change"},{"line_number":108,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":109,"context_line":""},{"line_number":110,"context_line":"We propose to add a generic archival framework to Nova that will allow a"},{"line_number":111,"context_line":"set of relations in the main Nova database to be deleted and stored in a"},{"line_number":112,"context_line":"separate archival storage system."}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a41bdd9_23451bb2","line":109,"in_reply_to":"5a890539_85f0d692","updated":"2015-07-14 22:53:58.000000000","message":"As a smaller cloud operator than Rackspace or HP, I don\u0027t agree. I find that it simplifies management to have the shadow/archive records put into the same database that we know we have available.","commit_id":"f7f1b3f4f951aed40f7144aa687cb598b8634125"},{"author":{"_account_id":8688,"name":"Alexis Lee","email":"openstack@lxsli.co.uk","username":"lxsli"},"change_message_id":"87393310b9764ea4787d8dda30cadb6ce6542717","unresolved":false,"context_lines":[{"line_number":107,"context_line":"Proposed change"},{"line_number":108,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":109,"context_line":""},{"line_number":110,"context_line":"We propose to add a generic archival framework to Nova that will allow a"},{"line_number":111,"context_line":"set of relations in the main Nova database to be deleted and stored in a"},{"line_number":112,"context_line":"separate archival storage system."},{"line_number":113,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"fa32b979_3e12820a","line":110,"updated":"2015-06-26 15:15:01.000000000","message":"not sure why this needs to be part of Nova? As a 3rd party thing, it could have multiple simple implementations sharing an API, rather than a single flexible (complex) in-tree implementation","commit_id":"f7f1b3f4f951aed40f7144aa687cb598b8634125"},{"author":{"_account_id":5371,"name":"Mathew Odden","email":"locke105@gmail.com","username":"locke105"},"change_message_id":"7ed81aa39d858646d4156b99d4f128c2e1726036","unresolved":false,"context_lines":[{"line_number":114,"context_line":"The archival framework should be driver-based, to allow for different"},{"line_number":115,"context_line":"backend storage systems to be used. Some operators may choose to store"},{"line_number":116,"context_line":"archived records in a traditional RDBMS. Others may prefer to use log"},{"line_number":117,"context_line":"storage or a NoSQL data store."},{"line_number":118,"context_line":""},{"line_number":119,"context_line":"The archival framework should have a single method for archival. This method"},{"line_number":120,"context_line":"should accept a set of `nova.objects` that represent the data to be archived."}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a41bdd9_83cf4728","line":117,"updated":"2015-07-14 22:53:58.000000000","message":"I could see this being extremely useful, but again, I like the fact that the current \"cleanup\" or archival process uses the same DB that we can assume is there already to store the data. As long as there are guarantees that the shadow table data can be cleaned up at will by another process and Nova won\u0027t complain, this is a nice simple/compact solution that is easy to manage.\n\nI think I see three cases here:\n\n1) Just cleanup the \"active data\" to ensure performance\n\n2) do the cleanup step above, but archive to the current database instance\n\n3) do the cleanup step in 1, but archive to an external service","commit_id":"f7f1b3f4f951aed40f7144aa687cb598b8634125"},{"author":{"_account_id":8688,"name":"Alexis Lee","email":"openstack@lxsli.co.uk","username":"lxsli"},"change_message_id":"87393310b9764ea4787d8dda30cadb6ce6542717","unresolved":false,"context_lines":[{"line_number":117,"context_line":"storage or a NoSQL data store."},{"line_number":118,"context_line":""},{"line_number":119,"context_line":"The archival framework should have a single method for archival. This method"},{"line_number":120,"context_line":"should accept a set of `nova.objects` that represent the data to be archived."},{"line_number":121,"context_line":"The framework should be store the records in backend storage and then fully"},{"line_number":122,"context_line":"delete the records from active storage in the Nova database. Data should be"},{"line_number":123,"context_line":"deleted from active storage in a fully transactional manner -- i.e. no orphaned"}],"source_content_type":"text/x-rst","patch_set":1,"id":"fa32b979_5e2ae630","line":120,"updated":"2015-06-26 15:15:01.000000000","message":"It\u0027d be easier to make the framework 3rd party if the objects were serialised to JSON, with o.vo versions embedded","commit_id":"f7f1b3f4f951aed40f7144aa687cb598b8634125"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"ab97120af76fee61b1ee17c9b1ae4e958cc70306","unresolved":false,"context_lines":[{"line_number":117,"context_line":"storage or a NoSQL data store."},{"line_number":118,"context_line":""},{"line_number":119,"context_line":"The archival framework should have a single method for archival. This method"},{"line_number":120,"context_line":"should accept a set of `nova.objects` that represent the data to be archived."},{"line_number":121,"context_line":"The framework should be store the records in backend storage and then fully"},{"line_number":122,"context_line":"delete the records from active storage in the Nova database. Data should be"},{"line_number":123,"context_line":"deleted from active storage in a fully transactional manner -- i.e. no orphaned"}],"source_content_type":"text/x-rst","patch_set":1,"id":"ba3cc151_5cfd0e07","line":120,"in_reply_to":"fa32b979_5e2ae630","updated":"2015-07-10 18:06:13.000000000","message":"Yes, JSON makes more sense here, since this is crossing the boundary of Nova. The delete operations in Nova could pass the objects to a single method that handles the serialization so that there would be no need to convert all over the codebase.","commit_id":"f7f1b3f4f951aed40f7144aa687cb598b8634125"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"06ccb17c6a5f7c04a6b15b42e5492081ed48e6fc","unresolved":false,"context_lines":[{"line_number":129,"context_line":"on object identifiers and/or a timestamp."},{"line_number":130,"context_line":""},{"line_number":131,"context_line":"A REST API for this archival framework is deliberately NOT in scope for this"},{"line_number":132,"context_line":"blueprint."},{"line_number":133,"context_line":""},{"line_number":134,"context_line":"A utility (perhaps as an addition to the `nova-manage` CLI utility) should be"},{"line_number":135,"context_line":"provided to archive records in existing databases using a batch processing,"}],"source_content_type":"text/x-rst","patch_set":1,"id":"5a890539_857536fc","line":132,"updated":"2014-11-27 16:26:38.000000000","message":"If we have a framework for looking up and deleting these, thats all fine.\n\nWhat is the \"reference implementation\" you will use?","commit_id":"f7f1b3f4f951aed40f7144aa687cb598b8634125"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"f36807f791d803d979e7eb57e4fe59dd35bc3b58","unresolved":false,"context_lines":[{"line_number":129,"context_line":"on object identifiers and/or a timestamp."},{"line_number":130,"context_line":""},{"line_number":131,"context_line":"A REST API for this archival framework is deliberately NOT in scope for this"},{"line_number":132,"context_line":"blueprint."},{"line_number":133,"context_line":""},{"line_number":134,"context_line":"A utility (perhaps as an addition to the `nova-manage` CLI utility) should be"},{"line_number":135,"context_line":"provided to archive records in existing databases using a batch processing,"}],"source_content_type":"text/x-rst","patch_set":1,"id":"5a890539_2e87fab7","line":132,"in_reply_to":"5a890539_857536fc","updated":"2014-11-28 03:29:30.000000000","message":"I\u0027d personally recommend something like log files; those are the most braindead and can be placed in any suitable backend that has a filesystem like interface (HDFS for example).\n\nKnowing how yahoo and others work, this is what is commonly done with apache access logs... and it seems appropriate here (just call it purged logs instead of access logs...)","commit_id":"f7f1b3f4f951aed40f7144aa687cb598b8634125"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"ab97120af76fee61b1ee17c9b1ae4e958cc70306","unresolved":false,"context_lines":[{"line_number":138,"context_line":""},{"line_number":139,"context_line":"Database schema migration scripts should be provided that check for the lack"},{"line_number":140,"context_line":"of deleted records in all tables of the Nova database -- which will indicate"},{"line_number":141,"context_line":"that the archival utility program has completed the archival of all records --"},{"line_number":142,"context_line":"and then proceed to remove the `deleted` and `deleted_at` columns from the"},{"line_number":143,"context_line":"database schemas, and remove the `deleted` columns from all unique constraints."},{"line_number":144,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"ba3cc151_1c6c66cf","line":141,"updated":"2015-07-10 18:06:13.000000000","message":"s/completed the archival/completed the archiving","commit_id":"f7f1b3f4f951aed40f7144aa687cb598b8634125"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"06ccb17c6a5f7c04a6b15b42e5492081ed48e6fc","unresolved":false,"context_lines":[{"line_number":140,"context_line":"of deleted records in all tables of the Nova database -- which will indicate"},{"line_number":141,"context_line":"that the archival utility program has completed the archival of all records --"},{"line_number":142,"context_line":"and then proceed to remove the `deleted` and `deleted_at` columns from the"},{"line_number":143,"context_line":"database schemas, and remove the `deleted` columns from all unique constraints."},{"line_number":144,"context_line":""},{"line_number":145,"context_line":"Alternatives"},{"line_number":146,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"5a890539_251002a8","line":143,"updated":"2014-11-27 16:26:38.000000000","message":"shouldn\u0027t this be dealt with using the regular online schema migration pattern?\n* all new deletes start using the new delete system\n* slow move old deletes into the new system, using objects \"data migration\" CLI\n* only then do the DB \"contract\"","commit_id":"f7f1b3f4f951aed40f7144aa687cb598b8634125"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"f36807f791d803d979e7eb57e4fe59dd35bc3b58","unresolved":false,"context_lines":[{"line_number":150,"context_line":"of marking a record as deleted and constructing unique constraints around"},{"line_number":151,"context_line":"primary keys and a `deleted` column. The second alternative is to move"},{"line_number":152,"context_line":"deleted records to a set of shadow tables using a \"purge\" process, and"},{"line_number":153,"context_line":"potentially joining main tables to shadown tables. Both of these alternative"},{"line_number":154,"context_line":"solutions are problematic due to the complexity they add to the Nova database"},{"line_number":155,"context_line":"layer and the fact that they treat all data -- old and active -- in the same"},{"line_number":156,"context_line":"way, and insist on storing each type of data using the same storage mechanism."}],"source_content_type":"text/x-rst","patch_set":1,"id":"5a890539_6efae244","line":153,"updated":"2014-11-28 03:29:30.000000000","message":"shadown -\u003e shadow (sp)","commit_id":"f7f1b3f4f951aed40f7144aa687cb598b8634125"},{"author":{"_account_id":11816,"name":"mike bayer","display_name":"Mike Bayer","email":"mike_mp@zzzcomputing.com","username":"zzzeek","status":"Red Hat"},"change_message_id":"07de1b51577c66c178d949b5aacdb480414f36b2","unresolved":false,"context_lines":[{"line_number":176,"context_line":"---------------"},{"line_number":177,"context_line":""},{"line_number":178,"context_line":"None."},{"line_number":179,"context_line":""},{"line_number":180,"context_line":"Notifications impact"},{"line_number":181,"context_line":"--------------------"},{"line_number":182,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"5a890539_e83313f5","line":179,"updated":"2014-11-29 00:28:16.000000000","message":"thinking of the use case of the archival system being generalized, so that all apps which currently have a \"soft delete\" system can make use of this;  when I\u0027ve worked with the \"archive old data\" use case out in financial-world, there was a security impact which was that we often had to encrypt the archive files, if the databases contained any PII or other sensitive info.  \n\nI don\u0027t know what kind of \"sensitive\" info Nova might have, e.g. password hashes, security tokens, that kind of thing?  If so, or if we intend to generalize the system to other applications that do, we might want to make a note about such tokens not being written directly in the archives, or otherwise protected in some way, assuming the security surrounding the archive is less than that of the relational DB.","commit_id":"f7f1b3f4f951aed40f7144aa687cb598b8634125"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"06ccb17c6a5f7c04a6b15b42e5492081ed48e6fc","unresolved":false,"context_lines":[{"line_number":192,"context_line":""},{"line_number":193,"context_line":"Performance improvements are expected for the main Nova database due to the"},{"line_number":194,"context_line":"decreased query complexity and schema narrowing."},{"line_number":195,"context_line":""},{"line_number":196,"context_line":"Other deployer impact"},{"line_number":197,"context_line":"---------------------"},{"line_number":198,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"5a890539_253f22d8","line":195,"updated":"2014-11-27 16:26:38.000000000","message":"What about things like the Nova API where you can list your deleted instances?\n\nWhat about the periodic tasks in the compute node where we check for running instances that are actually marked delete in the database, etc?\n\nI presume they will have much worse performance if you need to look up the data in some archive system?","commit_id":"f7f1b3f4f951aed40f7144aa687cb598b8634125"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"ab97120af76fee61b1ee17c9b1ae4e958cc70306","unresolved":false,"context_lines":[{"line_number":192,"context_line":""},{"line_number":193,"context_line":"Performance improvements are expected for the main Nova database due to the"},{"line_number":194,"context_line":"decreased query complexity and schema narrowing."},{"line_number":195,"context_line":""},{"line_number":196,"context_line":"Other deployer impact"},{"line_number":197,"context_line":"---------------------"},{"line_number":198,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"ba3cc151_1c24665e","line":195,"in_reply_to":"5a890539_253f22d8","updated":"2015-07-10 18:06:13.000000000","message":"I would imagine that those API calls would no longer do anything useful, and should probably be deprecated. Listing deleted instances should part of the archive system.","commit_id":"f7f1b3f4f951aed40f7144aa687cb598b8634125"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"06ccb17c6a5f7c04a6b15b42e5492081ed48e6fc","unresolved":false,"context_lines":[{"line_number":201,"context_line":"deployer takes with regards to updating the code base of Nova, running the"},{"line_number":202,"context_line":"`nova-manage archive` commands, and then the corresponding `nova-manage db"},{"line_number":203,"context_line":"expand/migrate/contract` commands should be provided in the developer reference"},{"line_number":204,"context_line":"and included in the operators manual."},{"line_number":205,"context_line":""},{"line_number":206,"context_line":"Developer impact"},{"line_number":207,"context_line":"----------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"5a890539_85e376a3","line":204,"updated":"2014-11-27 16:26:38.000000000","message":"I think we need a deprecation window for the existing soft delete.\n\nWe certainly have lots of operational tools build around the assumption that deleted instances are in the DB. Its going to take to re-write all those.","commit_id":"f7f1b3f4f951aed40f7144aa687cb598b8634125"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"06ccb17c6a5f7c04a6b15b42e5492081ed48e6fc","unresolved":false,"context_lines":[{"line_number":232,"context_line":"- Have a separate thread within the conductor that manages the archive queue"},{"line_number":233,"context_line":"  and have the `archive_records` RPC API command add an archival request to"},{"line_number":234,"context_line":"  the queue, to be processed by a separate worker threads in a low-priority"},{"line_number":235,"context_line":"  way."},{"line_number":236,"context_line":""},{"line_number":237,"context_line":"- Write the implementation for a simple log-file driver that writes archived"},{"line_number":238,"context_line":"  records (in the `nova.objects` JSON-serialization format) to disk, and has"}],"source_content_type":"text/x-rst","patch_set":1,"id":"5a890539_05e786ad","line":235,"updated":"2014-11-27 16:26:38.000000000","message":"Do you mean like compute_task_api is today? with a separate RPC api?","commit_id":"f7f1b3f4f951aed40f7144aa687cb598b8634125"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"06ccb17c6a5f7c04a6b15b42e5492081ed48e6fc","unresolved":false,"context_lines":[{"line_number":239,"context_line":"  a naive search records facility to find archived records."},{"line_number":240,"context_line":""},{"line_number":241,"context_line":"- Write the `nova-manage archive` utility that will go through all the tables"},{"line_number":242,"context_line":"  in the Nova database and archive all deleted records in batches."},{"line_number":243,"context_line":""},{"line_number":244,"context_line":"- Remove the `SoftDeleteMixin` class from all models in the Nova sqlalchemy"},{"line_number":245,"context_line":"  models.py file. Remove the `deleted` column from any join conditions in"}],"source_content_type":"text/x-rst","patch_set":1,"id":"5a890539_059ea607","line":242,"updated":"2014-11-27 16:26:38.000000000","message":"This should be part of the \"data migration\" tool we are adding for the flavor work, so it fits the standard DB migration system.","commit_id":"f7f1b3f4f951aed40f7144aa687cb598b8634125"}]}