)]}'
{"specs/liberty/approved/purge-soft-deleted-rows.rst":[{"author":{"_account_id":12898,"name":"Tony Breeds","email":"tony@bakeyournoodle.com","username":"tonyb"},"change_message_id":"2f9a2aaeb566889efa279715ac465b56c0b11962","unresolved":false,"context_lines":[{"line_number":66,"context_line":"This will impact API consumers. However (initially) as the operator has to"},{"line_number":67,"context_line":"enable this manually, or (after a cycle) has been properly notified, this"},{"line_number":68,"context_line":"impact should be well understood. It will also be simple for an operator to"},{"line_number":69,"context_line":"disable this by setting \u0027soft_delete_purge_days\u0027 to -1."},{"line_number":70,"context_line":""},{"line_number":71,"context_line":"Posting to get preliminary feedback on the scope of this spec."},{"line_number":72,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"7a016987_fb4330ad","line":69,"updated":"2015-05-21 22:10:40.000000000","message":"InstanceList.get_active_by_window_joined() doesn\u0027t explicitly have a  read_deleted flag BUT it does return data for deleted instances.  That method is called for os-simple-tenant-usage API.  I suspect that there are other calls where we query the database and don\u0027t explicitly flag (not)deleted.","commit_id":"6ce1e4cecded51cbb08cc5bcd97ede043c4bded2"},{"author":{"_account_id":8688,"name":"Alexis Lee","email":"openstack@lxsli.co.uk","username":"lxsli"},"change_message_id":"8ac5d2d62074af6f857246c2cdb9bd4fe81b164e","unresolved":false,"context_lines":[{"line_number":66,"context_line":"This will impact API consumers. However (initially) as the operator has to"},{"line_number":67,"context_line":"enable this manually, or (after a cycle) has been properly notified, this"},{"line_number":68,"context_line":"impact should be well understood. It will also be simple for an operator to"},{"line_number":69,"context_line":"disable this by setting \u0027soft_delete_purge_days\u0027 to -1."},{"line_number":70,"context_line":""},{"line_number":71,"context_line":"Posting to get preliminary feedback on the scope of this spec."},{"line_number":72,"context_line":""}],"source_content_type":"text/x-rst","patch_set":3,"id":"7a016987_cc055514","line":69,"in_reply_to":"7a016987_fb4330ad","updated":"2015-05-22 06:48:57.000000000","message":"Nice, I missed where session.query was used directly. Since that query is on the Instance table, it doesn\u0027t make a lot of difference. I also missed a query on VirtualInterface though, so thanks!","commit_id":"6ce1e4cecded51cbb08cc5bcd97ede043c4bded2"},{"author":{"_account_id":2271,"name":"Michael Still","email":"mikal@stillhq.com","username":"mikalstill"},"change_message_id":"2ba87e943bd8aad0bf1bd20c8475d1575c938e0d","unresolved":false,"context_lines":[{"line_number":77,"context_line":"is certainly simpler."},{"line_number":78,"context_line":""},{"line_number":79,"context_line":"We could force operators to take this behaviour. That would increase the API"},{"line_number":80,"context_line":"impact considerably."},{"line_number":81,"context_line":""},{"line_number":82,"context_line":"Data model impact"},{"line_number":83,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"7a016987_87fd0470","line":80,"updated":"2015-05-21 17:06:26.000000000","message":"So, wouldn\u0027t an operator want to control what time of day this ran to ensure it happens during offpeak times? I wonder if that means it should just be via nova-manage, and then they can kick it off with a cron job? Or do we think we need to make sure it actually happens?","commit_id":"6ce1e4cecded51cbb08cc5bcd97ede043c4bded2"},{"author":{"_account_id":12898,"name":"Tony Breeds","email":"tony@bakeyournoodle.com","username":"tonyb"},"change_message_id":"2f9a2aaeb566889efa279715ac465b56c0b11962","unresolved":false,"context_lines":[{"line_number":77,"context_line":"is certainly simpler."},{"line_number":78,"context_line":""},{"line_number":79,"context_line":"We could force operators to take this behaviour. That would increase the API"},{"line_number":80,"context_line":"impact considerably."},{"line_number":81,"context_line":""},{"line_number":82,"context_line":"Data model impact"},{"line_number":83,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"7a016987_db15149c","line":80,"in_reply_to":"7a016987_87fd0470","updated":"2015-05-21 22:10:40.000000000","message":"Hmm okay so if we added a nova-manage command can\u0027t we pass a timestanp there:\n   nova-manage purge_deleted 20150101T00000Z (yes it\u0027s ugly)\nThis would allow operators to run the purge command when they knew they\u0027d operated on the data (eg for customer/user billing)","commit_id":"6ce1e4cecded51cbb08cc5bcd97ede043c4bded2"},{"author":{"_account_id":8688,"name":"Alexis Lee","email":"openstack@lxsli.co.uk","username":"lxsli"},"change_message_id":"8ac5d2d62074af6f857246c2cdb9bd4fe81b164e","unresolved":false,"context_lines":[{"line_number":77,"context_line":"is certainly simpler."},{"line_number":78,"context_line":""},{"line_number":79,"context_line":"We could force operators to take this behaviour. That would increase the API"},{"line_number":80,"context_line":"impact considerably."},{"line_number":81,"context_line":""},{"line_number":82,"context_line":"Data model impact"},{"line_number":83,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"7a016987_0f17df48","line":80,"in_reply_to":"7a016987_db15149c","updated":"2015-05-22 06:48:57.000000000","message":"Adding it to nova-manage is an option. That\u0027s certainly better than not offering any help at all. However that makes it hard to default-enable this feature (after a notification cycle) to help out amateur operators. We\u0027re aiming for sensible defaults and I think that should include not filling up the DB.","commit_id":"6ce1e4cecded51cbb08cc5bcd97ede043c4bded2"},{"author":{"_account_id":2271,"name":"Michael Still","email":"mikal@stillhq.com","username":"mikalstill"},"change_message_id":"2ba87e943bd8aad0bf1bd20c8475d1575c938e0d","unresolved":false,"context_lines":[{"line_number":82,"context_line":"Data model impact"},{"line_number":83,"context_line":"-----------------"},{"line_number":84,"context_line":""},{"line_number":85,"context_line":"Changes which require modifications to the data model often have a wider impact"},{"line_number":86,"context_line":"on the system.  The community often has strong opinions on how the data model"},{"line_number":87,"context_line":"should be evolved, from both a functional and performance perspective. It is"},{"line_number":88,"context_line":"therefore important to capture and gain agreement as early as possible on any"}],"source_content_type":"text/x-rst","patch_set":3,"id":"7a016987_c7e09ccf","line":85,"updated":"2015-05-21 17:06:26.000000000","message":"Boilerplate below here.","commit_id":"6ce1e4cecded51cbb08cc5bcd97ede043c4bded2"},{"author":{"_account_id":9420,"name":"Matthew Gilliard","email":"matthew.gilliard@hp.com","username":"gilliard"},"change_message_id":"f8800c828bf89df65149093072b39c5880aae84d","unresolved":false,"context_lines":[{"line_number":51,"context_line":"Project Priority"},{"line_number":52,"context_line":"----------------"},{"line_number":53,"context_line":""},{"line_number":54,"context_line":"Yes!"},{"line_number":55,"context_line":""},{"line_number":56,"context_line":"Proposed change"},{"line_number":57,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":5,"id":"7a016987_2eec6428","line":54,"updated":"2015-05-27 14:44:54.000000000","message":"I think the question is supposed to be: Which project priority does this align with?","commit_id":"aebf6217a4c1d17543713ffae6152ebb499fcdb7"},{"author":{"_account_id":9420,"name":"Matthew Gilliard","email":"matthew.gilliard@hp.com","username":"gilliard"},"change_message_id":"f8800c828bf89df65149093072b39c5880aae84d","unresolved":false,"context_lines":[{"line_number":64,"context_line":"to not severely degrade performance while the purge is running."},{"line_number":65,"context_line":""},{"line_number":66,"context_line":"Every 30 mins a Nova periodic task will delete \u0027soft_delete_purge_count\u0027"},{"line_number":67,"context_line":"soft-deleted rows which are older than \u0027soft_delete_purge_days\u0027."},{"line_number":68,"context_line":""},{"line_number":69,"context_line":"Alternatives"},{"line_number":70,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":5,"id":"7a016987_2e634443","line":67,"updated":"2015-05-27 14:44:54.000000000","message":"I guess we could log a warning if this isn\u0027t keeping up.  We can only remove a maximum of 48*soft_delete_purge_count rows per day with this scheme (which sounds perfectly fine, but could be even awesomer for operators of busy clouds).","commit_id":"aebf6217a4c1d17543713ffae6152ebb499fcdb7"},{"author":{"_account_id":9420,"name":"Matthew Gilliard","email":"matthew.gilliard@hp.com","username":"gilliard"},"change_message_id":"f8800c828bf89df65149093072b39c5880aae84d","unresolved":false,"context_lines":[{"line_number":69,"context_line":"Alternatives"},{"line_number":70,"context_line":"------------"},{"line_number":71,"context_line":""},{"line_number":72,"context_line":"We could just use a Boolean. This could make the API impact more obvious and"},{"line_number":73,"context_line":"is slightly simpler. It\u0027s also much less useful to operators who need"},{"line_number":74,"context_line":"soft-deleted rows to be retained for a limited amount of time."},{"line_number":75,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"7a016987_4e9b20ca","line":72,"updated":"2015-05-27 14:44:54.000000000","message":"A boolean controlling what?","commit_id":"aebf6217a4c1d17543713ffae6152ebb499fcdb7"},{"author":{"_account_id":6062,"name":"jichenjc","email":"jichenjc@cn.ibm.com","username":"jichenjc"},"change_message_id":"a71fd1bb3a7acabfbf570835b13d2b5135aa27b3","unresolved":false,"context_lines":[{"line_number":57,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":58,"context_line":""},{"line_number":59,"context_line":"Add a config option \u0027soft_delete_purge_days\u0027, defaulting to -1 initially (off)"},{"line_number":60,"context_line":"then defaulting to 60 after one cycle."},{"line_number":61,"context_line":""},{"line_number":62,"context_line":"Add a config option \u0027soft_delete_purge_count\u0027, defaulting to 2000. This value"},{"line_number":63,"context_line":"should be high enough to keep up with the soft-deletion rate while low enough"}],"source_content_type":"text/x-rst","patch_set":6,"id":"7a016987_45065621","line":60,"updated":"2015-05-27 16:48:19.000000000","message":"https://review.openstack.org/#/c/109201/\nhttps://review.openstack.org/#/c/110875/\n\nare those related to the proposal here? \n\ncommit message:\nCurrently, We can\u0027t delete instance system metadata after instance\nis deleted because of following reason:\n1) We might have \u0027changed-since\u0027 request which will access\nthe soft deleted system_metadata\n2) We might has period audit function which also access the\nsoft deleted system_metadata\n\nseems #1 still need to be considered?","commit_id":"b78b778b0a49d17a2a7a65039eef9870e4094973"},{"author":{"_account_id":6062,"name":"jichenjc","email":"jichenjc@cn.ibm.com","username":"jichenjc"},"change_message_id":"9a7c8eb78b2145522bffa0660e5a325cbafd2796","unresolved":false,"context_lines":[{"line_number":57,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":58,"context_line":""},{"line_number":59,"context_line":"Add a config option \u0027soft_delete_purge_days\u0027, defaulting to -1 initially (off)"},{"line_number":60,"context_line":"then defaulting to 60 after one cycle."},{"line_number":61,"context_line":""},{"line_number":62,"context_line":"Add a config option \u0027soft_delete_purge_count\u0027, defaulting to 2000. This value"},{"line_number":63,"context_line":"should be high enough to keep up with the soft-deletion rate while low enough"}],"source_content_type":"text/x-rst","patch_set":6,"id":"7a016987_d13c21c1","line":60,"in_reply_to":"7a016987_1612af09","updated":"2015-05-28 15:05:14.000000000","message":"The instance API support \u0027change-since\u0027\n\nhttps://github.com/openstack/nova/blob/master/nova/api/openstack/compute/plugins/v3/servers.py#L293\n\njust FYI ,nothing special :)","commit_id":"b78b778b0a49d17a2a7a65039eef9870e4094973"},{"author":{"_account_id":8688,"name":"Alexis Lee","email":"openstack@lxsli.co.uk","username":"lxsli"},"change_message_id":"0c4174b6f771998beb23e2cc22fc7db861a46175","unresolved":false,"context_lines":[{"line_number":57,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":58,"context_line":""},{"line_number":59,"context_line":"Add a config option \u0027soft_delete_purge_days\u0027, defaulting to -1 initially (off)"},{"line_number":60,"context_line":"then defaulting to 60 after one cycle."},{"line_number":61,"context_line":""},{"line_number":62,"context_line":"Add a config option \u0027soft_delete_purge_count\u0027, defaulting to 2000. This value"},{"line_number":63,"context_line":"should be high enough to keep up with the soft-deletion rate while low enough"}],"source_content_type":"text/x-rst","patch_set":6,"id":"7a016987_67bdb3d6","line":60,"in_reply_to":"7a016987_45065621","updated":"2015-05-28 13:21:37.000000000","message":"Yes, looks related. I\u0027m happy with both patches. What are you asking please? Did you want me to add them as dependencies for this spec?","commit_id":"b78b778b0a49d17a2a7a65039eef9870e4094973"},{"author":{"_account_id":6062,"name":"jichenjc","email":"jichenjc@cn.ibm.com","username":"jichenjc"},"change_message_id":"d9f888068bcb4e411b005706f7112855e0aabf62","unresolved":false,"context_lines":[{"line_number":57,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":58,"context_line":""},{"line_number":59,"context_line":"Add a config option \u0027soft_delete_purge_days\u0027, defaulting to -1 initially (off)"},{"line_number":60,"context_line":"then defaulting to 60 after one cycle."},{"line_number":61,"context_line":""},{"line_number":62,"context_line":"Add a config option \u0027soft_delete_purge_count\u0027, defaulting to 2000. This value"},{"line_number":63,"context_line":"should be high enough to keep up with the soft-deletion rate while low enough"}],"source_content_type":"text/x-rst","patch_set":6,"id":"7a016987_962fbf87","line":60,"in_reply_to":"7a016987_67bdb3d6","updated":"2015-05-28 14:47:40.000000000","message":"not so much asking, still reading the spec :)\n\nonly FYI and the reason why the soft_delete can\u0027t be deleted directly due to \u0027changed-since\u0027 in case you need~\n\nthanks","commit_id":"b78b778b0a49d17a2a7a65039eef9870e4094973"},{"author":{"_account_id":8688,"name":"Alexis Lee","email":"openstack@lxsli.co.uk","username":"lxsli"},"change_message_id":"4c13abe11ae2338f15e85c4932d99af1fa34df39","unresolved":false,"context_lines":[{"line_number":57,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":58,"context_line":""},{"line_number":59,"context_line":"Add a config option \u0027soft_delete_purge_days\u0027, defaulting to -1 initially (off)"},{"line_number":60,"context_line":"then defaulting to 60 after one cycle."},{"line_number":61,"context_line":""},{"line_number":62,"context_line":"Add a config option \u0027soft_delete_purge_count\u0027, defaulting to 2000. This value"},{"line_number":63,"context_line":"should be high enough to keep up with the soft-deletion rate while low enough"}],"source_content_type":"text/x-rst","patch_set":6,"id":"7a016987_1612af09","line":60,"in_reply_to":"7a016987_962fbf87","updated":"2015-05-28 14:51:54.000000000","message":"OK, thanks for looking!\n\nBTW exactly which query from db/sqlalchemy/api.py do you mean by \u0027changed-since\u0027?","commit_id":"b78b778b0a49d17a2a7a65039eef9870e4094973"},{"author":{"_account_id":8688,"name":"Alexis Lee","email":"openstack@lxsli.co.uk","username":"lxsli"},"change_message_id":"0a9a03980a557c1872e29b75709a08ceafdb8cb0","unresolved":false,"context_lines":[{"line_number":57,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":58,"context_line":""},{"line_number":59,"context_line":"Add a config option \u0027soft_delete_purge_days\u0027, defaulting to -1 initially (off)"},{"line_number":60,"context_line":"then defaulting to 60 after one cycle."},{"line_number":61,"context_line":""},{"line_number":62,"context_line":"Add a config option \u0027soft_delete_purge_count\u0027, defaulting to 2000. This value"},{"line_number":63,"context_line":"should be high enough to keep up with the soft-deletion rate while low enough"}],"source_content_type":"text/x-rst","patch_set":6,"id":"5afe65bd_8f65591f","line":60,"in_reply_to":"7a016987_d13c21c1","updated":"2015-06-09 10:29:41.000000000","message":"I checked this, changes-since tests \"Instance.updated_at \u003e\u003d changes_since\".\n\nThe same DB query (instance_get_all_by_filters_sort) also supports a soft_deleted flag though, which is how it got included in my list on line 32 :)","commit_id":"b78b778b0a49d17a2a7a65039eef9870e4094973"},{"author":{"_account_id":1849,"name":"Joe Gordon","email":"joe.gordon0@gmail.com","username":"jogo"},"change_message_id":"fac394ea251cb47a994fbee923934e8ad35fd536","unresolved":false,"context_lines":[{"line_number":66,"context_line":"Every 30 mins a Nova periodic task will delete \u0027soft_delete_purge_count\u0027"},{"line_number":67,"context_line":"soft-deleted rows which are older than \u0027soft_delete_purge_days\u0027. The number of"},{"line_number":68,"context_line":"deleted rows will be logged at INFO. Only rows from the FixedIp, Instance,"},{"line_number":69,"context_line":"InstanceTypes and VirtualInterface tables will be deleted, blueprint"},{"line_number":70,"context_line":"no-more-soft-delete proposes removing the column from all other tables."},{"line_number":71,"context_line":""},{"line_number":72,"context_line":"There is a potential issue around related tables and foreign-key constraints."}],"source_content_type":"text/x-rst","patch_set":6,"id":"5afe65bd_32a6d710","line":69,"updated":"2015-06-04 05:31:35.000000000","message":"Why only these rows?","commit_id":"b78b778b0a49d17a2a7a65039eef9870e4094973"},{"author":{"_account_id":1849,"name":"Joe Gordon","email":"joe.gordon0@gmail.com","username":"jogo"},"change_message_id":"fac394ea251cb47a994fbee923934e8ad35fd536","unresolved":false,"context_lines":[{"line_number":99,"context_line":"it with cron. This is probably closest to what operators are already doing and"},{"line_number":100,"context_line":"presents no real barrier to large operators. It does mean a default Nova"},{"line_number":101,"context_line":"install is a time-bomb however, which fails the \"sensible defaults\" test and"},{"line_number":102,"context_line":"may provide a nasty surprise to smaller or amateur operators."},{"line_number":103,"context_line":""},{"line_number":104,"context_line":"Data model impact"},{"line_number":105,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":6,"id":"5afe65bd_5278338f","line":102,"updated":"2015-06-04 05:31:35.000000000","message":"I actually prefer this, at least as a first pass.\n\nI think its reasonable to say that installing nova requires setting up a few other things as well. You realistically cannot run nova without log collection etc.\n\nThis could be an option instead of the shadow table solution (which appears to never have really worked).","commit_id":"b78b778b0a49d17a2a7a65039eef9870e4094973"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"63cb0c356a3113dd97a75dbabb1553ba9272e2a3","unresolved":false,"context_lines":[{"line_number":99,"context_line":"it with cron. This is probably closest to what operators are already doing and"},{"line_number":100,"context_line":"presents no real barrier to large operators. It does mean a default Nova"},{"line_number":101,"context_line":"install is a time-bomb however, which fails the \"sensible defaults\" test and"},{"line_number":102,"context_line":"may provide a nasty surprise to smaller or amateur operators."},{"line_number":103,"context_line":""},{"line_number":104,"context_line":"Data model impact"},{"line_number":105,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":6,"id":"1af86dd1_a893d8a2","line":102,"in_reply_to":"5afe65bd_5278338f","updated":"2015-06-12 14:38:04.000000000","message":"I like this approach, too. It seems much more direct than adding yet another config option and another periodic task, both of which will later have to be deprecated and then removed.","commit_id":"b78b778b0a49d17a2a7a65039eef9870e4094973"}]}