)]}'
{"doc/source/upgrade.rst":[{"author":{"_account_id":20140,"name":"Karthik Prabhu Vinod","email":"karthik.prabhu.vinod@intel.com","username":"kprabhuv"},"change_message_id":"809f1576ad4a682bf3e1488cfcd3378b4deb3d74","unresolved":false,"context_lines":[{"line_number":119,"context_line":"      upgraded in lockstep so n-1 cells compatibility does not work."},{"line_number":120,"context_line":""},{"line_number":121,"context_line":"Online Configuration Reload"},{"line_number":122,"context_line":"    During the upgrade, we pin new serves at the older RPC version. When all"},{"line_number":123,"context_line":"    services are updated to use newer code, we need to unpin them so we are"},{"line_number":124,"context_line":"    able to use any new functionality."},{"line_number":125,"context_line":"    To avoid having to restart the service, using the current SIGHUP signal"}],"source_content_type":"text/x-rst","patch_set":1,"id":"ba5da102_e41ddc73","line":122,"range":{"start_line":122,"start_character":35,"end_line":122,"end_character":41},"updated":"2016-11-07 16:10:20.000000000","message":"s/serves/services","commit_id":"e6138c23e4e32c9398b861b77a35abbd4cde3e45"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"3fdbebbbe861c6fc9b30c00dc8e214edf83478d5","unresolved":false,"context_lines":[{"line_number":119,"context_line":"      upgraded in lockstep so n-1 cells compatibility does not work."},{"line_number":120,"context_line":""},{"line_number":121,"context_line":"Online Configuration Reload"},{"line_number":122,"context_line":"    During the upgrade, we pin new serves at the older RPC version. When all"},{"line_number":123,"context_line":"    services are updated to use newer code, we need to unpin them so we are"},{"line_number":124,"context_line":"    able to use any new functionality."},{"line_number":125,"context_line":"    To avoid having to restart the service, using the current SIGHUP signal"}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a629dbe_e73af7cc","line":122,"range":{"start_line":122,"start_character":35,"end_line":122,"end_character":41},"in_reply_to":"ba5da102_e41ddc73","updated":"2016-11-08 11:12:03.000000000","message":"Already fixed in #3 patchset.","commit_id":"e6138c23e4e32c9398b861b77a35abbd4cde3e45"},{"author":{"_account_id":20140,"name":"Karthik Prabhu Vinod","email":"karthik.prabhu.vinod@intel.com","username":"kprabhuv"},"change_message_id":"809f1576ad4a682bf3e1488cfcd3378b4deb3d74","unresolved":false,"context_lines":[{"line_number":128,"context_line":""},{"line_number":129,"context_line":"Graceful service shutdown"},{"line_number":130,"context_line":"    Many nova services are python processes listening for messages on a"},{"line_number":131,"context_line":"    AMQP queue, including nova-compute. When sending the process the SIGTERM"},{"line_number":132,"context_line":"    the process stops getting new work from its queue, completes any"},{"line_number":133,"context_line":"    outstanding work, then terminates. During this process, messages can be"},{"line_number":134,"context_line":"    left on the queue for when the python process starts back up."}],"source_content_type":"text/x-rst","patch_set":1,"id":"ba5da102_fbe70e44","line":131,"range":{"start_line":131,"start_character":40,"end_line":131,"end_character":44},"updated":"2016-11-07 16:10:20.000000000","message":"I think we can rephrase this as \"When an operator sends an SIGTERM signal to the process, it stops\"","commit_id":"e6138c23e4e32c9398b861b77a35abbd4cde3e45"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"3fdbebbbe861c6fc9b30c00dc8e214edf83478d5","unresolved":false,"context_lines":[{"line_number":128,"context_line":""},{"line_number":129,"context_line":"Graceful service shutdown"},{"line_number":130,"context_line":"    Many nova services are python processes listening for messages on a"},{"line_number":131,"context_line":"    AMQP queue, including nova-compute. When sending the process the SIGTERM"},{"line_number":132,"context_line":"    the process stops getting new work from its queue, completes any"},{"line_number":133,"context_line":"    outstanding work, then terminates. During this process, messages can be"},{"line_number":134,"context_line":"    left on the queue for when the python process starts back up."}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a629dbe_87d5bb44","line":131,"range":{"start_line":131,"start_character":40,"end_line":131,"end_character":44},"in_reply_to":"ba5da102_fbe70e44","updated":"2016-11-08 11:12:03.000000000","message":"Done","commit_id":"e6138c23e4e32c9398b861b77a35abbd4cde3e45"},{"author":{"_account_id":20140,"name":"Karthik Prabhu Vinod","email":"karthik.prabhu.vinod@intel.com","username":"kprabhuv"},"change_message_id":"809f1576ad4a682bf3e1488cfcd3378b4deb3d74","unresolved":false,"context_lines":[{"line_number":151,"context_line":"    writing to the database. Taking this a step further, we can make all DB"},{"line_number":152,"context_line":"    changes by first adding the new structures, expanding. Then you can slowly"},{"line_number":153,"context_line":"    move all the data into a new location and format. Once that is complete,"},{"line_number":154,"context_line":"    you can drop bits of the scheme that are no long needed, i.e. contract."},{"line_number":155,"context_line":"    We have plans to implement this here:"},{"line_number":156,"context_line":"    https://review.openstack.org/#/c/102545/5/specs/juno/online-schema-changes.rst,cm"},{"line_number":157,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"ba5da102_78b88b2a","line":154,"range":{"start_line":154,"start_character":52,"end_line":154,"end_character":53},"updated":"2016-11-07 16:10:20.000000000","message":"longer needed","commit_id":"e6138c23e4e32c9398b861b77a35abbd4cde3e45"},{"author":{"_account_id":20140,"name":"Karthik Prabhu Vinod","email":"karthik.prabhu.vinod@intel.com","username":"kprabhuv"},"change_message_id":"809f1576ad4a682bf3e1488cfcd3378b4deb3d74","unresolved":false,"context_lines":[{"line_number":151,"context_line":"    writing to the database. Taking this a step further, we can make all DB"},{"line_number":152,"context_line":"    changes by first adding the new structures, expanding. Then you can slowly"},{"line_number":153,"context_line":"    move all the data into a new location and format. Once that is complete,"},{"line_number":154,"context_line":"    you can drop bits of the scheme that are no long needed, i.e. contract."},{"line_number":155,"context_line":"    We have plans to implement this here:"},{"line_number":156,"context_line":"    https://review.openstack.org/#/c/102545/5/specs/juno/online-schema-changes.rst,cm"},{"line_number":157,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"ba5da102_f8e65bae","line":154,"range":{"start_line":154,"start_character":34,"end_line":154,"end_character":35},"updated":"2016-11-07 16:10:20.000000000","message":"s/scheme/schema","commit_id":"e6138c23e4e32c9398b861b77a35abbd4cde3e45"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"3fdbebbbe861c6fc9b30c00dc8e214edf83478d5","unresolved":false,"context_lines":[{"line_number":151,"context_line":"    writing to the database. Taking this a step further, we can make all DB"},{"line_number":152,"context_line":"    changes by first adding the new structures, expanding. Then you can slowly"},{"line_number":153,"context_line":"    move all the data into a new location and format. Once that is complete,"},{"line_number":154,"context_line":"    you can drop bits of the scheme that are no long needed, i.e. contract."},{"line_number":155,"context_line":"    We have plans to implement this here:"},{"line_number":156,"context_line":"    https://review.openstack.org/#/c/102545/5/specs/juno/online-schema-changes.rst,cm"},{"line_number":157,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a629dbe_27bf8ffc","line":154,"range":{"start_line":154,"start_character":34,"end_line":154,"end_character":35},"in_reply_to":"ba5da102_f8e65bae","updated":"2016-11-08 11:12:03.000000000","message":"This one is removed in next patchsets.","commit_id":"e6138c23e4e32c9398b861b77a35abbd4cde3e45"},{"author":{"_account_id":20140,"name":"Karthik Prabhu Vinod","email":"karthik.prabhu.vinod@intel.com","username":"kprabhuv"},"change_message_id":"809f1576ad4a682bf3e1488cfcd3378b4deb3d74","unresolved":false,"context_lines":[{"line_number":173,"context_line":"    those entries affects DB performance as indices grow very large and data"},{"line_number":174,"context_line":"    migrations take longer as there is more data to migrate."},{"line_number":175,"context_line":""},{"line_number":176,"context_line":"nova-conductor object backports"},{"line_number":177,"context_line":"    RPC pinning ensures new services can talk to the older service\u0027s method"},{"line_number":178,"context_line":"    signatures. But many of the parameters are objects that may well be too"},{"line_number":179,"context_line":"    new for the old service to understand, so you are able to send the object"}],"source_content_type":"text/x-rst","patch_set":1,"id":"ba5da102_14cf37b4","line":176,"updated":"2016-11-07 16:10:20.000000000","message":"I believe this becomes just \"Object version backporting\"","commit_id":"e6138c23e4e32c9398b861b77a35abbd4cde3e45"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"3fdbebbbe861c6fc9b30c00dc8e214edf83478d5","unresolved":false,"context_lines":[{"line_number":173,"context_line":"    those entries affects DB performance as indices grow very large and data"},{"line_number":174,"context_line":"    migrations take longer as there is more data to migrate."},{"line_number":175,"context_line":""},{"line_number":176,"context_line":"nova-conductor object backports"},{"line_number":177,"context_line":"    RPC pinning ensures new services can talk to the older service\u0027s method"},{"line_number":178,"context_line":"    signatures. But many of the parameters are objects that may well be too"},{"line_number":179,"context_line":"    new for the old service to understand, so you are able to send the object"}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a629dbe_873cdb94","line":176,"in_reply_to":"ba5da102_14cf37b4","updated":"2016-11-08 11:12:03.000000000","message":"Fixed in next patchsets.","commit_id":"e6138c23e4e32c9398b861b77a35abbd4cde3e45"},{"author":{"_account_id":20140,"name":"Karthik Prabhu Vinod","email":"karthik.prabhu.vinod@intel.com","username":"kprabhuv"},"change_message_id":"809f1576ad4a682bf3e1488cfcd3378b4deb3d74","unresolved":false,"context_lines":[{"line_number":174,"context_line":"    migrations take longer as there is more data to migrate."},{"line_number":175,"context_line":""},{"line_number":176,"context_line":"nova-conductor object backports"},{"line_number":177,"context_line":"    RPC pinning ensures new services can talk to the older service\u0027s method"},{"line_number":178,"context_line":"    signatures. But many of the parameters are objects that may well be too"},{"line_number":179,"context_line":"    new for the old service to understand, so you are able to send the object"},{"line_number":180,"context_line":"    back to the nova-conductor to be downgraded to a version the older service"}],"source_content_type":"text/x-rst","patch_set":1,"id":"ba5da102_7b17a244","line":177,"range":{"start_line":177,"start_character":4,"end_line":177,"end_character":7},"updated":"2016-11-07 16:10:20.000000000","message":"I believe they have this part of the Nova-conductor which is the only service which talks to the database.\n\nSince, in cinder all services talk could talk to the DB, object back-porting will have to be a part of all services.\n\nAren\u0027t all objects backported because of the serialize_entity in rpc.py","commit_id":"e6138c23e4e32c9398b861b77a35abbd4cde3e45"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"3fdbebbbe861c6fc9b30c00dc8e214edf83478d5","unresolved":false,"context_lines":[{"line_number":174,"context_line":"    migrations take longer as there is more data to migrate."},{"line_number":175,"context_line":""},{"line_number":176,"context_line":"nova-conductor object backports"},{"line_number":177,"context_line":"    RPC pinning ensures new services can talk to the older service\u0027s method"},{"line_number":178,"context_line":"    signatures. But many of the parameters are objects that may well be too"},{"line_number":179,"context_line":"    new for the old service to understand, so you are able to send the object"},{"line_number":180,"context_line":"    back to the nova-conductor to be downgraded to a version the older service"}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a629dbe_a758fffb","line":177,"range":{"start_line":177,"start_character":4,"end_line":177,"end_character":7},"in_reply_to":"ba5da102_7b17a244","updated":"2016-11-08 11:12:03.000000000","message":"That was copied from Nova\u0027s reference and is fixed in next patchsets.","commit_id":"e6138c23e4e32c9398b861b77a35abbd4cde3e45"},{"author":{"_account_id":11904,"name":"Sean McGinnis","email":"sean.mcginnis@gmail.com","username":"SeanM"},"change_message_id":"f17f492852068b28b334e4ce1e471407a089ed7c","unresolved":false,"context_lines":[{"line_number":46,"context_line":"* To avoid dependency hell it is advised to have your Cinder services deployed"},{"line_number":47,"context_line":"  separately in containers or Python venvs."},{"line_number":48,"context_line":""},{"line_number":49,"context_line":"* Note that Cinder is basing version detection on ``services`` table in the DB."},{"line_number":50,"context_line":"  Before upgrade make sure you don\u0027t have any orphaned old records there,"},{"line_number":51,"context_line":"  because these can block starting newer services."},{"line_number":52,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"9a629dbe_dc04d4c1","line":49,"range":{"start_line":49,"start_character":47,"end_line":49,"end_character":50},"updated":"2016-11-08 16:00:08.000000000","message":"I think it would sound be to say \"on the ``services``\"","commit_id":"a33d413cb66bcbe723fb2fc752bd3198603c8921"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"a791e92eadb4b245b4cb2324bf7f457d54101fc1","unresolved":false,"context_lines":[{"line_number":46,"context_line":"* To avoid dependency hell it is advised to have your Cinder services deployed"},{"line_number":47,"context_line":"  separately in containers or Python venvs."},{"line_number":48,"context_line":""},{"line_number":49,"context_line":"* Note that Cinder is basing version detection on ``services`` table in the DB."},{"line_number":50,"context_line":"  Before upgrade make sure you don\u0027t have any orphaned old records there,"},{"line_number":51,"context_line":"  because these can block starting newer services."},{"line_number":52,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"9a629dbe_b791fb54","line":49,"range":{"start_line":49,"start_character":47,"end_line":49,"end_character":50},"in_reply_to":"9a629dbe_dc04d4c1","updated":"2016-11-08 16:15:28.000000000","message":"I\u0027ve reworded this a little.","commit_id":"a33d413cb66bcbe723fb2fc752bd3198603c8921"},{"author":{"_account_id":11904,"name":"Sean McGinnis","email":"sean.mcginnis@gmail.com","username":"SeanM"},"change_message_id":"f17f492852068b28b334e4ce1e471407a089ed7c","unresolved":false,"context_lines":[{"line_number":59,"context_line":"To reduce downtime, the services can be upgraded in a rolling fashion. It"},{"line_number":60,"context_line":"means upgrading a few services at a time. To minimise downtime you need to have"},{"line_number":61,"context_line":"HA Cinder deployment, so at the moment a service is upgraded, you\u0027ll keep"},{"line_number":62,"context_line":"another service instances running."},{"line_number":63,"context_line":""},{"line_number":64,"context_line":"Before maintenance window"},{"line_number":65,"context_line":"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\""}],"source_content_type":"text/x-rst","patch_set":4,"id":"9a629dbe_bc6e4094","line":62,"range":{"start_line":62,"start_character":0,"end_line":62,"end_character":7},"updated":"2016-11-08 16:00:08.000000000","message":"s/another/other/","commit_id":"a33d413cb66bcbe723fb2fc752bd3198603c8921"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"a791e92eadb4b245b4cb2324bf7f457d54101fc1","unresolved":false,"context_lines":[{"line_number":59,"context_line":"To reduce downtime, the services can be upgraded in a rolling fashion. It"},{"line_number":60,"context_line":"means upgrading a few services at a time. To minimise downtime you need to have"},{"line_number":61,"context_line":"HA Cinder deployment, so at the moment a service is upgraded, you\u0027ll keep"},{"line_number":62,"context_line":"another service instances running."},{"line_number":63,"context_line":""},{"line_number":64,"context_line":"Before maintenance window"},{"line_number":65,"context_line":"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\""}],"source_content_type":"text/x-rst","patch_set":4,"id":"9a629dbe_f79b7370","line":62,"range":{"start_line":62,"start_character":0,"end_line":62,"end_character":7},"in_reply_to":"9a629dbe_bc6e4094","updated":"2016-11-08 16:15:28.000000000","message":"Done","commit_id":"a33d413cb66bcbe723fb2fc752bd3198603c8921"},{"author":{"_account_id":20140,"name":"Karthik Prabhu Vinod","email":"karthik.prabhu.vinod@intel.com","username":"kprabhuv"},"change_message_id":"dcb14840a1b75648a774e43109f9ff9779d108fb","unresolved":false,"context_lines":[{"line_number":93,"context_line":""},{"line_number":94,"context_line":"3. Next service is cinder-scheduler. It is load-balanced by the message queue,"},{"line_number":95,"context_line":"   so the only thing you need to worry about is to shut it down gracefully"},{"line_number":96,"context_line":"   (using ``SIGTERM`` signal) to make sure it will finish all the requests"},{"line_number":97,"context_line":"   beign processed before shutting down. Then you should upgrade the code and"},{"line_number":98,"context_line":"   restart the service."},{"line_number":99,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"9a629dbe_e9662a28","line":96,"range":{"start_line":96,"start_character":19,"end_line":96,"end_character":21},"updated":"2016-11-09 17:36:36.000000000","message":"SIGHUP? Are we handling it gracefully using any kill handle?","commit_id":"c053af700d04b9b6e1adbd8a7018707da0d81c14"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"75982bf9881a685aff5a7311a65bb885f14ac11b","unresolved":false,"context_lines":[{"line_number":93,"context_line":""},{"line_number":94,"context_line":"3. Next service is cinder-scheduler. It is load-balanced by the message queue,"},{"line_number":95,"context_line":"   so the only thing you need to worry about is to shut it down gracefully"},{"line_number":96,"context_line":"   (using ``SIGTERM`` signal) to make sure it will finish all the requests"},{"line_number":97,"context_line":"   beign processed before shutting down. Then you should upgrade the code and"},{"line_number":98,"context_line":"   restart the service."},{"line_number":99,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"9a629dbe_d8b417d6","line":96,"range":{"start_line":96,"start_character":19,"end_line":96,"end_character":21},"in_reply_to":"9a629dbe_e9662a28","updated":"2016-11-14 11:26:15.000000000","message":"SIGTERM will stop the service gracefully. SIGKILL will kill it without hesitation in a non-graceful manner. SIGHUP will gracefully stop service\u0027s threads and start them again. This isn\u0027t however a way of restarting a service with new code.\n\nBTW - here\u0027s [1] Gorka explaining how does it actually work and what are the issues with stopping Cinder.\n\n[1] https://youtu.be/VjQ6D4IZMBk?t\u003d1719","commit_id":"c053af700d04b9b6e1adbd8a7018707da0d81c14"},{"author":{"_account_id":17491,"name":"Lujin Luo","email":"luo.lujin@jp.fujitsu.com","username":"Lujin"},"change_message_id":"8999cdc5c95fa58eadaa8af517f412d3f24955a7","unresolved":false,"context_lines":[{"line_number":94,"context_line":"3. Next service is cinder-scheduler. It is load-balanced by the message queue,"},{"line_number":95,"context_line":"   so the only thing you need to worry about is to shut it down gracefully"},{"line_number":96,"context_line":"   (using ``SIGTERM`` signal) to make sure it will finish all the requests"},{"line_number":97,"context_line":"   beign processed before shutting down. Then you should upgrade the code and"},{"line_number":98,"context_line":"   restart the service."},{"line_number":99,"context_line":""},{"line_number":100,"context_line":"4. Repeat third step for all of your cinder-scheduler services."}],"source_content_type":"text/x-rst","patch_set":5,"id":"9a629dbe_87358df8","line":97,"range":{"start_line":97,"start_character":3,"end_line":97,"end_character":8},"updated":"2016-11-09 07:46:27.000000000","message":"nit: s/beign/being","commit_id":"c053af700d04b9b6e1adbd8a7018707da0d81c14"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"6f0bbd393fac5fdfad9efb334943d8b19ca348f8","unresolved":false,"context_lines":[{"line_number":94,"context_line":"3. Next service is cinder-scheduler. It is load-balanced by the message queue,"},{"line_number":95,"context_line":"   so the only thing you need to worry about is to shut it down gracefully"},{"line_number":96,"context_line":"   (using ``SIGTERM`` signal) to make sure it will finish all the requests"},{"line_number":97,"context_line":"   beign processed before shutting down. Then you should upgrade the code and"},{"line_number":98,"context_line":"   restart the service."},{"line_number":99,"context_line":""},{"line_number":100,"context_line":"4. Repeat third step for all of your cinder-scheduler services."}],"source_content_type":"text/x-rst","patch_set":5,"id":"9a629dbe_fa85fcac","line":97,"range":{"start_line":97,"start_character":3,"end_line":97,"end_character":8},"in_reply_to":"9a629dbe_87358df8","updated":"2016-11-09 07:58:40.000000000","message":"Done","commit_id":"c053af700d04b9b6e1adbd8a7018707da0d81c14"},{"author":{"_account_id":20140,"name":"Karthik Prabhu Vinod","email":"karthik.prabhu.vinod@intel.com","username":"kprabhuv"},"change_message_id":"dcb14840a1b75648a774e43109f9ff9779d108fb","unresolved":false,"context_lines":[{"line_number":99,"context_line":""},{"line_number":100,"context_line":"4. Repeat third step for all of your cinder-scheduler services."},{"line_number":101,"context_line":""},{"line_number":102,"context_line":"5. Then you proceed to upgrade cinder-volume services. The problem here is that"},{"line_number":103,"context_line":"   due to Active/Passive character of this service, you\u0027re unable to run"},{"line_number":104,"context_line":"   multiple instances of cinder-volume managing a single volume backend. This"},{"line_number":105,"context_line":"   means that there will be a moment when you won\u0027t have any cinder-volume in"},{"line_number":106,"context_line":"   your deployment and you want that disruption to be as short as possible."},{"line_number":107,"context_line":""},{"line_number":108,"context_line":"   .. note::"},{"line_number":109,"context_line":""},{"line_number":110,"context_line":"     The downtime here is non-disruptive as long as it doesn\u0027t exceed the"},{"line_number":111,"context_line":"     service heartbeat timeout. If you don\u0027t exceed that, then"},{"line_number":112,"context_line":"     cinder-schedulers will not notice that cinder-volume is gone and the"},{"line_number":113,"context_line":"     message queue will take care of queuing any RPC messages until"},{"line_number":114,"context_line":"     cinder-volume is back."},{"line_number":115,"context_line":""},{"line_number":116,"context_line":"     To make sure it\u0027s achieved, you can either lengthen the timeout by"},{"line_number":117,"context_line":"     tweaking ``service_down_time`` value in ``cinder.conf``, or prepare"},{"line_number":118,"context_line":"     upgraded cinder-volume on another node and do a very quick switch by"},{"line_number":119,"context_line":"     shutting down older service and starting the new one just after that."},{"line_number":120,"context_line":""},{"line_number":121,"context_line":"6. Repeat fifth step for all cinder-volume services."},{"line_number":122,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"9a629dbe_24d5a9b8","line":119,"range":{"start_line":102,"start_character":0,"end_line":119,"end_character":74},"updated":"2016-11-09 17:36:36.000000000","message":"+1","commit_id":"c053af700d04b9b6e1adbd8a7018707da0d81c14"},{"author":{"_account_id":20140,"name":"Karthik Prabhu Vinod","email":"karthik.prabhu.vinod@intel.com","username":"kprabhuv"},"change_message_id":"dcb14840a1b75648a774e43109f9ff9779d108fb","unresolved":false,"context_lines":[{"line_number":148,"context_line":""},{"line_number":149,"context_line":"* Now all services are upgraded, we need to send the ``SIGHUP`` signal, so"},{"line_number":150,"context_line":"  all the services clear any cached service version data. When a new service"},{"line_number":151,"context_line":"  starts, it automatically detects which version of the compute RPC protocol"},{"line_number":152,"context_line":"  to use, and will downgrade any communication to that version. Be advised"},{"line_number":153,"context_line":"  that cinder-api service doesn\u0027t handle ``SIGHUP`` so it needs to be"},{"line_number":154,"context_line":"  restarted."}],"source_content_type":"text/x-rst","patch_set":5,"id":"9a629dbe_1b10ff4d","line":151,"range":{"start_line":151,"start_character":56,"end_line":151,"end_character":63},"updated":"2016-11-09 17:36:36.000000000","message":"service\u0027s","commit_id":"c053af700d04b9b6e1adbd8a7018707da0d81c14"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"75982bf9881a685aff5a7311a65bb885f14ac11b","unresolved":false,"context_lines":[{"line_number":148,"context_line":""},{"line_number":149,"context_line":"* Now all services are upgraded, we need to send the ``SIGHUP`` signal, so"},{"line_number":150,"context_line":"  all the services clear any cached service version data. When a new service"},{"line_number":151,"context_line":"  starts, it automatically detects which version of the compute RPC protocol"},{"line_number":152,"context_line":"  to use, and will downgrade any communication to that version. Be advised"},{"line_number":153,"context_line":"  that cinder-api service doesn\u0027t handle ``SIGHUP`` so it needs to be"},{"line_number":154,"context_line":"  restarted."}],"source_content_type":"text/x-rst","patch_set":5,"id":"9a629dbe_78222b03","line":151,"range":{"start_line":151,"start_character":56,"end_line":151,"end_character":63},"in_reply_to":"9a629dbe_1b10ff4d","updated":"2016-11-14 11:26:15.000000000","message":"Done","commit_id":"c053af700d04b9b6e1adbd8a7018707da0d81c14"},{"author":{"_account_id":20140,"name":"Karthik Prabhu Vinod","email":"karthik.prabhu.vinod@intel.com","username":"kprabhuv"},"change_message_id":"dcb14840a1b75648a774e43109f9ff9779d108fb","unresolved":false,"context_lines":[{"line_number":154,"context_line":"  restarted."},{"line_number":155,"context_line":""},{"line_number":156,"context_line":"* Now all the services are upgraded, the system is able to use the latest"},{"line_number":157,"context_line":"  version of the RPC protocol and so get access to all the new features in"},{"line_number":158,"context_line":"  the new release."},{"line_number":159,"context_line":""},{"line_number":160,"context_line":"* At this point, you must also ensure you update the configuration, to stop"},{"line_number":161,"context_line":"  using any deprecated features or options, and perform any required work"}],"source_content_type":"text/x-rst","patch_set":5,"id":"9a629dbe_1b3e3fb8","line":158,"range":{"start_line":157,"start_character":30,"end_line":158,"end_character":18},"updated":"2016-11-09 17:36:36.000000000","message":"and able to access all the features of the new release.","commit_id":"c053af700d04b9b6e1adbd8a7018707da0d81c14"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"75982bf9881a685aff5a7311a65bb885f14ac11b","unresolved":false,"context_lines":[{"line_number":154,"context_line":"  restarted."},{"line_number":155,"context_line":""},{"line_number":156,"context_line":"* Now all the services are upgraded, the system is able to use the latest"},{"line_number":157,"context_line":"  version of the RPC protocol and so get access to all the new features in"},{"line_number":158,"context_line":"  the new release."},{"line_number":159,"context_line":""},{"line_number":160,"context_line":"* At this point, you must also ensure you update the configuration, to stop"},{"line_number":161,"context_line":"  using any deprecated features or options, and perform any required work"}],"source_content_type":"text/x-rst","patch_set":5,"id":"9a629dbe_b81cc3c2","line":158,"range":{"start_line":157,"start_character":30,"end_line":158,"end_character":18},"in_reply_to":"9a629dbe_1b3e3fb8","updated":"2016-11-14 11:26:15.000000000","message":"Done","commit_id":"c053af700d04b9b6e1adbd8a7018707da0d81c14"},{"author":{"_account_id":10058,"name":"Erlon R. Cruz","email":"erlon.rodrigues.cruz@canonical.com","username":"sombrafam"},"change_message_id":"898650da5cb0dc5a99f865ddf8a34aa6973c00c6","unresolved":false,"context_lines":[{"line_number":64,"context_line":"Before maintenance window"},{"line_number":65,"context_line":"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\""},{"line_number":66,"context_line":""},{"line_number":67,"context_line":"* Install new Cinder code in new venv or a container and run the DB"},{"line_number":68,"context_line":"  sync (``cinder-manage db sync``). These schema change operations should"},{"line_number":69,"context_line":"  have minimal or no effect on performance, and should not cause any"},{"line_number":70,"context_line":"  operations to fail."},{"line_number":71,"context_line":""},{"line_number":72,"context_line":"* At this point, new columns and tables may exist in the database. These"},{"line_number":73,"context_line":"  DB schema changes are done in a way that both the N and N+1 release can"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9a629dbe_ecd34ed3","line":70,"range":{"start_line":67,"start_character":2,"end_line":70,"end_character":21},"updated":"2016-11-10 11:28:58.000000000","message":"Is this new install only for the db upgrade or it is the definitive service being upgraded?","commit_id":"74c883d079768821902397f40b3bc007d9141ea6"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"725dfe139fe5bf1b5de55092bbf11c85f8c88333","unresolved":false,"context_lines":[{"line_number":64,"context_line":"Before maintenance window"},{"line_number":65,"context_line":"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\"\""},{"line_number":66,"context_line":""},{"line_number":67,"context_line":"* Install new Cinder code in new venv or a container and run the DB"},{"line_number":68,"context_line":"  sync (``cinder-manage db sync``). These schema change operations should"},{"line_number":69,"context_line":"  have minimal or no effect on performance, and should not cause any"},{"line_number":70,"context_line":"  operations to fail."},{"line_number":71,"context_line":""},{"line_number":72,"context_line":"* At this point, new columns and tables may exist in the database. These"},{"line_number":73,"context_line":"  DB schema changes are done in a way that both the N and N+1 release can"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9a629dbe_23b5e87d","line":70,"range":{"start_line":67,"start_character":2,"end_line":70,"end_character":21},"in_reply_to":"9a629dbe_ecd34ed3","updated":"2016-11-14 11:26:08.000000000","message":"DB schema migrations only. I\u0027ve extended the description a little.","commit_id":"74c883d079768821902397f40b3bc007d9141ea6"},{"author":{"_account_id":10058,"name":"Erlon R. Cruz","email":"erlon.rodrigues.cruz@canonical.com","username":"sombrafam"},"change_message_id":"898650da5cb0dc5a99f865ddf8a34aa6973c00c6","unresolved":false,"context_lines":[{"line_number":129,"context_line":"     without interrupting ongoing requests can take time. It may be useful to"},{"line_number":130,"context_line":"     disable the service first using ``cinder service-disable`` command, so it"},{"line_number":131,"context_line":"     won\u0027t accept new requests, and wait a reasonable amount of time until all"},{"line_number":132,"context_line":"     the in-progress jobs are completed. Then you can proceed with the upgrade."},{"line_number":133,"context_line":""},{"line_number":134,"context_line":"   .. note::"},{"line_number":135,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"9a629dbe_6c2ade91","line":132,"range":{"start_line":132,"start_character":8,"end_line":132,"end_character":20},"updated":"2016-11-10 11:28:58.000000000","message":"Would be good to point a way to check this.","commit_id":"74c883d079768821902397f40b3bc007d9141ea6"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"725dfe139fe5bf1b5de55092bbf11c85f8c88333","unresolved":false,"context_lines":[{"line_number":129,"context_line":"     without interrupting ongoing requests can take time. It may be useful to"},{"line_number":130,"context_line":"     disable the service first using ``cinder service-disable`` command, so it"},{"line_number":131,"context_line":"     won\u0027t accept new requests, and wait a reasonable amount of time until all"},{"line_number":132,"context_line":"     the in-progress jobs are completed. Then you can proceed with the upgrade."},{"line_number":133,"context_line":""},{"line_number":134,"context_line":"   .. note::"},{"line_number":135,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"9a629dbe_8308742f","line":132,"range":{"start_line":132,"start_character":8,"end_line":132,"end_character":20},"in_reply_to":"9a629dbe_6c2ade91","updated":"2016-11-14 11:26:08.000000000","message":"I think that currently only logs are able to provide such info. I\u0027ve added this to the paragraph.","commit_id":"74c883d079768821902397f40b3bc007d9141ea6"},{"author":{"_account_id":10058,"name":"Erlon R. Cruz","email":"erlon.rodrigues.cruz@canonical.com","username":"sombrafam"},"change_message_id":"898650da5cb0dc5a99f865ddf8a34aa6973c00c6","unresolved":false,"context_lines":[{"line_number":167,"context_line":"Concepts"},{"line_number":168,"context_line":"--------"},{"line_number":169,"context_line":""},{"line_number":170,"context_line":"Here are the key concepts you need to know before reading the section on the"},{"line_number":171,"context_line":"upgrade process:"},{"line_number":172,"context_line":""},{"line_number":173,"context_line":"RPC version pinning"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9a629dbe_0f9874bf","line":170,"range":{"start_line":170,"start_character":26,"end_line":170,"end_character":57},"updated":"2016-11-10 11:28:58.000000000","message":"Now I have read already. :) This should go in the beginning of the document.","commit_id":"74c883d079768821902397f40b3bc007d9141ea6"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"725dfe139fe5bf1b5de55092bbf11c85f8c88333","unresolved":false,"context_lines":[{"line_number":167,"context_line":"Concepts"},{"line_number":168,"context_line":"--------"},{"line_number":169,"context_line":""},{"line_number":170,"context_line":"Here are the key concepts you need to know before reading the section on the"},{"line_number":171,"context_line":"upgrade process:"},{"line_number":172,"context_line":""},{"line_number":173,"context_line":"RPC version pinning"}],"source_content_type":"text/x-rst","patch_set":6,"id":"9a629dbe_5e12af59","line":170,"range":{"start_line":170,"start_character":26,"end_line":170,"end_character":57},"in_reply_to":"9a629dbe_0f9874bf","updated":"2016-11-14 11:26:08.000000000","message":"Done","commit_id":"74c883d079768821902397f40b3bc007d9141ea6"},{"author":{"_account_id":17491,"name":"Lujin Luo","email":"luo.lujin@jp.fujitsu.com","username":"Lujin"},"change_message_id":"2837b1e20426664ed951d2d8a70e4b1d2e04f770","unresolved":false,"context_lines":[{"line_number":23,"context_line":"interfere with data plane, its upgrade shouldn\u0027t affect any volumes being"},{"line_number":24,"context_line":"accessed by virtual machines."},{"line_number":25,"context_line":""},{"line_number":26,"context_line":"Keeping the control plane running during an upgrade is more difficult. This"},{"line_number":27,"context_line":"document goal is to provide preliminaries and a detailed procedure of such"},{"line_number":28,"context_line":"upgrade."},{"line_number":29,"context_line":""},{"line_number":30,"context_line":"Concepts"}],"source_content_type":"text/x-rst","patch_set":7,"id":"7a77a97e_846fa491","line":27,"range":{"start_line":26,"start_character":71,"end_line":27,"end_character":13},"updated":"2016-11-15 02:11:11.000000000","message":"s/document/document\u0027s","commit_id":"f07432c072236f843fd6d444ad68a7854e7bd09c"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"ecc5cefb45dd3bcf7c3cafde2f4cfd6f75bc2dc5","unresolved":false,"context_lines":[{"line_number":23,"context_line":"interfere with data plane, its upgrade shouldn\u0027t affect any volumes being"},{"line_number":24,"context_line":"accessed by virtual machines."},{"line_number":25,"context_line":""},{"line_number":26,"context_line":"Keeping the control plane running during an upgrade is more difficult. This"},{"line_number":27,"context_line":"document goal is to provide preliminaries and a detailed procedure of such"},{"line_number":28,"context_line":"upgrade."},{"line_number":29,"context_line":""},{"line_number":30,"context_line":"Concepts"}],"source_content_type":"text/x-rst","patch_set":7,"id":"7a77a97e_cf02813e","line":27,"range":{"start_line":26,"start_character":71,"end_line":27,"end_character":13},"in_reply_to":"7a77a97e_846fa491","updated":"2016-11-15 13:16:28.000000000","message":"Done","commit_id":"f07432c072236f843fd6d444ad68a7854e7bd09c"},{"author":{"_account_id":17491,"name":"Lujin Luo","email":"luo.lujin@jp.fujitsu.com","username":"Lujin"},"change_message_id":"2837b1e20426664ed951d2d8a70e4b1d2e04f770","unresolved":false,"context_lines":[{"line_number":206,"context_line":"* Now all services are upgraded, we need to send the ``SIGHUP`` signal, so"},{"line_number":207,"context_line":"  all the services clear any cached service version data. When a new service"},{"line_number":208,"context_line":"  starts, it automatically detects which version of the service\u0027s RPC protocol"},{"line_number":209,"context_line":"  to use, and will downgrade any communication to that version. Be advised"},{"line_number":210,"context_line":"  that cinder-api service doesn\u0027t handle ``SIGHUP`` so it needs to be"},{"line_number":211,"context_line":"  restarted."},{"line_number":212,"context_line":""},{"line_number":213,"context_line":"* Now all the services are upgraded, the system is able to use the latest"},{"line_number":214,"context_line":"  version of the RPC protocol and able to access all the features of the new"}],"source_content_type":"text/x-rst","patch_set":7,"id":"7a77a97e_e43698b3","line":211,"range":{"start_line":209,"start_character":64,"end_line":211,"end_character":12},"updated":"2016-11-15 02:11:11.000000000","message":"One question here: which one should go first? restaring cinder-api or sending the ``SIGHUP`` signal? Does it mean either way is fine?","commit_id":"f07432c072236f843fd6d444ad68a7854e7bd09c"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"ecc5cefb45dd3bcf7c3cafde2f4cfd6f75bc2dc5","unresolved":false,"context_lines":[{"line_number":206,"context_line":"* Now all services are upgraded, we need to send the ``SIGHUP`` signal, so"},{"line_number":207,"context_line":"  all the services clear any cached service version data. When a new service"},{"line_number":208,"context_line":"  starts, it automatically detects which version of the service\u0027s RPC protocol"},{"line_number":209,"context_line":"  to use, and will downgrade any communication to that version. Be advised"},{"line_number":210,"context_line":"  that cinder-api service doesn\u0027t handle ``SIGHUP`` so it needs to be"},{"line_number":211,"context_line":"  restarted."},{"line_number":212,"context_line":""},{"line_number":213,"context_line":"* Now all the services are upgraded, the system is able to use the latest"},{"line_number":214,"context_line":"  version of the RPC protocol and able to access all the features of the new"}],"source_content_type":"text/x-rst","patch_set":7,"id":"7a77a97e_2f7f75ab","line":211,"range":{"start_line":209,"start_character":64,"end_line":211,"end_character":12},"in_reply_to":"7a77a97e_e43698b3","updated":"2016-11-15 13:16:28.000000000","message":"Oh, excellent question. It should be best to end with c-api services, to make sure no request will get downgraded anywhere lower in the chain. I\u0027ll update the patch to explain that.","commit_id":"f07432c072236f843fd6d444ad68a7854e7bd09c"},{"author":{"_account_id":20140,"name":"Karthik Prabhu Vinod","email":"karthik.prabhu.vinod@intel.com","username":"kprabhuv"},"change_message_id":"a22b6085b864e4da2e0e64425292114ea5798538","unresolved":false,"context_lines":[{"line_number":112,"context_line":"* To avoid dependency hell it is advised to have your Cinder services deployed"},{"line_number":113,"context_line":"  separately in containers or Python venvs."},{"line_number":114,"context_line":""},{"line_number":115,"context_line":"* Note that Cinder is basing version detection on what\u0027s reported in the"},{"line_number":116,"context_line":"  ``services`` table in the DB. Before upgrade make sure you don\u0027t have any"},{"line_number":117,"context_line":"  orphaned old records there, because these can block starting newer services."},{"line_number":118,"context_line":""}],"source_content_type":"text/x-rst","patch_set":10,"id":"7a77a97e_f09f2763","line":115,"range":{"start_line":115,"start_character":50,"end_line":115,"end_character":54},"updated":"2016-11-22 00:10:26.000000000","message":"what is","commit_id":"60183ed88e84bc5ac9905967361958157f455e97"},{"author":{"_account_id":20140,"name":"Karthik Prabhu Vinod","email":"karthik.prabhu.vinod@intel.com","username":"kprabhuv"},"change_message_id":"a22b6085b864e4da2e0e64425292114ea5798538","unresolved":false,"context_lines":[{"line_number":162,"context_line":"   so the only thing you need to worry about is to shut it down gracefully"},{"line_number":163,"context_line":"   (using ``SIGTERM`` signal) to make sure it will finish all the requests"},{"line_number":164,"context_line":"   being processed before shutting down. Then you should upgrade the code and"},{"line_number":165,"context_line":"   restart the service."},{"line_number":166,"context_line":""},{"line_number":167,"context_line":"4. Repeat third step for all of your cinder-scheduler services."},{"line_number":168,"context_line":""}],"source_content_type":"text/x-rst","patch_set":10,"id":"7a77a97e_3a5fbd76","line":165,"range":{"start_line":165,"start_character":22,"end_line":165,"end_character":23},"updated":"2016-11-22 00:10:26.000000000","message":"I just tried a particular use case 1 and the volume got stuck in a creating state:\n\nNode 1[stable-mitaka]\n--------------------------\nc-api, c-vol, c-bak, c-sch\n\nNode 2 [stable-newton]\n--------------------------\nc-api, c-sch\n\nSo, 50% of the requests get routed through to new scheduler and the remaining through the old one in this case. So it is load-balanced.\n+1.\n\nBut when it gets routed through the old scheduler. we get an unsupported error.\n\n2016-11-21 23:17:40.346 TRACE oslo_messaging.rpc.server UnsupportedVersion: Endpoint does not support RPC version 3.0. Attempted method: create_volume\n\nIs this expected or the messages should be downgraded to 2.0?","commit_id":"60183ed88e84bc5ac9905967361958157f455e97"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"cec69d2054962397bba4c60f1e412ade0e330b38","unresolved":false,"context_lines":[{"line_number":162,"context_line":"   so the only thing you need to worry about is to shut it down gracefully"},{"line_number":163,"context_line":"   (using ``SIGTERM`` signal) to make sure it will finish all the requests"},{"line_number":164,"context_line":"   being processed before shutting down. Then you should upgrade the code and"},{"line_number":165,"context_line":"   restart the service."},{"line_number":166,"context_line":""},{"line_number":167,"context_line":"4. Repeat third step for all of your cinder-scheduler services."},{"line_number":168,"context_line":""}],"source_content_type":"text/x-rst","patch_set":10,"id":"7a77a97e_7e62ecae","line":165,"range":{"start_line":165,"start_character":22,"end_line":165,"end_character":23},"in_reply_to":"7a77a97e_3a5fbd76","updated":"2016-11-22 09:37:33.000000000","message":"I suspect you\u0027ve started services in wrong order. There\u0027s assumption that older services are in `services` table before newer services start (pretty natural assumption, as this is upgrade :)). Try shutting Cinder services down, execute `DELETE FROM services;`, start services on Node 1 (with older stable/mitaka), then start services on Node 2 (with newer stable/newton) and rerun the tests scenario.\n\nBTW - which API service is getting the request? How you\u0027re load-balancing HTTP API? HAProxy? There shouldn\u0027t really be scenario when you\u0027re having both c-api\u0027s and c-sch\u0027s in mixed versions.","commit_id":"60183ed88e84bc5ac9905967361958157f455e97"},{"author":{"_account_id":20140,"name":"Karthik Prabhu Vinod","email":"karthik.prabhu.vinod@intel.com","username":"kprabhuv"},"change_message_id":"a22b6085b864e4da2e0e64425292114ea5798538","unresolved":false,"context_lines":[{"line_number":237,"context_line":"  upgrade is performed."},{"line_number":238,"context_line":""},{"line_number":239,"context_line":"* Since Ocata, you also need to run ``cinder-manage db online-data-migrations``"},{"line_number":240,"context_line":"  command to make sure data migrations are apllied. The tool let\u0027s you limit"},{"line_number":241,"context_line":"  the impact of the data migrations by using ``--max_number`` option to limit"},{"line_number":242,"context_line":"  number of migrations executed in one run. You need to complete all of the"},{"line_number":243,"context_line":"  migrations before starting upgrade to the next version (e.g. you need to"}],"source_content_type":"text/x-rst","patch_set":10,"id":"7a77a97e_b773dbe6","line":240,"range":{"start_line":240,"start_character":43,"end_line":240,"end_character":50},"updated":"2016-11-22 00:10:26.000000000","message":"applied","commit_id":"60183ed88e84bc5ac9905967361958157f455e97"},{"author":{"_account_id":5997,"name":"Walt","display_name":"Hemna","email":"waboring@hemna.com","username":"walter-boring","status":"SAP"},"change_message_id":"25c9f4f0de72989fee429299bb9e82a009170a46","unresolved":false,"context_lines":[{"line_number":1,"context_line":".."},{"line_number":2,"context_line":"      Copyright 2016 Intel"},{"line_number":3,"context_line":"      All Rights Reserved."},{"line_number":4,"context_line":""},{"line_number":5,"context_line":"      Licensed under the Apache License, Version 2.0 (the \"License\"); you may"},{"line_number":6,"context_line":"      not use this file except in compliance with the License. You may obtain"}],"source_content_type":"text/x-rst","patch_set":11,"id":"1a6eadb0_6d156b3e","line":3,"updated":"2016-12-15 16:36:09.000000000","message":"is this really required ?","commit_id":"d9bf794df6ed839c6f7564a003e57c66fad79432"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"9646a07dc1b3a0167a206c79526e35e8f94abea3","unresolved":false,"context_lines":[{"line_number":1,"context_line":".."},{"line_number":2,"context_line":"      Copyright 2016 Intel"},{"line_number":3,"context_line":"      All Rights Reserved."},{"line_number":4,"context_line":""},{"line_number":5,"context_line":"      Licensed under the Apache License, Version 2.0 (the \"License\"); you may"},{"line_number":6,"context_line":"      not use this file except in compliance with the License. You may obtain"}],"source_content_type":"text/x-rst","patch_set":11,"id":"1a6eadb0_03c3cc57","line":3,"in_reply_to":"1a6eadb0_6d156b3e","updated":"2016-12-16 10:10:10.000000000","message":"It was in the file I\u0027ve copied from Nova. In Cinder it looks like we\u0027re not including license in the docs, so I\u0027ll remove it.","commit_id":"d9bf794df6ed839c6f7564a003e57c66fad79432"},{"author":{"_account_id":5997,"name":"Walt","display_name":"Hemna","email":"waboring@hemna.com","username":"walter-boring","status":"SAP"},"change_message_id":"25c9f4f0de72989fee429299bb9e82a009170a46","unresolved":false,"context_lines":[{"line_number":54,"context_line":""},{"line_number":55,"context_line":".. note::"},{"line_number":56,"context_line":"  This was tested with RabbitMQ messaging backend and may vary with other"},{"line_number":57,"context_line":"  backends."},{"line_number":58,"context_line":""},{"line_number":59,"context_line":"Online Data Migrations"},{"line_number":60,"context_line":"\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027"}],"source_content_type":"text/x-rst","patch_set":11,"id":"1a6eadb0_4da8afdb","line":57,"updated":"2016-12-15 16:36:09.000000000","message":"This shutdown might also take a while depending on what Cinder work is outstanding, such as a slow copy volume to image operation.","commit_id":"d9bf794df6ed839c6f7564a003e57c66fad79432"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"9646a07dc1b3a0167a206c79526e35e8f94abea3","unresolved":false,"context_lines":[{"line_number":54,"context_line":""},{"line_number":55,"context_line":".. note::"},{"line_number":56,"context_line":"  This was tested with RabbitMQ messaging backend and may vary with other"},{"line_number":57,"context_line":"  backends."},{"line_number":58,"context_line":""},{"line_number":59,"context_line":"Online Data Migrations"},{"line_number":60,"context_line":"\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027"}],"source_content_type":"text/x-rst","patch_set":11,"id":"1a6eadb0_06675ab7","line":57,"in_reply_to":"1a6eadb0_4da8afdb","updated":"2016-12-16 10:10:10.000000000","message":"Done","commit_id":"d9bf794df6ed839c6f7564a003e57c66fad79432"},{"author":{"_account_id":5997,"name":"Walt","display_name":"Hemna","email":"waboring@hemna.com","username":"walter-boring","status":"SAP"},"change_message_id":"25c9f4f0de72989fee429299bb9e82a009170a46","unresolved":false,"context_lines":[{"line_number":61,"context_line":""},{"line_number":62,"context_line":"To make DB schema migrations less painful to execute, since Liberty, all data"},{"line_number":63,"context_line":"migrations are banned from schema migration scripts. Instead, the migrations"},{"line_number":64,"context_line":"should be done by backgroud process in a live manner. In Ocata a new"},{"line_number":65,"context_line":"``cinder-manage db online-data-migrations`` utility was added for that purpose."},{"line_number":66,"context_line":"Before upgrading Ocata to Pike, you need to run this tool in the background,"},{"line_number":67,"context_line":"until it tells you no more migrations are needed. Note that you won\u0027t be able"}],"source_content_type":"text/x-rst","patch_set":11,"id":"1a6eadb0_ade5c338","line":64,"updated":"2016-12-15 16:36:09.000000000","message":"what do you mean here by a \u0027live manner\u0027?  Do you have to start cinder first and have the services running before issuing this?  Or can you start the API service only and then run the migration?","commit_id":"d9bf794df6ed839c6f7564a003e57c66fad79432"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"9646a07dc1b3a0167a206c79526e35e8f94abea3","unresolved":false,"context_lines":[{"line_number":61,"context_line":""},{"line_number":62,"context_line":"To make DB schema migrations less painful to execute, since Liberty, all data"},{"line_number":63,"context_line":"migrations are banned from schema migration scripts. Instead, the migrations"},{"line_number":64,"context_line":"should be done by backgroud process in a live manner. In Ocata a new"},{"line_number":65,"context_line":"``cinder-manage db online-data-migrations`` utility was added for that purpose."},{"line_number":66,"context_line":"Before upgrading Ocata to Pike, you need to run this tool in the background,"},{"line_number":67,"context_line":"until it tells you no more migrations are needed. Note that you won\u0027t be able"}],"source_content_type":"text/x-rst","patch_set":11,"id":"1a6eadb0_a670660c","line":64,"in_reply_to":"1a6eadb0_ade5c338","updated":"2016-12-16 10:10:10.000000000","message":"Done","commit_id":"d9bf794df6ed839c6f7564a003e57c66fad79432"},{"author":{"_account_id":7173,"name":"Scott DAngelo","email":"scott.dangelo@gmail.com","username":"scottda"},"change_message_id":"cfb858ddcdfff2f5a0c1878fc7a22b5da32970a9","unresolved":false,"context_lines":[{"line_number":83,"context_line":"However, most people have a data retention policy, of say 30 days or 90 days"},{"line_number":84,"context_line":"after which they will want to delete those entries. Not deleting those entries"},{"line_number":85,"context_line":"affects DB performance as indices grow very large and data migrations take"},{"line_number":86,"context_line":"longer as there is more data to migrate."},{"line_number":87,"context_line":""},{"line_number":88,"context_line":"Versioned object backports"},{"line_number":89,"context_line":"\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027"}],"source_content_type":"text/x-rst","patch_set":11,"id":"1a6eadb0_2d24535e","line":86,"updated":"2016-12-15 16:48:21.000000000","message":"Should we mention \u0027cinder-mange db purge \u003cdays\u003e\u0027 here?","commit_id":"d9bf794df6ed839c6f7564a003e57c66fad79432"},{"author":{"_account_id":5997,"name":"Walt","display_name":"Hemna","email":"waboring@hemna.com","username":"walter-boring","status":"SAP"},"change_message_id":"25c9f4f0de72989fee429299bb9e82a009170a46","unresolved":false,"context_lines":[{"line_number":83,"context_line":"However, most people have a data retention policy, of say 30 days or 90 days"},{"line_number":84,"context_line":"after which they will want to delete those entries. Not deleting those entries"},{"line_number":85,"context_line":"affects DB performance as indices grow very large and data migrations take"},{"line_number":86,"context_line":"longer as there is more data to migrate."},{"line_number":87,"context_line":""},{"line_number":88,"context_line":"Versioned object backports"},{"line_number":89,"context_line":"\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027"}],"source_content_type":"text/x-rst","patch_set":11,"id":"1a6eadb0_0d3eb78b","line":86,"updated":"2016-12-15 16:36:09.000000000","message":"shouldn\u0027t we have a policy that \u0027deleted\u0027 rows are never migrated?  I guess it depends on the migration.","commit_id":"d9bf794df6ed839c6f7564a003e57c66fad79432"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"9646a07dc1b3a0167a206c79526e35e8f94abea3","unresolved":false,"context_lines":[{"line_number":83,"context_line":"However, most people have a data retention policy, of say 30 days or 90 days"},{"line_number":84,"context_line":"after which they will want to delete those entries. Not deleting those entries"},{"line_number":85,"context_line":"affects DB performance as indices grow very large and data migrations take"},{"line_number":86,"context_line":"longer as there is more data to migrate."},{"line_number":87,"context_line":""},{"line_number":88,"context_line":"Versioned object backports"},{"line_number":89,"context_line":"\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027"}],"source_content_type":"text/x-rst","patch_set":11,"id":"1a6eadb0_66e72eac","line":86,"in_reply_to":"1a6eadb0_0d3eb78b","updated":"2016-12-16 10:10:10.000000000","message":"@Walt: Yup, it could depend on migration, so every case is different. But it\u0027s a good point that we should always look on minimizing affected rows.","commit_id":"d9bf794df6ed839c6f7564a003e57c66fad79432"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"9646a07dc1b3a0167a206c79526e35e8f94abea3","unresolved":false,"context_lines":[{"line_number":83,"context_line":"However, most people have a data retention policy, of say 30 days or 90 days"},{"line_number":84,"context_line":"after which they will want to delete those entries. Not deleting those entries"},{"line_number":85,"context_line":"affects DB performance as indices grow very large and data migrations take"},{"line_number":86,"context_line":"longer as there is more data to migrate."},{"line_number":87,"context_line":""},{"line_number":88,"context_line":"Versioned object backports"},{"line_number":89,"context_line":"\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027\u0027"}],"source_content_type":"text/x-rst","patch_set":11,"id":"1a6eadb0_2698d61b","line":86,"in_reply_to":"1a6eadb0_2d24535e","updated":"2016-12-16 10:10:10.000000000","message":"@Scott: Good point, I\u0027ll add it.","commit_id":"d9bf794df6ed839c6f7564a003e57c66fad79432"},{"author":{"_account_id":5997,"name":"Walt","display_name":"Hemna","email":"waboring@hemna.com","username":"walter-boring","status":"SAP"},"change_message_id":"25c9f4f0de72989fee429299bb9e82a009170a46","unresolved":false,"context_lines":[{"line_number":110,"context_line":"  directly into Newton, you need to upgrade to Mitaka first."},{"line_number":111,"context_line":""},{"line_number":112,"context_line":"* To avoid dependency hell it is advised to have your Cinder services deployed"},{"line_number":113,"context_line":"  separately in containers or Python venvs."},{"line_number":114,"context_line":""},{"line_number":115,"context_line":"* Note that Cinder is basing version detection on what is reported in the"},{"line_number":116,"context_line":"  ``services`` table in the DB. Before upgrade make sure you don\u0027t have any"}],"source_content_type":"text/x-rst","patch_set":11,"id":"1a6eadb0_8d71e778","line":113,"updated":"2016-12-15 16:36:09.000000000","message":"Do we even support running every service in a container yet?  Where is that testing that proves it works?  We should be careful stating that here.","commit_id":"d9bf794df6ed839c6f7564a003e57c66fad79432"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"9646a07dc1b3a0167a206c79526e35e8f94abea3","unresolved":false,"context_lines":[{"line_number":110,"context_line":"  directly into Newton, you need to upgrade to Mitaka first."},{"line_number":111,"context_line":""},{"line_number":112,"context_line":"* To avoid dependency hell it is advised to have your Cinder services deployed"},{"line_number":113,"context_line":"  separately in containers or Python venvs."},{"line_number":114,"context_line":""},{"line_number":115,"context_line":"* Note that Cinder is basing version detection on what is reported in the"},{"line_number":116,"context_line":"  ``services`` table in the DB. Before upgrade make sure you don\u0027t have any"}],"source_content_type":"text/x-rst","patch_set":11,"id":"1a6eadb0_51566a06","line":113,"in_reply_to":"1a6eadb0_8d71e778","updated":"2016-12-16 10:10:10.000000000","message":"@Walt: Kolla is deploying all of the Cinder services in containers, so it\u0027s definitely possible. From the perspective of Cinder service, running in a privileged container with host networking is no different than running as a process.","commit_id":"d9bf794df6ed839c6f7564a003e57c66fad79432"},{"author":{"_account_id":7173,"name":"Scott DAngelo","email":"scott.dangelo@gmail.com","username":"scottda"},"change_message_id":"cfb858ddcdfff2f5a0c1878fc7a22b5da32970a9","unresolved":false,"context_lines":[{"line_number":114,"context_line":""},{"line_number":115,"context_line":"* Note that Cinder is basing version detection on what is reported in the"},{"line_number":116,"context_line":"  ``services`` table in the DB. Before upgrade make sure you don\u0027t have any"},{"line_number":117,"context_line":"  orphaned old records there, because these can block starting newer services."},{"line_number":118,"context_line":""},{"line_number":119,"context_line":"* Assumed service upgrade order is cinder-api, cinder-scheduler, cinder-volume"},{"line_number":120,"context_line":"  and finally cinder-backup."}],"source_content_type":"text/x-rst","patch_set":11,"id":"1a6eadb0_0da8778c","line":117,"updated":"2016-12-15 16:48:21.000000000","message":"TODO: if this is a common problem (having orphaned records), perhaps we need an API to clean them up. I assume that direct DB access is the only way currently.","commit_id":"d9bf794df6ed839c6f7564a003e57c66fad79432"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"9646a07dc1b3a0167a206c79526e35e8f94abea3","unresolved":false,"context_lines":[{"line_number":114,"context_line":""},{"line_number":115,"context_line":"* Note that Cinder is basing version detection on what is reported in the"},{"line_number":116,"context_line":"  ``services`` table in the DB. Before upgrade make sure you don\u0027t have any"},{"line_number":117,"context_line":"  orphaned old records there, because these can block starting newer services."},{"line_number":118,"context_line":""},{"line_number":119,"context_line":"* Assumed service upgrade order is cinder-api, cinder-scheduler, cinder-volume"},{"line_number":120,"context_line":"  and finally cinder-backup."}],"source_content_type":"text/x-rst","patch_set":11,"id":"1a6eadb0_d1619ad4","line":117,"in_reply_to":"1a6eadb0_0da8778c","updated":"2016-12-16 10:10:10.000000000","message":"We do have such command in cinder-manage. I don\u0027t know why I\u0027m not mentioning it here. I\u0027ll add it.","commit_id":"d9bf794df6ed839c6f7564a003e57c66fad79432"},{"author":{"_account_id":7173,"name":"Scott DAngelo","email":"scott.dangelo@gmail.com","username":"scottda"},"change_message_id":"cfb858ddcdfff2f5a0c1878fc7a22b5da32970a9","unresolved":false,"context_lines":[{"line_number":124,"context_line":""},{"line_number":125,"context_line":"To reduce downtime, the services can be upgraded in a rolling fashion. It"},{"line_number":126,"context_line":"means upgrading a few services at a time. To minimise downtime you need to have"},{"line_number":127,"context_line":"HA Cinder deployment, so at the moment a service is upgraded, you\u0027ll keep"},{"line_number":128,"context_line":"other service instances running."},{"line_number":129,"context_line":""},{"line_number":130,"context_line":"Before maintenance window"}],"source_content_type":"text/x-rst","patch_set":11,"id":"1a6eadb0_0d8f972a","line":127,"updated":"2016-12-15 16:48:21.000000000","message":"We should call out what is currently HA (c-api, c-sched), and what will be (hopefully) in O or future release (c-vol).\nI see the detail of this comes later, so maybe it\u0027s OK to leave as is...","commit_id":"d9bf794df6ed839c6f7564a003e57c66fad79432"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"9646a07dc1b3a0167a206c79526e35e8f94abea3","unresolved":false,"context_lines":[{"line_number":124,"context_line":""},{"line_number":125,"context_line":"To reduce downtime, the services can be upgraded in a rolling fashion. It"},{"line_number":126,"context_line":"means upgrading a few services at a time. To minimise downtime you need to have"},{"line_number":127,"context_line":"HA Cinder deployment, so at the moment a service is upgraded, you\u0027ll keep"},{"line_number":128,"context_line":"other service instances running."},{"line_number":129,"context_line":""},{"line_number":130,"context_line":"Before maintenance window"}],"source_content_type":"text/x-rst","patch_set":11,"id":"1a6eadb0_91fc82b9","line":127,"in_reply_to":"1a6eadb0_0d8f972a","updated":"2016-12-16 10:10:10.000000000","message":"Well, c-vol supports HA as well - it\u0027s just Active-Passive. And it\u0027s definitely recommended to have A/P setup for your c-vol to do a live upgrade.","commit_id":"d9bf794df6ed839c6f7564a003e57c66fad79432"},{"author":{"_account_id":5997,"name":"Walt","display_name":"Hemna","email":"waboring@hemna.com","username":"walter-boring","status":"SAP"},"change_message_id":"25c9f4f0de72989fee429299bb9e82a009170a46","unresolved":false,"context_lines":[{"line_number":134,"context_line":"  without interrupting your existing installation, install new Cinder code in"},{"line_number":135,"context_line":"  new venv or a container and run the DB sync (``cinder-manage db sync``)."},{"line_number":136,"context_line":"  These schema change operations should have minimal or no effect on"},{"line_number":137,"context_line":"  performance, and should not cause any operations to fail."},{"line_number":138,"context_line":""},{"line_number":139,"context_line":"* At this point, new columns and tables may exist in the database. These"},{"line_number":140,"context_line":"  DB schema changes are done in a way that both the N and N+1 release can"}],"source_content_type":"text/x-rst","patch_set":11,"id":"1a6eadb0_4d298f36","line":137,"updated":"2016-12-15 16:36:09.000000000","message":"are you sure about this?   Some DB schema migrations can cause columns to be dropped and new ones added.   an older cinder service might need those columns to work.","commit_id":"d9bf794df6ed839c6f7564a003e57c66fad79432"},{"author":{"_account_id":12112,"name":"Tom Swanson","email":"tom_swanson@dell.com","username":"tomswanson"},"change_message_id":"ae184ff0878cb2e9f400bbd3b74e7cd661b19379","unresolved":false,"context_lines":[{"line_number":134,"context_line":"  without interrupting your existing installation, install new Cinder code in"},{"line_number":135,"context_line":"  new venv or a container and run the DB sync (``cinder-manage db sync``)."},{"line_number":136,"context_line":"  These schema change operations should have minimal or no effect on"},{"line_number":137,"context_line":"  performance, and should not cause any operations to fail."},{"line_number":138,"context_line":""},{"line_number":139,"context_line":"* At this point, new columns and tables may exist in the database. These"},{"line_number":140,"context_line":"  DB schema changes are done in a way that both the N and N+1 release can"}],"source_content_type":"text/x-rst","patch_set":11,"id":"1a6eadb0_338112c4","line":137,"in_reply_to":"1a6eadb0_4d298f36","updated":"2016-12-15 17:22:31.000000000","message":"Just deny dropping columns that are required in the previous release. Deprecate them for at least one release. Since this requires you go o to p to q rather than o to q?","commit_id":"d9bf794df6ed839c6f7564a003e57c66fad79432"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"9646a07dc1b3a0167a206c79526e35e8f94abea3","unresolved":false,"context_lines":[{"line_number":134,"context_line":"  without interrupting your existing installation, install new Cinder code in"},{"line_number":135,"context_line":"  new venv or a container and run the DB sync (``cinder-manage db sync``)."},{"line_number":136,"context_line":"  These schema change operations should have minimal or no effect on"},{"line_number":137,"context_line":"  performance, and should not cause any operations to fail."},{"line_number":138,"context_line":""},{"line_number":139,"context_line":"* At this point, new columns and tables may exist in the database. These"},{"line_number":140,"context_line":"  DB schema changes are done in a way that both the N and N+1 release can"}],"source_content_type":"text/x-rst","patch_set":11,"id":"1a6eadb0_11e072cc","line":137,"in_reply_to":"1a6eadb0_4d298f36","updated":"2016-12-16 10:10:10.000000000","message":"There\u0027s a Mitaka spec addressing this [1]. Basically we aim not to drop columns immediately.\n\nNote that other teams are exploring other possibilities of achieving this ([2], [3]), and we\u0027ll certainly look to adopt an emerging solution if it\u0027s better than what we have now.\n\n[1] https://specs.openstack.org/openstack/cinder-specs/specs/mitaka/online-schema-upgrades.html\n[2] https://specs.openstack.org/openstack/glance-specs/specs/ocata/approved/glance/database-strategy-for-rolling-upgrades.html\n[3] https://specs.openstack.org/openstack/keystone-specs/specs/mitaka/online-schema-migration.html","commit_id":"d9bf794df6ed839c6f7564a003e57c66fad79432"},{"author":{"_account_id":12825,"name":"Szymon Wróblewski","email":"szymon.wroblewski@ovhcloud.com","username":"bluex"},"change_message_id":"81153f94606bb8b2f9b7650ea22e8c3ef5667e0a","unresolved":false,"context_lines":[{"line_number":153,"context_line":""},{"line_number":154,"context_line":"   .. note::"},{"line_number":155,"context_line":""},{"line_number":156,"context_line":"     You may want to start another instance of older c-api to handle the load"},{"line_number":157,"context_line":"     while you\u0027re upgrading your original services."},{"line_number":158,"context_line":""},{"line_number":159,"context_line":"2. Then you should repeat first step for all of the cinder-api services."},{"line_number":160,"context_line":""}],"source_content_type":"text/x-rst","patch_set":11,"id":"5a74a57a_a8547621","line":157,"range":{"start_line":156,"start_character":5,"end_line":157,"end_character":51},"updated":"2016-11-29 21:11:54.000000000","message":"Maybe reverse order of 1. then - first plug in new c-api to load balancer then remove old one? That would make this note unnecessary.","commit_id":"d9bf794df6ed839c6f7564a003e57c66fad79432"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"aeae8ead0ecccb308a920575c287083ca83b636b","unresolved":false,"context_lines":[{"line_number":153,"context_line":""},{"line_number":154,"context_line":"   .. note::"},{"line_number":155,"context_line":""},{"line_number":156,"context_line":"     You may want to start another instance of older c-api to handle the load"},{"line_number":157,"context_line":"     while you\u0027re upgrading your original services."},{"line_number":158,"context_line":""},{"line_number":159,"context_line":"2. Then you should repeat first step for all of the cinder-api services."},{"line_number":160,"context_line":""}],"source_content_type":"text/x-rst","patch_set":11,"id":"5a74a57a_ceda83e9","line":157,"range":{"start_line":156,"start_character":5,"end_line":157,"end_character":51},"in_reply_to":"5a74a57a_a8547621","updated":"2016-11-30 10:10:55.000000000","message":"Hm, it depends. Sometimes you don\u0027t have additional resources to spawn a new API service and your load is small enough to be handled by 2 nodes. In that case you don\u0027t want to add new c-api first.","commit_id":"d9bf794df6ed839c6f7564a003e57c66fad79432"},{"author":{"_account_id":12825,"name":"Szymon Wróblewski","email":"szymon.wroblewski@ovhcloud.com","username":"bluex"},"change_message_id":"bc06f2bea46b4b23262ba3c4b59a5e6cb5d008a1","unresolved":false,"context_lines":[{"line_number":153,"context_line":""},{"line_number":154,"context_line":"   .. note::"},{"line_number":155,"context_line":""},{"line_number":156,"context_line":"     You may want to start another instance of older c-api to handle the load"},{"line_number":157,"context_line":"     while you\u0027re upgrading your original services."},{"line_number":158,"context_line":""},{"line_number":159,"context_line":"2. Then you should repeat first step for all of the cinder-api services."},{"line_number":160,"context_line":""}],"source_content_type":"text/x-rst","patch_set":11,"id":"5a74a57a_7fc028e1","line":157,"range":{"start_line":156,"start_character":5,"end_line":157,"end_character":51},"in_reply_to":"5a74a57a_ceda83e9","updated":"2016-11-30 16:23:31.000000000","message":"Ok, makes sense.","commit_id":"d9bf794df6ed839c6f7564a003e57c66fad79432"},{"author":{"_account_id":19554,"name":"Nam","email":"namnh68@fsoft.com.vn","username":"namnh"},"change_message_id":"091a396718f560b9e326ef5918a4db385ee579ab","unresolved":false,"context_lines":[{"line_number":167,"context_line":"4. Repeat third step for all of your cinder-scheduler services."},{"line_number":168,"context_line":""},{"line_number":169,"context_line":"5. Then you proceed to upgrade cinder-volume services. The problem here is that"},{"line_number":170,"context_line":"   due to Active/Passive character of this service, you\u0027re unable to run"},{"line_number":171,"context_line":"   multiple instances of cinder-volume managing a single volume backend. This"},{"line_number":172,"context_line":"   means that there will be a moment when you won\u0027t have any cinder-volume in"},{"line_number":173,"context_line":"   your deployment and you want that disruption to be as short as possible."}],"source_content_type":"text/x-rst","patch_set":11,"id":"1a6eadb0_9488fd72","line":170,"range":{"start_line":170,"start_character":10,"end_line":170,"end_character":50},"updated":"2016-12-16 02:50:16.000000000","message":"In my option, it\u0027s better if we can add more information for a scenario:\n\n- Scenario: When users are creating volumes then admin stop cinder-volume at N release and start new cinder-volume at (N+1) release.\n\n- Result: There will be some volumes always show \"creating\" status and the volumes cannot be created.\n\n- Reason: When we stop cinder-volume at N release on server1, cinder-scheduler did not update the status of the cinder-volume real time so it still send the requests to create volumes on server1, after period of time, cinder-scheduler will update real status and send the remain requests to cinder-volume at (N+1) release on server2.\n\n- Solution:\n\n- 1. Before killing the cinder-volume on server1, we need to disable it by using \"cinder service-disable\"\n\nNote: If we have only one cinder-volume (architecture A/P) then we will get NoVailidHost for volumes\n\n- 2. Making both of our cinder-volume to listen on the same queue.\n\nThat\u0027s content after I have conversation with the author of this path set :)","commit_id":"d9bf794df6ed839c6f7564a003e57c66fad79432"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"9646a07dc1b3a0167a206c79526e35e8f94abea3","unresolved":false,"context_lines":[{"line_number":167,"context_line":"4. Repeat third step for all of your cinder-scheduler services."},{"line_number":168,"context_line":""},{"line_number":169,"context_line":"5. Then you proceed to upgrade cinder-volume services. The problem here is that"},{"line_number":170,"context_line":"   due to Active/Passive character of this service, you\u0027re unable to run"},{"line_number":171,"context_line":"   multiple instances of cinder-volume managing a single volume backend. This"},{"line_number":172,"context_line":"   means that there will be a moment when you won\u0027t have any cinder-volume in"},{"line_number":173,"context_line":"   your deployment and you want that disruption to be as short as possible."}],"source_content_type":"text/x-rst","patch_set":11,"id":"1a6eadb0_ac93e559","line":170,"range":{"start_line":170,"start_character":10,"end_line":170,"end_character":50},"in_reply_to":"1a6eadb0_9488fd72","updated":"2016-12-16 10:10:10.000000000","message":"Done","commit_id":"d9bf794df6ed839c6f7564a003e57c66fad79432"},{"author":{"_account_id":7173,"name":"Scott DAngelo","email":"scott.dangelo@gmail.com","username":"scottda"},"change_message_id":"cfb858ddcdfff2f5a0c1878fc7a22b5da32970a9","unresolved":false,"context_lines":[{"line_number":213,"context_line":""},{"line_number":214,"context_line":"* Once all services are running the new code, double check in the DB that"},{"line_number":215,"context_line":"  there are no old orphaned service records. Cinder is basing its RPC version"},{"line_number":216,"context_line":"  detection on that, so stale records can prevent you from going forward."},{"line_number":217,"context_line":""},{"line_number":218,"context_line":"* Now all services are upgraded, we need to send the ``SIGHUP`` signal, so"},{"line_number":219,"context_line":"  all the services clear any cached service version data. When a new service"}],"source_content_type":"text/x-rst","patch_set":11,"id":"1a6eadb0_0d67974d","line":216,"updated":"2016-12-15 16:48:21.000000000","message":"This could use some more detail, i.e. What these orphaned records would look like, and how to clean them up.","commit_id":"d9bf794df6ed839c6f7564a003e57c66fad79432"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"9646a07dc1b3a0167a206c79526e35e8f94abea3","unresolved":false,"context_lines":[{"line_number":213,"context_line":""},{"line_number":214,"context_line":"* Once all services are running the new code, double check in the DB that"},{"line_number":215,"context_line":"  there are no old orphaned service records. Cinder is basing its RPC version"},{"line_number":216,"context_line":"  detection on that, so stale records can prevent you from going forward."},{"line_number":217,"context_line":""},{"line_number":218,"context_line":"* Now all services are upgraded, we need to send the ``SIGHUP`` signal, so"},{"line_number":219,"context_line":"  all the services clear any cached service version data. When a new service"}],"source_content_type":"text/x-rst","patch_set":11,"id":"1a6eadb0_accf6554","line":216,"in_reply_to":"1a6eadb0_0d67974d","updated":"2016-12-16 10:10:10.000000000","message":"Done","commit_id":"d9bf794df6ed839c6f7564a003e57c66fad79432"},{"author":{"_account_id":11904,"name":"Sean McGinnis","email":"sean.mcginnis@gmail.com","username":"SeanM"},"change_message_id":"16d5dec5caf4b1a8cb0c1a7b53e27532a0d43384","unresolved":false,"context_lines":[{"line_number":240,"context_line":"  command to make sure data migrations are applied. The tool let\u0027s you limit"},{"line_number":241,"context_line":"  the impact of the data migrations by using ``--max_number`` option to limit"},{"line_number":242,"context_line":"  number of migrations executed in one run. You need to complete all of the"},{"line_number":243,"context_line":"  migrations before starting upgrade to the next version (e.g. you need to"},{"line_number":244,"context_line":"  complete Ocata\u0027s data migrations before proceeding with upgrade to Pike; you"},{"line_number":245,"context_line":"  won\u0027t be able to execute Pike\u0027s DB schema migrations before completing"},{"line_number":246,"context_line":"  Ocata\u0027s data migrations)."}],"source_content_type":"text/x-rst","patch_set":11,"id":"5a74a57a_ca5481cb","line":243,"updated":"2016-11-28 21:19:06.000000000","message":"Actually, you should be able to upgrade from Newton directly to Pike. Migrations will see the highest version number migrated to and apply additional migration versions from that point.\n\nOr am I missing something in the context here?","commit_id":"d9bf794df6ed839c6f7564a003e57c66fad79432"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"fca58d60983bfd71f8877bec93a3b70d39523beb","unresolved":false,"context_lines":[{"line_number":240,"context_line":"  command to make sure data migrations are applied. The tool let\u0027s you limit"},{"line_number":241,"context_line":"  the impact of the data migrations by using ``--max_number`` option to limit"},{"line_number":242,"context_line":"  number of migrations executed in one run. You need to complete all of the"},{"line_number":243,"context_line":"  migrations before starting upgrade to the next version (e.g. you need to"},{"line_number":244,"context_line":"  complete Ocata\u0027s data migrations before proceeding with upgrade to Pike; you"},{"line_number":245,"context_line":"  won\u0027t be able to execute Pike\u0027s DB schema migrations before completing"},{"line_number":246,"context_line":"  Ocata\u0027s data migrations)."}],"source_content_type":"text/x-rst","patch_set":11,"id":"5a74a57a_f9dbbd4c","line":243,"in_reply_to":"5a74a57a_ca5481cb","updated":"2016-11-29 13:14:08.000000000","message":"We\u0027re talking here about online *data* migrations, not classic *schema* migrations. These are a little bit different and it would be quite hard to be able to stretch them through multiple releases. In case of an upgrade with downtime from X to X+2, you should run X+2 schema migrations (you DB schema will go from X to X+2), then X+1 and X+2 data migrations.\n\nI\u0027m not too worried about that as operators should be experienced with same mechanism being present in Nova since Kilo.","commit_id":"d9bf794df6ed839c6f7564a003e57c66fad79432"},{"author":{"_account_id":11600,"name":"Michał Dulko","email":"michal.dulko@gmail.com","username":"dulek"},"change_message_id":"9c27c97be86e2b6b5a1f95082133a80ee3035546","unresolved":false,"context_lines":[{"line_number":240,"context_line":"  command to make sure data migrations are applied. The tool let\u0027s you limit"},{"line_number":241,"context_line":"  the impact of the data migrations by using ``--max_number`` option to limit"},{"line_number":242,"context_line":"  number of migrations executed in one run. You need to complete all of the"},{"line_number":243,"context_line":"  migrations before starting upgrade to the next version (e.g. you need to"},{"line_number":244,"context_line":"  complete Ocata\u0027s data migrations before proceeding with upgrade to Pike; you"},{"line_number":245,"context_line":"  won\u0027t be able to execute Pike\u0027s DB schema migrations before completing"},{"line_number":246,"context_line":"  Ocata\u0027s data migrations)."}],"source_content_type":"text/x-rst","patch_set":11,"id":"5a74a57a_77e17129","line":243,"in_reply_to":"5a74a57a_f9dbbd4c","updated":"2016-11-29 15:08:22.000000000","message":"One update - actually as in every release first schema migration should explode if previous release data migrations weren\u0027t applied (e.g. [1]), the user can be forced to apply schema migrations for each release separately. Anyway he will get notified about required actions by cinder-manage db sync error message.\n\n[1] https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/migrate_repo/versions/291_enforce_flavors_migrated.py#L30-L35","commit_id":"d9bf794df6ed839c6f7564a003e57c66fad79432"}]}