)]}'
{"specs/pike/approved/memory-iteration-based-live-migration.rst":[{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"b04aa9e0dee649258c09c8e2db77373bc6ada205","unresolved":false,"context_lines":[{"line_number":15,"context_line":""},{"line_number":16,"context_line":"Problem description"},{"line_number":17,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":18,"context_line":""},{"line_number":19,"context_line":"Nova tries to ensure the maximum liveness of a VM during the live-migration"},{"line_number":20,"context_line":"process and it is based on following libvirt config options:"},{"line_number":21,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a30ddce_1a01c905","line":18,"updated":"2017-03-09 12:09:57.000000000","message":"We should discuss how the size of the VM, the network bandwidth and the business of the VM affect how long live-migration takes.\n\nNoting that the current config only looks at scaling time values up with the size of the VM.\n\nThe API being added is to help with per VM business tracking.","commit_id":"afe6c4559f9db674b22e5c4c3229a58a7953a5f0"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"b04aa9e0dee649258c09c8e2db77373bc6ada205","unresolved":false,"context_lines":[{"line_number":26,"context_line":" downtime value."},{"line_number":27,"context_line":""},{"line_number":28,"context_line":"* ``live_migration_downtime_delay`` - Time to wait, in seconds, between each"},{"line_number":29,"context_line":" step increase of the migration downtime."},{"line_number":30,"context_line":""},{"line_number":31,"context_line":"All these options and its usage is very complicated and require deep"},{"line_number":32,"context_line":"knowledge by end user to properly tune it for their cluster. With current"}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a30ddce_777020dd","line":29,"updated":"2017-03-09 12:09:57.000000000","message":"Value is per GiB of guest RAM + disk to be transferred, with lower bound of a minimum of 2 GiB per device.\n\nIt doesn\u0027t take into account the network bandwidth that is available.","commit_id":"afe6c4559f9db674b22e5c4c3229a58a7953a5f0"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"b04aa9e0dee649258c09c8e2db77373bc6ada205","unresolved":false,"context_lines":[{"line_number":28,"context_line":"* ``live_migration_downtime_delay`` - Time to wait, in seconds, between each"},{"line_number":29,"context_line":" step increase of the migration downtime."},{"line_number":30,"context_line":""},{"line_number":31,"context_line":"All these options and its usage is very complicated and require deep"},{"line_number":32,"context_line":"knowledge by end user to properly tune it for their cluster. With current"},{"line_number":33,"context_line":"complexity and default values ``live_migration_completion_timeout`` will hit"},{"line_number":34,"context_line":"even before VM reach maximum ``live_migration_downtime``."}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a30ddce_57e6bcb0","line":31,"range":{"start_line":31,"start_character":32,"end_line":31,"end_character":34},"updated":"2017-03-09 12:09:57.000000000","message":"Nit: are","commit_id":"afe6c4559f9db674b22e5c4c3229a58a7953a5f0"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"b04aa9e0dee649258c09c8e2db77373bc6ada205","unresolved":false,"context_lines":[{"line_number":29,"context_line":" step increase of the migration downtime."},{"line_number":30,"context_line":""},{"line_number":31,"context_line":"All these options and its usage is very complicated and require deep"},{"line_number":32,"context_line":"knowledge by end user to properly tune it for their cluster. With current"},{"line_number":33,"context_line":"complexity and default values ``live_migration_completion_timeout`` will hit"},{"line_number":34,"context_line":"even before VM reach maximum ``live_migration_downtime``."},{"line_number":35,"context_line":""},{"line_number":36,"context_line":""},{"line_number":37,"context_line":"Use Cases"}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a30ddce_b77618d5","line":34,"range":{"start_line":32,"start_character":61,"end_line":34,"end_character":57},"updated":"2017-03-09 12:09:57.000000000","message":"I don\u0027t believe that to be true, I think its close though.\n\n75 * 10 \u003d 750 seconds?","commit_id":"afe6c4559f9db674b22e5c4c3229a58a7953a5f0"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"b04aa9e0dee649258c09c8e2db77373bc6ada205","unresolved":false,"context_lines":[{"line_number":33,"context_line":"complexity and default values ``live_migration_completion_timeout`` will hit"},{"line_number":34,"context_line":"even before VM reach maximum ``live_migration_downtime``."},{"line_number":35,"context_line":""},{"line_number":36,"context_line":""},{"line_number":37,"context_line":"Use Cases"},{"line_number":38,"context_line":"---------"},{"line_number":39,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a30ddce_171a847d","line":36,"updated":"2017-03-09 12:09:57.000000000","message":"I would describe the completion_timeout: \nValue is per GiB of guest RAM + disk to be transferred, with lower bound of a minimum of 2 GiB. Should usually be larger than downtime delay * downtime steps. Set to 0 to disable timeouts.","commit_id":"afe6c4559f9db674b22e5c4c3229a58a7953a5f0"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"b04aa9e0dee649258c09c8e2db77373bc6ada205","unresolved":false,"context_lines":[{"line_number":38,"context_line":"---------"},{"line_number":39,"context_line":""},{"line_number":40,"context_line":"As an operator, I want simple and easy to understand process to rampup for"},{"line_number":41,"context_line":"live-migration maximum downtime ."},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"Proposed change"},{"line_number":44,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a30ddce_9ab1394a","line":41,"updated":"2017-03-09 12:09:57.000000000","message":"I think you missed the aim here.\n\n1) default\n\nAn operator ideally wants live-migration to complete for most VMs. For the VMs where they are very busy, and will not be able to complete, without having more downtime than live_migration_downtime, we abort the live-migration so as not to affect the guest.\n\nFor guests that are going to succeed, we try to wait a little bit longer before triggering the live-migration switchover to get the minimal impact to the guest.\n\nThe default keeps 500 second downtime, but has \n\n3)\nOperator is happy with more downtime, and they have many busy VMs that don\u0027t want to move. So they just tweak live_migration_downtime\n\ni.e. no change here, except is clearer we will now reach the max downtime sooner.\n\n3)\nAnother use case is to try and make all live-migrates as fast as possible by having no ramp up, just start with the max downtime.\nlive_migration_iterations_to_max_downtime \u003d 0, and live_migration_downtime increased to 1000ms\n\n4)\nHas lots of busy VMs, that take longer to reach the max_downtime, and willing to wait, so increases live_migration_iterations_to_timeout","commit_id":"afe6c4559f9db674b22e5c4c3229a58a7953a5f0"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"b04aa9e0dee649258c09c8e2db77373bc6ada205","unresolved":false,"context_lines":[{"line_number":46,"context_line":"Deprecate the current config options ``live_migration_downtime_steps`` and"},{"line_number":47,"context_line":"```live_migration_downtime_delay` to calculate the steps and delay in"},{"line_number":48,"context_line":"live-migration process and replace it with a new single config option"},{"line_number":49,"context_line":"``live_migration_iterations_to_max_downtime`` based on memory iterations."},{"line_number":50,"context_line":"This option describes the number of iterations required to reach the maximum"},{"line_number":51,"context_line":"permitted downtime for live migration switchover, as defined by"},{"line_number":52,"context_line":"``live_migration_downtime``."}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a30ddce_ba399dd9","line":49,"updated":"2017-03-09 12:09:57.000000000","message":"if it \u003d\u003d 0, then we start with the max downtime? Maybe make that explicit.\n\nHow many steps do we take to reach max downtime? what value of downtime to we start with?\n\nI think we go for the current min_downtime as the start value, and the number of steps as either 5 or 10?\n\nI would write out an example iteration so its very clear what we mean.","commit_id":"afe6c4559f9db674b22e5c4c3229a58a7953a5f0"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"b04aa9e0dee649258c09c8e2db77373bc6ada205","unresolved":false,"context_lines":[{"line_number":49,"context_line":"``live_migration_iterations_to_max_downtime`` based on memory iterations."},{"line_number":50,"context_line":"This option describes the number of iterations required to reach the maximum"},{"line_number":51,"context_line":"permitted downtime for live migration switchover, as defined by"},{"line_number":52,"context_line":"``live_migration_downtime``."},{"line_number":53,"context_line":""},{"line_number":54,"context_line":"TODO: Add more here"},{"line_number":55,"context_line":"Alternatives"}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a30ddce_1aeb8914","line":52,"updated":"2017-03-09 12:09:57.000000000","message":"What is the default?  How does this related to live_migration_completion_timeout?\n\nI think we probably want also add live_migration_iterations_to_timeout.\n\nThat way the default of live_migration_iterations_to_max_downtime is live_migration_iterations_to_timeout \\ 2\n\nThis value is naturally independent of both the size of the VM, and the available bandwidth.","commit_id":"afe6c4559f9db674b22e5c4c3229a58a7953a5f0"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"d5863e02d371e73d59aa04f8836913e8dfa56b80","unresolved":false,"context_lines":[{"line_number":41,"context_line":"complexity and default values VM will ramp upto maximum downtime (500) in 750"},{"line_number":42,"context_line":"seconds (steps(10) * delay(75)) with only 50 seconds left till"},{"line_number":43,"context_line":"``live_migration_completion_timeout\u003d800`` expires and abort the operration. It"},{"line_number":44,"context_line":"does not allow enough time for a VM to move after maximum downtime is reached."},{"line_number":45,"context_line":""},{"line_number":46,"context_line":"Use Cases"},{"line_number":47,"context_line":"---------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"9a30ddce_a0a3b5b0","line":44,"updated":"2017-03-13 17:47:59.000000000","message":"Lets try describe live-migration, and get the users mental model of live-migration very clear.\n\nFirst there are phases:\n\n* There are ramp up allowed downtime phase (we wait longer to reduce downtime). Describe the switch over decision.\n* Now we reach max, we give time for VMs that keep dirtying memory\n* Finally we hit completion timeout, where we have waited so long we assume this operation will never complete. As such, we either abort or force the live-migration. Abort to minimize impact to the VM, force to maximize the chances of live-migration completing\n\nNow things that affect this:\n* VM site: how much memory needs to be copied\n* network bandwidth: how fast you can copy memory\n* How busy the VM is: how many iterations are needed, potentially infinite, before all the memory changes are copied.\n\nWhat we want to do:\n\n* make live-migration complete most of the time, using out the box defaults, and make a reasonable trade off around VM live-ness\n* when admin wants to take a different trade off, make it easy to change and understand that trade off.","commit_id":"f72f0249f4fb157f9e8e73eeb6fe4e985e82529e"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"d5863e02d371e73d59aa04f8836913e8dfa56b80","unresolved":false,"context_lines":[{"line_number":45,"context_line":""},{"line_number":46,"context_line":"Use Cases"},{"line_number":47,"context_line":"---------"},{"line_number":48,"context_line":""},{"line_number":49,"context_line":"* Operators ideally want live-migration to complete for all of the VMs with"},{"line_number":50,"context_line":"  default config values. For busy VMs, which might not complete without having"},{"line_number":51,"context_line":"  more downtime than ``live_migration_downtime``, we abort the live-migration"}],"source_content_type":"text/x-rst","patch_set":3,"id":"9a30ddce_40c83955","line":48,"updated":"2017-03-13 17:47:59.000000000","message":"Maybe summarize the issues:\n\nIts a trade off between live-migration completing and VMs not being impacted by the live-migration.\n\nTalk about specific use cases:\n\n* empty host before reboot, it really has to move\n* workload rebalance, i.e. move noisy neighbour, really don\u0027t want to affect the workload.\n\n\nWhen it comes to the above tuning of the workflow:\n\n* Ramp up downtime quicker, to make live-migrate complete quicker, at the cost of more VM downtime\n* Allow more max downtime of the VM, to ensure live-migrate is more likely to complete, given your network bandwidth","commit_id":"f72f0249f4fb157f9e8e73eeb6fe4e985e82529e"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"068a7e5a81bc261012b5d4fb0dd072a52bcab4a3","unresolved":false,"context_lines":[{"line_number":68,"context_line":""},{"line_number":69,"context_line":"Proposed change"},{"line_number":70,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":71,"context_line":""},{"line_number":72,"context_line":"Deprecate the current config options ``live_migration_downtime_steps`` and"},{"line_number":73,"context_line":"```live_migration_downtime_delay` to calculate the steps and delay in"},{"line_number":74,"context_line":"live-migration process and replace it with a new single config option"}],"source_content_type":"text/x-rst","patch_set":3,"id":"7a25e98e_26ad1632","line":71,"updated":"2017-03-13 18:01:37.000000000","message":"Move to using memory iteration instead of time multiplied by VM size estimate, our defaults should be independent of both VM size and bandwidth differences. We also naturally wait longer for very busy VMs, but not so longer for not so busy VMs. This should reduce the need for many people to change the defaults we pick, as they should be a better match for more situations.\n\nHowever, this all ignores the disk copying. During the majority of the disk copy is done before we start the memory copy. To protect against this taking too long, we should keep a time based, multiplied by VM size completion timeout.\n\nTODO - need go profile to this to see if memory iterations help.","commit_id":"f72f0249f4fb157f9e8e73eeb6fe4e985e82529e"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"d5863e02d371e73d59aa04f8836913e8dfa56b80","unresolved":false,"context_lines":[{"line_number":89,"context_line":"Alternatives"},{"line_number":90,"context_line":"------------"},{"line_number":91,"context_line":""},{"line_number":92,"context_line":"Keep using the current available options."},{"line_number":93,"context_line":""},{"line_number":94,"context_line":"Data model impact"},{"line_number":95,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"9a30ddce_2098a577","line":92,"updated":"2017-03-13 17:47:59.000000000","message":"And documenting it better, so its easier to understand.","commit_id":"f72f0249f4fb157f9e8e73eeb6fe4e985e82529e"}]}