)]}'
{"specs/mitaka/approved/live-migration-support-xbzrle-compress.rst":[{"author":{"_account_id":12175,"name":"Eli Qiao","email":"qiaoliyong@gmail.com","username":"Eli"},"change_message_id":"f6f43af08bc56b3c4cfca8ffd0c76f5992046cce","unresolved":false,"context_lines":[{"line_number":6,"context_line":""},{"line_number":7,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":8,"context_line":"support XBZRLE compress for live migration"},{"line_number":9,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":10,"context_line":""},{"line_number":11,"context_line":"Blueprint:"},{"line_number":12,"context_line":"https://blueprints.launchpad.net/nova/+spec/live-migrate-support-xbzrle-compress"}],"source_content_type":"text/x-rst","patch_set":1,"id":"ba8a016a_79eb42b1","line":9,"updated":"2015-11-23 01:42:59.000000000","message":"can you clarify in what version of libvirt support XBZRLE? I mean maybe we need to checking libvirt version before we set XBZRLE parameter, in some older version of libvirt, it may not support compress when doing live-migration. \n\neven libvirt supports, should be check qemu too?","commit_id":"f4aae77a288cd1efd12b3f836337fc506953471e"},{"author":{"_account_id":11189,"name":"yuntongjin","email":"yuntong.jin@intel.com","username":"yuntong"},"change_message_id":"3f44e4e1071a15d4fa9c59e935f11fc6b575ef9c","unresolved":false,"context_lines":[{"line_number":50,"context_line":"Proposed change"},{"line_number":51,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":52,"context_line":""},{"line_number":53,"context_line":"A new option compress will be added in migrate_server API. And it will be"},{"line_number":54,"context_line":"passed to libvirt/driver _live_migration_operation, then libvirt will enable"},{"line_number":55,"context_line":"the auto-converge capability of hypervisor."},{"line_number":56,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"ba8a016a_f4bad709","line":53,"updated":"2015-11-23 05:49:44.000000000","message":"A new compress option","commit_id":"f4aae77a288cd1efd12b3f836337fc506953471e"},{"author":{"_account_id":12175,"name":"Eli Qiao","email":"qiaoliyong@gmail.com","username":"Eli"},"change_message_id":"0b81174462bbb569f4e0623cec6c63f807cc15ec","unresolved":false,"context_lines":[{"line_number":50,"context_line":"Proposed change"},{"line_number":51,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":52,"context_line":""},{"line_number":53,"context_line":"A new option compress will be added in migrate_server API. And it will be"},{"line_number":54,"context_line":"passed to libvirt/driver _live_migration_operation, then libvirt will enable"},{"line_number":55,"context_line":"the auto-converge capability of hypervisor."},{"line_number":56,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"ba8a016a_68fe0bb0","line":53,"in_reply_to":"ba8a016a_f4bad709","updated":"2015-11-23 06:57:32.000000000","message":"Done","commit_id":"f4aae77a288cd1efd12b3f836337fc506953471e"},{"author":{"_account_id":11189,"name":"yuntongjin","email":"yuntong.jin@intel.com","username":"yuntong"},"change_message_id":"3f44e4e1071a15d4fa9c59e935f11fc6b575ef9c","unresolved":false,"context_lines":[{"line_number":51,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":52,"context_line":""},{"line_number":53,"context_line":"A new option compress will be added in migrate_server API. And it will be"},{"line_number":54,"context_line":"passed to libvirt/driver _live_migration_operation, then libvirt will enable"},{"line_number":55,"context_line":"the auto-converge capability of hypervisor."},{"line_number":56,"context_line":""},{"line_number":57,"context_line":"Now the compress option just support XBZRLE, we will support other compression"}],"source_content_type":"text/x-rst","patch_set":1,"id":"ba8a016a_aad70a22","line":54,"updated":"2015-11-23 05:49:44.000000000","message":"specifically, it will pass VIR_MIGRATE_COMPRESSED flag to libvirt migrate API.","commit_id":"f4aae77a288cd1efd12b3f836337fc506953471e"},{"author":{"_account_id":11189,"name":"yuntongjin","email":"yuntong.jin@intel.com","username":"yuntong"},"change_message_id":"3f44e4e1071a15d4fa9c59e935f11fc6b575ef9c","unresolved":false,"context_lines":[{"line_number":52,"context_line":""},{"line_number":53,"context_line":"A new option compress will be added in migrate_server API. And it will be"},{"line_number":54,"context_line":"passed to libvirt/driver _live_migration_operation, then libvirt will enable"},{"line_number":55,"context_line":"the auto-converge capability of hypervisor."},{"line_number":56,"context_line":""},{"line_number":57,"context_line":"Now the compress option just support XBZRLE, we will support other compression"},{"line_number":58,"context_line":"methods, such as multi-thread compress."}],"source_content_type":"text/x-rst","patch_set":1,"id":"ba8a016a_34485f2d","line":55,"updated":"2015-11-23 05:49:44.000000000","message":"how is this related to auto-converge?","commit_id":"f4aae77a288cd1efd12b3f836337fc506953471e"},{"author":{"_account_id":12175,"name":"Eli Qiao","email":"qiaoliyong@gmail.com","username":"Eli"},"change_message_id":"0b81174462bbb569f4e0623cec6c63f807cc15ec","unresolved":false,"context_lines":[{"line_number":52,"context_line":""},{"line_number":53,"context_line":"A new option compress will be added in migrate_server API. And it will be"},{"line_number":54,"context_line":"passed to libvirt/driver _live_migration_operation, then libvirt will enable"},{"line_number":55,"context_line":"the auto-converge capability of hypervisor."},{"line_number":56,"context_line":""},{"line_number":57,"context_line":"Now the compress option just support XBZRLE, we will support other compression"},{"line_number":58,"context_line":"methods, such as multi-thread compress."}],"source_content_type":"text/x-rst","patch_set":1,"id":"ba8a016a_085e5fe9","line":55,"in_reply_to":"ba8a016a_34485f2d","updated":"2015-11-23 06:57:32.000000000","message":"Done","commit_id":"f4aae77a288cd1efd12b3f836337fc506953471e"},{"author":{"_account_id":11189,"name":"yuntongjin","email":"yuntong.jin@intel.com","username":"yuntong"},"change_message_id":"3f44e4e1071a15d4fa9c59e935f11fc6b575ef9c","unresolved":false,"context_lines":[{"line_number":55,"context_line":"the auto-converge capability of hypervisor."},{"line_number":56,"context_line":""},{"line_number":57,"context_line":"Now the compress option just support XBZRLE, we will support other compression"},{"line_number":58,"context_line":"methods, such as multi-thread compress."},{"line_number":59,"context_line":""},{"line_number":60,"context_line":"XBZRLE need to retrieve the old page content from an LRU cache (default size"},{"line_number":61,"context_line":"of 64 MB). So the cache size is a key performance for XBZRLE. Two APIs,"}],"source_content_type":"text/x-rst","patch_set":1,"id":"ba8a016a_d45473cb","line":58,"updated":"2015-11-23 05:49:44.000000000","message":"Currently in libvirt\u0027s implement, when pass VIR_MIGRATE_COMPRESSED flag to libvirt migrate API, it will use XBZRLE, so, no api exposed yet to support other compress methods.","commit_id":"f4aae77a288cd1efd12b3f836337fc506953471e"},{"author":{"_account_id":12175,"name":"Eli Qiao","email":"qiaoliyong@gmail.com","username":"Eli"},"change_message_id":"0b81174462bbb569f4e0623cec6c63f807cc15ec","unresolved":false,"context_lines":[{"line_number":55,"context_line":"the auto-converge capability of hypervisor."},{"line_number":56,"context_line":""},{"line_number":57,"context_line":"Now the compress option just support XBZRLE, we will support other compression"},{"line_number":58,"context_line":"methods, such as multi-thread compress."},{"line_number":59,"context_line":""},{"line_number":60,"context_line":"XBZRLE need to retrieve the old page content from an LRU cache (default size"},{"line_number":61,"context_line":"of 64 MB). So the cache size is a key performance for XBZRLE. Two APIs,"}],"source_content_type":"text/x-rst","patch_set":1,"id":"ba8a016a_48778702","line":58,"in_reply_to":"ba8a016a_d45473cb","updated":"2015-11-23 06:57:32.000000000","message":"sure, shaohe and me are working on adding new methods in libvirt.","commit_id":"f4aae77a288cd1efd12b3f836337fc506953471e"},{"author":{"_account_id":11189,"name":"yuntongjin","email":"yuntong.jin@intel.com","username":"yuntong"},"change_message_id":"3f44e4e1071a15d4fa9c59e935f11fc6b575ef9c","unresolved":false,"context_lines":[{"line_number":59,"context_line":""},{"line_number":60,"context_line":"XBZRLE need to retrieve the old page content from an LRU cache (default size"},{"line_number":61,"context_line":"of 64 MB). So the cache size is a key performance for XBZRLE. Two APIs,"},{"line_number":62,"context_line":"set_live_migrate_parameters and get_live_migrate_parameters."},{"line_number":63,"context_line":""},{"line_number":64,"context_line":"These two APIs should be extensible to support parameters of other method"},{"line_number":65,"context_line":"compressions."}],"source_content_type":"text/x-rst","patch_set":1,"id":"ba8a016a_745ca7ed","line":62,"updated":"2015-11-23 05:49:44.000000000","message":"can use these two APIs","commit_id":"f4aae77a288cd1efd12b3f836337fc506953471e"},{"author":{"_account_id":14131,"name":"shaohef","email":"shaohe.feng@intel.com","username":"shaohefeng"},"change_message_id":"cbcf00404e3e1492a2812876730246272cb836cc","unresolved":false,"context_lines":[{"line_number":69,"context_line":"less downtime and duration time."},{"line_number":70,"context_line":""},{"line_number":71,"context_line":"In order to monitor the xbzrle cache statistics, it need to extend"},{"line_number":72,"context_line":"os-migrations to teturn a xbzrle cache field."},{"line_number":73,"context_line":""},{"line_number":74,"context_line":"Alternatives"},{"line_number":75,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"ba8a016a_855c7fee","line":72,"updated":"2015-11-22 15:16:28.000000000","message":"maybe need another API to get the  xbzrle cache statistics.","commit_id":"f4aae77a288cd1efd12b3f836337fc506953471e"},{"author":{"_account_id":14131,"name":"shaohef","email":"shaohe.feng@intel.com","username":"shaohefeng"},"change_message_id":"cbcf00404e3e1492a2812876730246272cb836cc","unresolved":false,"context_lines":[{"line_number":124,"context_line":"      },"},{"line_number":125,"context_line":"  }"},{"line_number":126,"context_line":""},{"line_number":127,"context_line":"Get Parameters Body::"},{"line_number":128,"context_line":""},{"line_number":129,"context_line":"  {"},{"line_number":130,"context_line":"    \"get_live_migrate_parameters\":"}],"source_content_type":"text/x-rst","patch_set":1,"id":"ba8a016a_85131f0d","line":127,"updated":"2015-11-22 15:16:28.000000000","message":"This should be a \"GET\" method.\n Maybe we do not need \"Get Parameters.\"\n \n Set Parameters Body:: \n\n  {\t\n    \"set_live_migrate_parameters\":\t\n      \u0027xbzrle\u0027: {\t\n        \u0027cache-size\u0027: None,\t\n      },\t\n  }\n set  \u0027cache-size\u0027 is None to get the current cache-size.","commit_id":"f4aae77a288cd1efd12b3f836337fc506953471e"},{"author":{"_account_id":11189,"name":"yuntongjin","email":"yuntong.jin@intel.com","username":"yuntong"},"change_message_id":"3f44e4e1071a15d4fa9c59e935f11fc6b575ef9c","unresolved":false,"context_lines":[{"line_number":233,"context_line":""},{"line_number":234,"context_line":"* A new option compress is added in migrate_server API."},{"line_number":235,"context_line":"* Two extended API set_live_migrate_parameters and get_live_migrate_parameters"},{"line_number":236,"context_line":"  are added."},{"line_number":237,"context_line":"* extend os-migrations to get xbzrle-cache statistics."},{"line_number":238,"context_line":""},{"line_number":239,"context_line":"Dependencies"}],"source_content_type":"text/x-rst","patch_set":1,"id":"ba8a016a_34cb7f74","line":236,"updated":"2015-11-23 05:49:44.000000000","message":"I don\u0027t think we need this.","commit_id":"f4aae77a288cd1efd12b3f836337fc506953471e"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"a7475c3820f5cc3a0c29a283c6131987b1bf0253","unresolved":false,"context_lines":[{"line_number":52,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":53,"context_line":""},{"line_number":54,"context_line":"A new compress option will be added in os-migrateLive API. The list indicates"},{"line_number":55,"context_line":"compress method when doing live migration."},{"line_number":56,"context_line":""},{"line_number":57,"context_line":"Now the compress option only supports XBZRLE, we will support other compression"},{"line_number":58,"context_line":"methods, such as multi-thread compress."}],"source_content_type":"text/x-rst","patch_set":2,"id":"ba8a016a_4550c50a","line":55,"updated":"2015-11-23 10:47:25.000000000","message":"I\u0027m absolutely against exposing this kind of control over the operation of live migration in the API. The availability \u0026 use of compression is a low level QEMU specific implementation detail that has no business being exposed in the public Nova API. If we want to make use of compression, this is something that Nova itself should decide to enable/disable, not the API user.","commit_id":"371910e52e825a29a2c16f4d0bd36918a3ee8ec9"},{"author":{"_account_id":14131,"name":"shaohef","email":"shaohe.feng@intel.com","username":"shaohefeng"},"change_message_id":"9a95186bf8695dee0b46bbc94c7bf82eb4267464","unresolved":false,"context_lines":[{"line_number":52,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":53,"context_line":""},{"line_number":54,"context_line":"A new compress option will be added in os-migrateLive API. The list indicates"},{"line_number":55,"context_line":"compress method when doing live migration."},{"line_number":56,"context_line":""},{"line_number":57,"context_line":"Now the compress option only supports XBZRLE, we will support other compression"},{"line_number":58,"context_line":"methods, such as multi-thread compress."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9a8ffd7b_bbac65ee","line":55,"in_reply_to":"ba8a016a_4550c50a","updated":"2015-11-24 16:32:33.000000000","message":"ACK.\n and today\u0027s sub-team meeting, we agree on: \n both enable xbzrle and set  xbzrle-cache size in nova.conf. \n\n as PaulMurray suggested, we could start stupid and then improve.\n\n Daniel, do you agree?","commit_id":"371910e52e825a29a2c16f4d0bd36918a3ee8ec9"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"c8b0d7001be037ec212477d062a0827e0ec1b4b9","unresolved":false,"context_lines":[{"line_number":23,"context_line":"By using XBZRLE compress we can reduce VM downtime and total live-migration"},{"line_number":24,"context_line":"time of VMs running memory write intensive workloads typical of large enterprise"},{"line_number":25,"context_line":"applications such as SAP ERP Systems, and generally speaking for any"},{"line_number":26,"context_line":"application with a sparse memory update pattern."},{"line_number":27,"context_line":""},{"line_number":28,"context_line":"XBZRLE need to retrieve the old page content from an LRU cache (default size"},{"line_number":29,"context_line":"of 64 MB). So the cache size is a performance key for XBZRLE compress."}],"source_content_type":"text/x-rst","patch_set":3,"id":"9a8ffd7b_15de796d","line":26,"updated":"2015-11-25 10:07:34.000000000","message":"Please mention that this is at the cost of significantly increased CPU overhead for migration. QEMU migration maintainers only recommend its use when the network link is slow enough that migration would not otherwise succeed.","commit_id":"72a19d2598f23fa7e905955331e70f57561ba85b"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"c8b0d7001be037ec212477d062a0827e0ec1b4b9","unresolved":false,"context_lines":[{"line_number":26,"context_line":"application with a sparse memory update pattern."},{"line_number":27,"context_line":""},{"line_number":28,"context_line":"XBZRLE need to retrieve the old page content from an LRU cache (default size"},{"line_number":29,"context_line":"of 64 MB). So the cache size is a performance key for XBZRLE compress."},{"line_number":30,"context_line":""},{"line_number":31,"context_line":"Use Cases"},{"line_number":32,"context_line":"----------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"9a8ffd7b_753a6538","line":29,"updated":"2015-11-25 10:07:34.000000000","message":"To expand on this, the choice of cache size is critical for the effectiveness of compression. For XBZRLE to be effective, the cache size has to match the RAM working set size of the guest workload. In the worst case the cache size has to be potentially the same as the total guest RAM allocation.   IOW a  2 GB guest potentially has to have 2 GB of cache, in order to get a good cache hit rate.","commit_id":"72a19d2598f23fa7e905955331e70f57561ba85b"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"c8b0d7001be037ec212477d062a0827e0ec1b4b9","unresolved":false,"context_lines":[{"line_number":39,"context_line":""},{"line_number":40,"context_line":"Add a option for XBZRLE compress cache size in nova.conf, under libvirt section."},{"line_number":41,"context_line":"when the VIR_MIGRATE_COMPRESSED flag is set in live_migration_flag, libvirt driver"},{"line_number":42,"context_line":"will set the compress cache according to the option."},{"line_number":43,"context_line":""},{"line_number":44,"context_line":""},{"line_number":45,"context_line":"Alternatives"}],"source_content_type":"text/x-rst","patch_set":3,"id":"9a8ffd7b_702593a9","line":42,"updated":"2015-11-25 10:07:34.000000000","message":"Based on my understanding of XBZRLE compression CPU \u0026 memory overheads, combined with its cache size tuning requirements, I don\u0027t think that providing static configuration in nova.conf is a satisfactory solution.\n\nIn particular the cache size is definitely going to need to vary according to the guest RAM size, so a one-size-fits-all setting for that is just wrong. It will be too small to be useful for some guests, and too large and waste memory for other guests.\n\nIn addition, XBZRLE is not something you want to unconditionally enable for all guests due to its very significant CPU and memory overheads while in use. It really only wants to be used if the network is not able to keep up with the dirtying rate of the guest RAM.\n\nThis all says to me that, if we want to support compression, we need to dynamically choose when to activate it and how to configure the cache size.","commit_id":"72a19d2598f23fa7e905955331e70f57561ba85b"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"f99171034c9f7fc3e4337b173e44c148aa621c4c","unresolved":false,"context_lines":[{"line_number":39,"context_line":""},{"line_number":40,"context_line":"Add a option for XBZRLE compress cache size in nova.conf, under libvirt section."},{"line_number":41,"context_line":"when the VIR_MIGRATE_COMPRESSED flag is set in live_migration_flag, libvirt driver"},{"line_number":42,"context_line":"will set the compress cache according to the option."},{"line_number":43,"context_line":""},{"line_number":44,"context_line":""},{"line_number":45,"context_line":"Alternatives"}],"source_content_type":"text/x-rst","patch_set":3,"id":"9a8ffd7b_42d404bc","line":42,"in_reply_to":"9a8ffd7b_649cc2c0","updated":"2015-11-26 10:12:28.000000000","message":"That benchmark result is pretty meaningless in the context of openstack. The test program illustrated there allocates a single 16 MB buffer and just dirties that forever. That is not in any way representative of a real world workload where the working set memory is likely to be *MUCH* larger than 16 MB in size.","commit_id":"72a19d2598f23fa7e905955331e70f57561ba85b"},{"author":{"_account_id":15447,"name":"yongfeng","email":"dolpher.du@intel.com","username":"dolpher"},"change_message_id":"9a4be9672cd9d7c914a43ca4c22d128520292116","unresolved":false,"context_lines":[{"line_number":39,"context_line":""},{"line_number":40,"context_line":"Add a option for XBZRLE compress cache size in nova.conf, under libvirt section."},{"line_number":41,"context_line":"when the VIR_MIGRATE_COMPRESSED flag is set in live_migration_flag, libvirt driver"},{"line_number":42,"context_line":"will set the compress cache according to the option."},{"line_number":43,"context_line":""},{"line_number":44,"context_line":""},{"line_number":45,"context_line":"Alternatives"}],"source_content_type":"text/x-rst","patch_set":3,"id":"9a8ffd7b_7278fb8f","line":42,"in_reply_to":"9a8ffd7b_702593a9","updated":"2015-11-26 08:29:05.000000000","message":"Daniel, I\u0027m not sure when the LM is going to be used in production, so  I don\u0027t know whether the below assumption is valid.\n\nAssume such condition, a physical server that can hold 50 VMs, and 3 of the VMs are in the process of Live migration, and we allocate the cache size as large as the VM instances\u0027 memory.\n\nIn this case, the overhead of XBZRLE compression is 3 extra VMs in a short period of time(hours), and it\u0027s not a big burden compared to the capability of the physical server.(If there is no extra resources, just fail the XBZRLE and back to not compressed behavior)\n\nIf so, I think, even with a static configuration, this feature will still benefit the live migration under some circumstances?","commit_id":"72a19d2598f23fa7e905955331e70f57561ba85b"},{"author":{"_account_id":14131,"name":"shaohef","email":"shaohe.feng@intel.com","username":"shaohefeng"},"change_message_id":"d89d927f8dd1b80f4e4c73922b11ed366c4a7000","unresolved":false,"context_lines":[{"line_number":39,"context_line":""},{"line_number":40,"context_line":"Add a option for XBZRLE compress cache size in nova.conf, under libvirt section."},{"line_number":41,"context_line":"when the VIR_MIGRATE_COMPRESSED flag is set in live_migration_flag, libvirt driver"},{"line_number":42,"context_line":"will set the compress cache according to the option."},{"line_number":43,"context_line":""},{"line_number":44,"context_line":""},{"line_number":45,"context_line":"Alternatives"}],"source_content_type":"text/x-rst","patch_set":3,"id":"9a8ffd7b_649cc2c0","line":42,"in_reply_to":"9a8ffd7b_702593a9","updated":"2015-11-26 04:59:49.000000000","message":"about the  xbzrle cache missing rate. \n ChenLiang \u003cchenliang88@huawei.com\u003e test:\n http://git.greensocs.com/fkonrad/mttcg/commit/27af7d6ea5015e5ef1f7985eab94a8a218267a2b\n 1G RAM with XBZRLE size 8M, missing rate is 48.59.","commit_id":"72a19d2598f23fa7e905955331e70f57561ba85b"},{"author":{"_account_id":12299,"name":"Paweł Koniszewski","email":"pawel.koniszewski@gmail.com","username":"pawel.koniszewski"},"change_message_id":"50c25247b3af1397a043d1a34303167d39ce3daf","unresolved":false,"context_lines":[{"line_number":39,"context_line":""},{"line_number":40,"context_line":"Add a option for XBZRLE compress cache size in nova.conf, under libvirt section."},{"line_number":41,"context_line":"when the VIR_MIGRATE_COMPRESSED flag is set in live_migration_flag, libvirt driver"},{"line_number":42,"context_line":"will set the compress cache according to the option."},{"line_number":43,"context_line":""},{"line_number":44,"context_line":""},{"line_number":45,"context_line":"Alternatives"}],"source_content_type":"text/x-rst","patch_set":3,"id":"9a8ffd7b_ed92ce9e","line":42,"in_reply_to":"9a8ffd7b_7278fb8f","updated":"2015-11-26 08:54:44.000000000","message":"According to simple maths you should never live migrate more than 1 VM at a time to maximize success chance and to minimize total time of live migrations.","commit_id":"72a19d2598f23fa7e905955331e70f57561ba85b"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"f99171034c9f7fc3e4337b173e44c148aa621c4c","unresolved":false,"context_lines":[{"line_number":39,"context_line":""},{"line_number":40,"context_line":"Add a option for XBZRLE compress cache size in nova.conf, under libvirt section."},{"line_number":41,"context_line":"when the VIR_MIGRATE_COMPRESSED flag is set in live_migration_flag, libvirt driver"},{"line_number":42,"context_line":"will set the compress cache according to the option."},{"line_number":43,"context_line":""},{"line_number":44,"context_line":""},{"line_number":45,"context_line":"Alternatives"}],"source_content_type":"text/x-rst","patch_set":3,"id":"9a8ffd7b_8238acf0","line":42,"in_reply_to":"9a8ffd7b_7278fb8f","updated":"2015-11-26 10:12:28.000000000","message":"The memory overhead of the compression cache is certainly enough to push the server into an out-of-memory scenario, which is a big deal.  As a static configuration of cache size is essentially useless as described earlier.","commit_id":"72a19d2598f23fa7e905955331e70f57561ba85b"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"c8b0d7001be037ec212477d062a0827e0ec1b4b9","unresolved":false,"context_lines":[{"line_number":77,"context_line":"Performance Impact"},{"line_number":78,"context_line":"------------------"},{"line_number":79,"context_line":""},{"line_number":80,"context_line":"None"},{"line_number":81,"context_line":""},{"line_number":82,"context_line":"Other deployer impact"},{"line_number":83,"context_line":"---------------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"9a8ffd7b_b52bed81","line":80,"updated":"2015-11-25 10:07:34.000000000","message":"The use of compression for migration has a very significant CPU performance overhead. It also has a very significant RAM overhead, potentially doubling the RAM needs of a guest.  The RAM requirement in particular is a big concern.  The scheduler is not accounting for the fact that migration will require a non-negligble increase in RAM usage, so by triggering migration with XZBRLE compression enable d we are potentially going to push the compute host into an out-of-memory scenario. I think we really need to consider this impact.","commit_id":"72a19d2598f23fa7e905955331e70f57561ba85b"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"c8b0d7001be037ec212477d062a0827e0ec1b4b9","unresolved":false,"context_lines":[{"line_number":82,"context_line":"Other deployer impact"},{"line_number":83,"context_line":"---------------------"},{"line_number":84,"context_line":""},{"line_number":85,"context_line":"None"},{"line_number":86,"context_line":""},{"line_number":87,"context_line":"Developer impact"},{"line_number":88,"context_line":"----------------"}],"source_content_type":"text/x-rst","patch_set":3,"id":"9a8ffd7b_156999be","line":85,"updated":"2015-11-25 10:07:34.000000000","message":"...errr this feature is entirely aimed at deployers","commit_id":"72a19d2598f23fa7e905955331e70f57561ba85b"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"d24addbb3cfe030af9b9c93a7d157c06701aecd4","unresolved":false,"context_lines":[{"line_number":56,"context_line":"Add a option for XBZRLE compress cache size in nova.conf, under libvirt section."},{"line_number":57,"context_line":"It is a default value."},{"line_number":58,"context_line":""},{"line_number":59,"context_line":"Just providing static configuration in nova.conf is a satisfactory solution,"},{"line_number":60,"context_line":"based the understanding of XBZRLE compression CPU \u0026 memory overheads,"},{"line_number":61,"context_line":"combined with its cache size tuning requirements."},{"line_number":62,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"9a8ffd7b_251eaa5f","line":59,"updated":"2015-11-26 10:31:36.000000000","message":"s/is/is not/","commit_id":"445a58e6ba1b3462eb803d9433a32f3a207a9c60"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"d24addbb3cfe030af9b9c93a7d157c06701aecd4","unresolved":false,"context_lines":[{"line_number":75,"context_line":""},{"line_number":76,"context_line":"1. XBZRLE enable."},{"line_number":77,"context_line":""},{"line_number":78,"context_line":"   Firstly, the VIR_MIGRATE_COMPRESSED flag must be set in live_migration_flag."},{"line_number":79,"context_line":"   There\u0027s the propose:"},{"line_number":80,"context_line":""},{"line_number":81,"context_line":"   a. experience learning"}],"source_content_type":"text/x-rst","patch_set":5,"id":"9a8ffd7b_c51b1ec9","line":78,"updated":"2015-11-26 10:31:36.000000000","message":"The existence of the live_migration_flag parameter is one of the stupider decisions in Nova\u0027s past, because it assumes all migrations want to use the same set of flags, which is wrong. We already see example of the wrongness by the need to add a separate block_migration_flag parameter. As such we shouldn\u0027t encourage further use of.\n\nWe should have a dedicated  \u0027live_migration_compression\u003don|off\" parameter if we want admins to request compression.","commit_id":"445a58e6ba1b3462eb803d9433a32f3a207a9c60"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"d24addbb3cfe030af9b9c93a7d157c06701aecd4","unresolved":false,"context_lines":[{"line_number":81,"context_line":"   a. experience learning"},{"line_number":82,"context_line":"      If DB no migrations record on this hypervisor, then the XBZRLE will not"},{"line_number":83,"context_line":"      be enable for this migration. And DB will record this migration dirty"},{"line_number":84,"context_line":"      info (transfered data, total data)."},{"line_number":85,"context_line":"   #. migration bandwidth"},{"line_number":86,"context_line":"      bellow a certain threshold value, then enable XBZRLE."},{"line_number":87,"context_line":"   #. hypervisor environment"}],"source_content_type":"text/x-rst","patch_set":5,"id":"9a8ffd7b_65dfd20d","line":84,"updated":"2015-11-26 10:31:36.000000000","message":"I\u0027m not really seeing what this is doing. Previous migration operations are not really an accurate guide to the future, for a number of reasons. The guest workload in a previous migration can be totally different to the guest workload in new migration. The network contention at the time could be totally different. The target host may not be the same ether, which can impact on network bandwidth rates.","commit_id":"445a58e6ba1b3462eb803d9433a32f3a207a9c60"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"d24addbb3cfe030af9b9c93a7d157c06701aecd4","unresolved":false,"context_lines":[{"line_number":83,"context_line":"      be enable for this migration. And DB will record this migration dirty"},{"line_number":84,"context_line":"      info (transfered data, total data)."},{"line_number":85,"context_line":"   #. migration bandwidth"},{"line_number":86,"context_line":"      bellow a certain threshold value, then enable XBZRLE."},{"line_number":87,"context_line":"   #. hypervisor environment"},{"line_number":88,"context_line":"      CPU is not activity and not high RAM usage."},{"line_number":89,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"9a8ffd7b_28567fce","line":86,"updated":"2015-11-26 10:31:36.000000000","message":"To measure bandwidth, the migration needs to be already running. You can\u0027t enable compression though, once migration is running, so you have a chicken \u0026 egg problem here.","commit_id":"445a58e6ba1b3462eb803d9433a32f3a207a9c60"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"d24addbb3cfe030af9b9c93a7d157c06701aecd4","unresolved":false,"context_lines":[{"line_number":86,"context_line":"      bellow a certain threshold value, then enable XBZRLE."},{"line_number":87,"context_line":"   #. hypervisor environment"},{"line_number":88,"context_line":"      CPU is not activity and not high RAM usage."},{"line_number":89,"context_line":""},{"line_number":90,"context_line":"#. configure the cache size"},{"line_number":91,"context_line":""},{"line_number":92,"context_line":"   a. If no cache size is setting, use hypervisor default value. Or use the"}],"source_content_type":"text/x-rst","patch_set":5,"id":"9a8ffd7b_08ddbb6e","line":89,"updated":"2015-11-26 10:31:36.000000000","message":"This is inherently racy because the schedular has no knowledge of the fact that we\u0027re using a bunch of RAM for migration cache. So any time while the migration is running, the schedular might boot another VM on the host pushing us into an out-of-memory scenario as a result.","commit_id":"445a58e6ba1b3462eb803d9433a32f3a207a9c60"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"d24addbb3cfe030af9b9c93a7d157c06701aecd4","unresolved":false,"context_lines":[{"line_number":90,"context_line":"#. configure the cache size"},{"line_number":91,"context_line":""},{"line_number":92,"context_line":"   a. If no cache size is setting, use hypervisor default value. Or use the"},{"line_number":93,"context_line":"      setting value."},{"line_number":94,"context_line":"   #. XBZRLE compress are executed after bulk stage, during the bulk stage,"},{"line_number":95,"context_line":"      calculate an average transfer rate. If the transfer rate bellow a certain"},{"line_number":96,"context_line":"      threshold value, then increase cache size."}],"source_content_type":"text/x-rst","patch_set":5,"id":"9a8ffd7b_08205b47","line":93,"updated":"2015-11-26 10:31:36.000000000","message":"Which setting are you referring to here ?","commit_id":"445a58e6ba1b3462eb803d9433a32f3a207a9c60"},{"author":{"_account_id":782,"name":"John Garbutt","email":"john@johngarbutt.com","username":"johngarbutt"},"change_message_id":"73b6657d5ea635e37d39f1484f42af0b40f52a00","unresolved":false,"context_lines":[{"line_number":96,"context_line":"      threshold value, then increase cache size."},{"line_number":97,"context_line":"   #. Monitor the migration status, increase cache size with dirty page rapidly"},{"line_number":98,"context_line":"      increase. The (transfered + remaining - total) can sign memory write"},{"line_number":99,"context_line":"      intensive."},{"line_number":100,"context_line":""},{"line_number":101,"context_line":""},{"line_number":102,"context_line":"Alternatives"}],"source_content_type":"text/x-rst","patch_set":5,"id":"9a8ffd7b_498c0eda","line":99,"updated":"2015-11-26 17:15:29.000000000","message":"So I really don\u0027t want an API to enable compression for live-migrate. That seems horrible.\n\nI am hoping this is more to do with your network capacity that anything else? If you want to take the CPU load hit, because you have a small amount of bandwidth, then turn this on for all instances.\n\nWhile it *could* be workload specific, I would love to avoid the need for that API if at all possible.\n\nAgreed we are unlikely to be able to \"detect\" when its best to use this. Seems like a trade off that needs to be make.","commit_id":"445a58e6ba1b3462eb803d9433a32f3a207a9c60"},{"author":{"_account_id":1779,"name":"Daniel Berrange","email":"berrange@redhat.com","username":"berrange"},"change_message_id":"802cf039b6f00a898e7c59a3be817ec4f7ebf523","unresolved":false,"context_lines":[{"line_number":96,"context_line":"      threshold value, then increase cache size."},{"line_number":97,"context_line":"   #. Monitor the migration status, increase cache size with dirty page rapidly"},{"line_number":98,"context_line":"      increase. The (transfered + remaining - total) can sign memory write"},{"line_number":99,"context_line":"      intensive."},{"line_number":100,"context_line":""},{"line_number":101,"context_line":""},{"line_number":102,"context_line":"Alternatives"}],"source_content_type":"text/x-rst","patch_set":5,"id":"9a8ffd7b_249bf173","line":99,"in_reply_to":"9a8ffd7b_498c0eda","updated":"2015-11-26 17:26:15.000000000","message":"I totally agree about not exposing compression related stuff in the API - I don\u0027t think this was actually suggesting that though.\n\nWhile it has significant CPU overhead, that\u0027s not the end of the world because the CPU tuning settings applied to QEMU will also control the CPU used by migration. So if the guest has CPU SLAs set, compression won\u0027t violate them.\n\nThe real big problem is the memory overhead of compression, due to the need for the compression cache to be equivalent in size to the guest OS memory working set. IOW, potentially doubles the amount of RAM the QEMU process will use, if we want the compression to get a decent enough hit rate to make it useful. This makes it something we can\u0027t just turn on via a static config parameter IMHO.","commit_id":"445a58e6ba1b3462eb803d9433a32f3a207a9c60"}]}