)]}'
{"specs/queens/approved/compute-node-ha-strategy.rst":[{"author":{"_account_id":2394,"name":"Adam Spiers","email":"aspiers@suse.com","username":"adam.spiers"},"change_message_id":"edb07111189b69ad833879bd626e4289db5560ee","unresolved":false,"context_lines":[{"line_number":49,"context_line":"  and has been in \u0027down\u0027 state for more than the given parameter \"time\","},{"line_number":50,"context_line":"  evacuate all the instances of the node."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"This strategy just consider how to evacuate all instances of the failure"},{"line_number":53,"context_line":"compute node, further optim rely on other strategies."},{"line_number":54,"context_line":""},{"line_number":55,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"9f436f4f_6ed8d37e","line":52,"updated":"2017-08-18 11:36:37.000000000","message":"Sorry, but you are at great risk of reinventing wheels here. The OpenStack HA community has been working on compute node HA for several years now, so please can you collaborate with us rather than implementing another independent solution?\n\nPlease see here for (a lot) more information:\n\nhttp://specs.openstack.org/openstack/openstack-user-stories/user-stories/proposed/ha_vm.html\nhttps://aspiers.github.io/openstack-day-israel-2017-compute-ha/ \nhttps://github.com/openstack/openstack-resource-agents-specs/tree/master/specs/newton/approved\n\nThis raises a bigger question: what is the scope of the Watcher ?\n\nAccording to the mission statement at https://wiki.openstack.org/wiki/Watcher, Watcher is supposed to be an optimization service.  (This is corroborated by https://launchpad.net/watcher.)  But IMHO automatic recovery from failures is a separate thing to optimization: it is high availability.  High availability is for automated recovery from failures, whereas optimization is for improving performance when there are no failures.\n\nTherefore I would suggest that in general any HA topics should remain out of scope of Watcher, otherwise there will be duplicated effort between Watcher and other HA initiatives within OpenStack.  If there are areas where it makes sense for Watcher to be aware of HA considerations, then let\u0027s work together to build a coherent HA architecture across all OpenStack projects.  I\u0027d very happy to discuss this more - you can find me as @aspiers on #openstack-ha.  Thanks!","commit_id":"ad913b2ac7d37a790a2b52a5abcb1269d27e147e"},{"author":{"_account_id":2394,"name":"Adam Spiers","email":"aspiers@suse.com","username":"adam.spiers"},"change_message_id":"f3cfe2d98ce73c0aa92361f5a48adbe34783eaf7","unresolved":false,"context_lines":[{"line_number":49,"context_line":"  and has been in \u0027down\u0027 state for more than the given parameter \"time\","},{"line_number":50,"context_line":"  evacuate all the instances of the node."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"This strategy just consider how to evacuate all instances of the failure"},{"line_number":53,"context_line":"compute node, further optim rely on other strategies."},{"line_number":54,"context_line":""},{"line_number":55,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"9f436f4f_6cd5401b","line":52,"in_reply_to":"9f436f4f_616fa2fc","updated":"2017-08-21 21:42:52.000000000","message":"Actually detecting the failure is a solved problem, and is described here:\n\nhttps://github.com/openstack/openstack-resource-agents-specs/blob/master/specs/newton/approved/newton-instance-ha-host-monitoring-spec.rst\n\nHandling the failure is the harder bit.  But I strongly believe that this does not belong in Watcher.  Handling failures is not optimisation, and is already handled by a combination of Pacemaker and Masakari.  If you implement in Watcher then you are just duplicating that architecture.\n\n\u003e In my opinion, to evacuate instances away from failure resources\n\nBut that is not evacuation, it is resurrection.  After failure the VMs are dead, so they cannot be migrated.  Please understand that \"nova-evacuate\" is a misnomer.  nova-evacuate is not evacuation.  The nova developers chose the wrong word, and they even admit this mistake.  This is described in the Youtube talk I sent you, but also here:\n\nhttp://www.danplanet.com/blog/2016/03/03/evacuate-in-nova-one-command-to-confuse-us-all/\n\n\u003e In my opinion, to evacuate instances away from failure resources could be regarded as part work of Watcher. It is optimization for particular scene.\n\nI\u0027m sorry but I really disagree.  Recovering from failure is not optimisation, it is HA.  They are different things.  Optimisation is for when there are not failures, but you are still making things better (\"more optimal\").\n\nAs I said before, I think there *is* a good reason to do real evacuation, i.e. live migration, of VMs in Watcher: it is possible to optimise energy consumption by consolidating VMs onto fewer compute nodes (VM consolidation) and then reducing the number of compute node hosts which are powered on. But this is a totally different scenario to compute node HA, because there are no failures to recover from.  I wrote more about this and other optimisation use cases here: https://blog.adamspiers.org/2015/05/17/cloud-rearrangement/","commit_id":"ad913b2ac7d37a790a2b52a5abcb1269d27e147e"},{"author":{"_account_id":24501,"name":"sue","display_name":"suzhengwei","email":"sugar-2008@163.com","username":"suzhengwei"},"change_message_id":"9a25c37dec5b16445748928a61b084d0ea3edfd7","unresolved":false,"context_lines":[{"line_number":49,"context_line":"  and has been in \u0027down\u0027 state for more than the given parameter \"time\","},{"line_number":50,"context_line":"  evacuate all the instances of the node."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"This strategy just consider how to evacuate all instances of the failure"},{"line_number":53,"context_line":"compute node, further optim rely on other strategies."},{"line_number":54,"context_line":""},{"line_number":55,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"9f436f4f_616fa2fc","line":52,"in_reply_to":"9f436f4f_6ed8d37e","updated":"2017-08-21 06:08:42.000000000","message":"Thank you, and I understand your work. The most complicated and difficult part of the work is how to detect the failure in the openstack env.\nIn my application scenario, I depend on a third application, it will force-down nova-compute service or poweroff the compute node once it detect failure in the compute node. Then the administrator need to evacuate all instances away from the failure node.\nIn my opinion, to evacuate instances away from failure resources could be regarded as part work of Watcher. It is optimization for particular scene. This spec just evacuate instances away from nova-compute \u0027down\u0027 nodes. It does\u0027t know the specific failure reason. Maybe I should not name it in \u0027compute node ha\u0027.","commit_id":"ad913b2ac7d37a790a2b52a5abcb1269d27e147e"}],"specs/queens/approved/workload-evacuate-strategy.rst":[{"author":{"_account_id":2394,"name":"Adam Spiers","email":"aspiers@suse.com","username":"adam.spiers"},"change_message_id":"44a30c370db543e235a9132c8733eec286e75c59","unresolved":false,"context_lines":[{"line_number":14,"context_line":"Problem description"},{"line_number":15,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":16,"context_line":""},{"line_number":17,"context_line":"Sometimes if the failure has happened to one compute node, it would be"},{"line_number":18,"context_line":"marked as down. Then we need to evacuate the instances which are running"},{"line_number":19,"context_line":"on this failure compute node."},{"line_number":20,"context_line":""},{"line_number":21,"context_line":"It needs to do following things:"},{"line_number":22,"context_line":"* add an action to evacuate all the instances of the failure compute node"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9f436f4f_2cb858e3","line":19,"range":{"start_line":17,"start_character":0,"end_line":19,"end_character":29},"updated":"2017-08-21 22:01:09.000000000","message":"I agree that Watcher needs an evacuation (i.e. live migration) strategy, but as explained in my other comments, the problem described here is an HA problem which should not be solved by Watcher, because it is already solved by Pacemaker+Masakari and there is a lot of work already happening in the OpenStack community to improve this existing solution.  Also this failure problem can only be solved by resurrection (nova evacuate), not evacuation (nova live-migration).  So I would instead suggest the following problem description:\n\n\u003e Sometimes compute nodes need to be evacuated as part of an optimisation, for example in order to consolidate VMs onto fewer compute nodes in order to be able to shut down compute nodes and reduce power consumption.","commit_id":"6bb6b8682c57afed8e2c216407ff40ac50d95804"},{"author":{"_account_id":2394,"name":"Adam Spiers","email":"aspiers@suse.com","username":"adam.spiers"},"change_message_id":"f32188f1e88cdffc9c605c418b147d6388aaf65f","unresolved":false,"context_lines":[{"line_number":14,"context_line":"Problem description"},{"line_number":15,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":16,"context_line":""},{"line_number":17,"context_line":"Sometimes if the failure has happened to one compute node, it would be"},{"line_number":18,"context_line":"marked as down. Then we need to evacuate the instances which are running"},{"line_number":19,"context_line":"on this failure compute node."},{"line_number":20,"context_line":""},{"line_number":21,"context_line":"It needs to do following things:"},{"line_number":22,"context_line":"* add an action to evacuate all the instances of the failure compute node"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7f287b81_cfa576b7","line":19,"range":{"start_line":17,"start_character":0,"end_line":19,"end_character":29},"in_reply_to":"7f287b81_35d7b6f0","updated":"2017-08-23 13:21:27.000000000","message":"\u003e If we restrict Watcher to previous goals, how it evolves? \n\nI did not say we should restrict Watcher to previous goals.  I said that Watcher\u0027s public mission statement at https://wiki.openstack.org/wiki/Watcher#Mission is around optimizing OpenStack and nothing else.  And I agree 100% with this mission statement.  Optimization in itself is a huge area, in which there is easily enough work for the next 10 years at least, so there is plenty of room for Watcher to evolve without changing the mission statement.\n\nIf you want to start extending Watcher beyond optimization, into HA or other areas, then you should communicate that to the rest of the OpenStack community and reword Watcher\u0027s mission goal.  But by definition, optimization does not include HA.  Efficiency and performance are separate areas to availability and resilience.\n\n\u003e So long as it reschedules the resources for a reasonable reason, I think it should be regarded as optimization\n\nWhy?  That approach ignores the definition of optimization.","commit_id":"6bb6b8682c57afed8e2c216407ff40ac50d95804"},{"author":{"_account_id":2394,"name":"Adam Spiers","email":"aspiers@suse.com","username":"adam.spiers"},"change_message_id":"00dc5ebf9dbb311df6ce836646f3c293fcd516d5","unresolved":false,"context_lines":[{"line_number":14,"context_line":"Problem description"},{"line_number":15,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":16,"context_line":""},{"line_number":17,"context_line":"Sometimes if the failure has happened to one compute node, it would be"},{"line_number":18,"context_line":"marked as down. Then we need to evacuate the instances which are running"},{"line_number":19,"context_line":"on this failure compute node."},{"line_number":20,"context_line":""},{"line_number":21,"context_line":"It needs to do following things:"},{"line_number":22,"context_line":"* add an action to evacuate all the instances of the failure compute node"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7f287b81_e2186076","line":19,"range":{"start_line":17,"start_character":0,"end_line":19,"end_character":29},"in_reply_to":"7f287b81_98059a7d","updated":"2017-08-22 18:24:40.000000000","message":"The action (evacuation of all VMs from a compute host via live migration) is the same, but the use case is different.  The use case which I am mentioning here is VM consolidation in order to reduce power consumption, whereas the use case in https://review.openstack.org/#/c/464602/ is to facilitate compute node maintenance without interruption.  The former is a type of automated optimisation, whereas the latter is not, so I wonder why it is being proposed for Watcher.  But we can discuss that in the other spec.","commit_id":"6bb6b8682c57afed8e2c216407ff40ac50d95804"},{"author":{"_account_id":24501,"name":"sue","display_name":"suzhengwei","email":"sugar-2008@163.com","username":"suzhengwei"},"change_message_id":"5ff0e0445d16379fc1eb800906336a1a617e4b9d","unresolved":false,"context_lines":[{"line_number":14,"context_line":"Problem description"},{"line_number":15,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":16,"context_line":""},{"line_number":17,"context_line":"Sometimes if the failure has happened to one compute node, it would be"},{"line_number":18,"context_line":"marked as down. Then we need to evacuate the instances which are running"},{"line_number":19,"context_line":"on this failure compute node."},{"line_number":20,"context_line":""},{"line_number":21,"context_line":"It needs to do following things:"},{"line_number":22,"context_line":"* add an action to evacuate all the instances of the failure compute node"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7f287b81_fb4fbbba","line":19,"range":{"start_line":17,"start_character":0,"end_line":19,"end_character":29},"in_reply_to":"7f287b81_cfa576b7","updated":"2017-08-24 11:03:56.000000000","message":"Compute HA is one huge and complicated solution. This spec is just to solve specific problem. I think this spec is for optization.\nWhat\u0027s more, watcher supports a pluggable architecture, one strategy for one independent problem. That makes the implement more easly once you can split your complicated problem.","commit_id":"6bb6b8682c57afed8e2c216407ff40ac50d95804"},{"author":{"_account_id":24501,"name":"sue","display_name":"suzhengwei","email":"sugar-2008@163.com","username":"suzhengwei"},"change_message_id":"bbbc4ca5245415dbceab974b92ca0a3ea15a2fb8","unresolved":false,"context_lines":[{"line_number":14,"context_line":"Problem description"},{"line_number":15,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":16,"context_line":""},{"line_number":17,"context_line":"Sometimes if the failure has happened to one compute node, it would be"},{"line_number":18,"context_line":"marked as down. Then we need to evacuate the instances which are running"},{"line_number":19,"context_line":"on this failure compute node."},{"line_number":20,"context_line":""},{"line_number":21,"context_line":"It needs to do following things:"},{"line_number":22,"context_line":"* add an action to evacuate all the instances of the failure compute node"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7f287b81_35d7b6f0","line":19,"range":{"start_line":17,"start_character":0,"end_line":19,"end_character":29},"in_reply_to":"7f287b81_e2186076","updated":"2017-08-23 08:34:09.000000000","message":"If we restrict Watcher to previous goals, how it evolves? \nWe judge whether it is optimization not only by the goal name but also the actions it takes, most important, the result it achieves. So long as it reschedules the resources for a reasonable reason, I think it should be regarded as optimization, and can be proposed for watcher.","commit_id":"6bb6b8682c57afed8e2c216407ff40ac50d95804"},{"author":{"_account_id":2394,"name":"Adam Spiers","email":"aspiers@suse.com","username":"adam.spiers"},"change_message_id":"979c01a86f37752a33a8b3046744767ed84d5d51","unresolved":false,"context_lines":[{"line_number":14,"context_line":"Problem description"},{"line_number":15,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":16,"context_line":""},{"line_number":17,"context_line":"Sometimes if the failure has happened to one compute node, it would be"},{"line_number":18,"context_line":"marked as down. Then we need to evacuate the instances which are running"},{"line_number":19,"context_line":"on this failure compute node."},{"line_number":20,"context_line":""},{"line_number":21,"context_line":"It needs to do following things:"},{"line_number":22,"context_line":"* add an action to evacuate all the instances of the failure compute node"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7f287b81_fc720e49","line":19,"range":{"start_line":17,"start_character":0,"end_line":19,"end_character":29},"in_reply_to":"7f287b81_fb4fbbba","updated":"2017-08-24 14:16:11.000000000","message":"\u003e Compute HA is one huge and complicated solution.\n\nYes it is quite complicated, which is why there is a whole project called Masakari dedicated to it.  So why start a new one?\n\n\u003e This spec is just to solve specific problem.\n\nIt is trying to solve a problem which is already solved by Masakari.  And unless you have some fencing mechanism your solution will not work anyway:\n\nhttps://youtu.be/lddtWUP_IKQ?t\u003d6m07s\n\nWatcher does not have any fencing mechanism.\n\n\u003e I think this spec is for optization.\n\nThen we will have to agree to disagree.  High availability and optimization are completely different things by definition.  Optimization means taking a cloud which is already working and making it run even better (more efficiently, better performance etc.)  In contrast HA means maximising a cloud\u0027s uptime, by automatically recovering from failures.\n\n\u003e What\u0027s more, watcher supports a pluggable architecture, one strategy for one independent problem. That makes the implement more easly once you can split your complicated problem.\n\nThe presence of a pluggable architecture does not justify solving every problem in Watcher.  You would not reimplement Cinder using Watcher\u0027s pluggable architecture, so why reimplement Masakari?","commit_id":"6bb6b8682c57afed8e2c216407ff40ac50d95804"},{"author":{"_account_id":8716,"name":"Sampath Priyankara","email":"sam47priya@gmail.com","username":"sampath"},"change_message_id":"1a71ce197cf6bed7079af97a74fc004dc0b9082e","unresolved":false,"context_lines":[{"line_number":14,"context_line":"Problem description"},{"line_number":15,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":16,"context_line":""},{"line_number":17,"context_line":"Sometimes if the failure has happened to one compute node, it would be"},{"line_number":18,"context_line":"marked as down. Then we need to evacuate the instances which are running"},{"line_number":19,"context_line":"on this failure compute node."},{"line_number":20,"context_line":""},{"line_number":21,"context_line":"It needs to do following things:"},{"line_number":22,"context_line":"* add an action to evacuate all the instances of the failure compute node"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7f287b81_fe915b48","line":19,"range":{"start_line":17,"start_character":0,"end_line":19,"end_character":29},"in_reply_to":"7f287b81_fc720e49","updated":"2017-08-30 09:47:50.000000000","message":"Please take a look at Masakari architecture and it\u0027s functionality [1]. This is exactly what we do in Masakari.\nNot only the compute node down, we also support VM recovery from process failures and instance failures also. I think,\nWatcher and Masakari should collaborate to implement any\nmissing pieces in the puzzle. We are open to discuss any issues.\n[1] https://wiki.openstack.org/wiki/Masakari","commit_id":"6bb6b8682c57afed8e2c216407ff40ac50d95804"},{"author":{"_account_id":19055,"name":"Alexander Chadin","email":"joker946@gmail.com","username":"joker946"},"change_message_id":"756a41606b92fc3e8cb2d3f65ce17aac81df6dec","unresolved":false,"context_lines":[{"line_number":14,"context_line":"Problem description"},{"line_number":15,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":16,"context_line":""},{"line_number":17,"context_line":"Sometimes if the failure has happened to one compute node, it would be"},{"line_number":18,"context_line":"marked as down. Then we need to evacuate the instances which are running"},{"line_number":19,"context_line":"on this failure compute node."},{"line_number":20,"context_line":""},{"line_number":21,"context_line":"It needs to do following things:"},{"line_number":22,"context_line":"* add an action to evacuate all the instances of the failure compute node"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7f287b81_98059a7d","line":19,"range":{"start_line":17,"start_character":0,"end_line":19,"end_character":29},"in_reply_to":"9f436f4f_2cb858e3","updated":"2017-08-22 14:05:42.000000000","message":"You propose to live migrate instances from the host to another one. Isn\u0027t it the same as described here? https://review.openstack.org/#/c/464602/18","commit_id":"6bb6b8682c57afed8e2c216407ff40ac50d95804"},{"author":{"_account_id":2394,"name":"Adam Spiers","email":"aspiers@suse.com","username":"adam.spiers"},"change_message_id":"44a30c370db543e235a9132c8733eec286e75c59","unresolved":false,"context_lines":[{"line_number":19,"context_line":"on this failure compute node."},{"line_number":20,"context_line":""},{"line_number":21,"context_line":"It needs to do following things:"},{"line_number":22,"context_line":"* add an action to evacuate all the instances of the failure compute node"},{"line_number":23,"context_line":"* add a strategy which will trgger the evacuate action"},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"Use Cases"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9f436f4f_6cdf800c","line":22,"range":{"start_line":22,"start_character":19,"end_line":22,"end_character":73},"updated":"2017-08-21 22:01:09.000000000","message":"As per above, this should say \"live migrate all the instances of the compute node\".","commit_id":"6bb6b8682c57afed8e2c216407ff40ac50d95804"},{"author":{"_account_id":2394,"name":"Adam Spiers","email":"aspiers@suse.com","username":"adam.spiers"},"change_message_id":"979c01a86f37752a33a8b3046744767ed84d5d51","unresolved":false,"context_lines":[{"line_number":19,"context_line":"on this failure compute node."},{"line_number":20,"context_line":""},{"line_number":21,"context_line":"It needs to do following things:"},{"line_number":22,"context_line":"* add an action to evacuate all the instances of the failure compute node"},{"line_number":23,"context_line":"* add a strategy which will trgger the evacuate action"},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"Use Cases"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7f287b81_5ccc8269","line":22,"range":{"start_line":22,"start_character":19,"end_line":22,"end_character":73},"in_reply_to":"7f287b81_fbda1b15","updated":"2017-08-24 14:16:11.000000000","message":"Please read what I wrote above.  I am proposing that this strategy should not deal with the scenario of nova-compute being down, because that is already dealt with by Masakari.","commit_id":"6bb6b8682c57afed8e2c216407ff40ac50d95804"},{"author":{"_account_id":24501,"name":"sue","display_name":"suzhengwei","email":"sugar-2008@163.com","username":"suzhengwei"},"change_message_id":"5ff0e0445d16379fc1eb800906336a1a617e4b9d","unresolved":false,"context_lines":[{"line_number":19,"context_line":"on this failure compute node."},{"line_number":20,"context_line":""},{"line_number":21,"context_line":"It needs to do following things:"},{"line_number":22,"context_line":"* add an action to evacuate all the instances of the failure compute node"},{"line_number":23,"context_line":"* add a strategy which will trgger the evacuate action"},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"Use Cases"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7f287b81_fbda1b15","line":22,"range":{"start_line":22,"start_character":19,"end_line":22,"end_character":73},"in_reply_to":"9f436f4f_6cdf800c","updated":"2017-08-24 11:03:56.000000000","message":"Evacuate action. Live migration doesn\u0027t work when nova-compute is down.","commit_id":"6bb6b8682c57afed8e2c216407ff40ac50d95804"},{"author":{"_account_id":2394,"name":"Adam Spiers","email":"aspiers@suse.com","username":"adam.spiers"},"change_message_id":"44a30c370db543e235a9132c8733eec286e75c59","unresolved":false,"context_lines":[{"line_number":20,"context_line":""},{"line_number":21,"context_line":"It needs to do following things:"},{"line_number":22,"context_line":"* add an action to evacuate all the instances of the failure compute node"},{"line_number":23,"context_line":"* add a strategy which will trgger the evacuate action"},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"Use Cases"},{"line_number":26,"context_line":"---------"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9f436f4f_2ce978b6","line":23,"range":{"start_line":23,"start_character":28,"end_line":23,"end_character":54},"updated":"2017-08-21 22:01:09.000000000","message":"\"trigger the live-migration action\"","commit_id":"6bb6b8682c57afed8e2c216407ff40ac50d95804"},{"author":{"_account_id":24501,"name":"sue","display_name":"suzhengwei","email":"sugar-2008@163.com","username":"suzhengwei"},"change_message_id":"5ff0e0445d16379fc1eb800906336a1a617e4b9d","unresolved":false,"context_lines":[{"line_number":20,"context_line":""},{"line_number":21,"context_line":"It needs to do following things:"},{"line_number":22,"context_line":"* add an action to evacuate all the instances of the failure compute node"},{"line_number":23,"context_line":"* add a strategy which will trgger the evacuate action"},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"Use Cases"},{"line_number":26,"context_line":"---------"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7f287b81_fb77bb39","line":23,"range":{"start_line":23,"start_character":28,"end_line":23,"end_character":54},"in_reply_to":"9f436f4f_2ce978b6","updated":"2017-08-24 11:03:56.000000000","message":"ditto","commit_id":"6bb6b8682c57afed8e2c216407ff40ac50d95804"},{"author":{"_account_id":2394,"name":"Adam Spiers","email":"aspiers@suse.com","username":"adam.spiers"},"change_message_id":"44a30c370db543e235a9132c8733eec286e75c59","unresolved":false,"context_lines":[{"line_number":25,"context_line":"Use Cases"},{"line_number":26,"context_line":"---------"},{"line_number":27,"context_line":""},{"line_number":28,"context_line":"As an openstack adminisrator, I want to evacuate all the instances if"},{"line_number":29,"context_line":"one compute node is down."},{"line_number":30,"context_line":""},{"line_number":31,"context_line":""},{"line_number":32,"context_line":"Project Priority"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9f436f4f_acd448ea","line":29,"range":{"start_line":28,"start_character":0,"end_line":29,"end_character":25},"updated":"2017-08-21 22:01:09.000000000","message":"As an OpenStack operator, I want to live-migrate all instances off a compute node in order to be able to shut the compute node down.","commit_id":"6bb6b8682c57afed8e2c216407ff40ac50d95804"},{"author":{"_account_id":2394,"name":"Adam Spiers","email":"aspiers@suse.com","username":"adam.spiers"},"change_message_id":"979c01a86f37752a33a8b3046744767ed84d5d51","unresolved":false,"context_lines":[{"line_number":25,"context_line":"Use Cases"},{"line_number":26,"context_line":"---------"},{"line_number":27,"context_line":""},{"line_number":28,"context_line":"As an openstack adminisrator, I want to evacuate all the instances if"},{"line_number":29,"context_line":"one compute node is down."},{"line_number":30,"context_line":""},{"line_number":31,"context_line":""},{"line_number":32,"context_line":"Project Priority"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7f287b81_5cb24296","line":29,"range":{"start_line":28,"start_character":0,"end_line":29,"end_character":25},"in_reply_to":"7f287b81_fb4a1b53","updated":"2017-08-24 14:16:11.000000000","message":"So please explain why you want to reinvent Masakari.","commit_id":"6bb6b8682c57afed8e2c216407ff40ac50d95804"},{"author":{"_account_id":24501,"name":"sue","display_name":"suzhengwei","email":"sugar-2008@163.com","username":"suzhengwei"},"change_message_id":"5ff0e0445d16379fc1eb800906336a1a617e4b9d","unresolved":false,"context_lines":[{"line_number":25,"context_line":"Use Cases"},{"line_number":26,"context_line":"---------"},{"line_number":27,"context_line":""},{"line_number":28,"context_line":"As an openstack adminisrator, I want to evacuate all the instances if"},{"line_number":29,"context_line":"one compute node is down."},{"line_number":30,"context_line":""},{"line_number":31,"context_line":""},{"line_number":32,"context_line":"Project Priority"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7f287b81_fb4a1b53","line":29,"range":{"start_line":28,"start_character":0,"end_line":29,"end_character":25},"in_reply_to":"9f436f4f_acd448ea","updated":"2017-08-24 11:03:56.000000000","message":"You can submit another spec if you want. But this spec is about what to do for workload after nova-compute down.","commit_id":"6bb6b8682c57afed8e2c216407ff40ac50d95804"},{"author":{"_account_id":2394,"name":"Adam Spiers","email":"aspiers@suse.com","username":"adam.spiers"},"change_message_id":"44a30c370db543e235a9132c8733eec286e75c59","unresolved":false,"context_lines":[{"line_number":37,"context_line":"Proposed change"},{"line_number":38,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":39,"context_line":""},{"line_number":40,"context_line":"This spec will import one goal and strategy, which will trigger the"},{"line_number":41,"context_line":"evacuate action, if one compute node is down."},{"line_number":42,"context_line":"* Add one new goal - \"Workload Evacuate\""},{"line_number":43,"context_line":"* Add one new strategy for this goal - \"Workload Evacuate\""},{"line_number":44,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"9f436f4f_2c029870","line":41,"range":{"start_line":40,"start_character":45,"end_line":41,"end_character":45},"updated":"2017-08-21 22:01:09.000000000","message":"\"which will trigger the live migration action, if a compute node needs to be shut down.\"","commit_id":"6bb6b8682c57afed8e2c216407ff40ac50d95804"},{"author":{"_account_id":2394,"name":"Adam Spiers","email":"aspiers@suse.com","username":"adam.spiers"},"change_message_id":"979c01a86f37752a33a8b3046744767ed84d5d51","unresolved":false,"context_lines":[{"line_number":37,"context_line":"Proposed change"},{"line_number":38,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":39,"context_line":""},{"line_number":40,"context_line":"This spec will import one goal and strategy, which will trigger the"},{"line_number":41,"context_line":"evacuate action, if one compute node is down."},{"line_number":42,"context_line":"* Add one new goal - \"Workload Evacuate\""},{"line_number":43,"context_line":"* Add one new strategy for this goal - \"Workload Evacuate\""},{"line_number":44,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"7f287b81_f768e787","line":41,"range":{"start_line":40,"start_character":45,"end_line":41,"end_character":45},"in_reply_to":"7f287b81_fb1c5b6b","updated":"2017-08-24 14:16:11.000000000","message":"This is the same discussion as above, so I won\u0027t repeat myself.","commit_id":"6bb6b8682c57afed8e2c216407ff40ac50d95804"},{"author":{"_account_id":24501,"name":"sue","display_name":"suzhengwei","email":"sugar-2008@163.com","username":"suzhengwei"},"change_message_id":"5ff0e0445d16379fc1eb800906336a1a617e4b9d","unresolved":false,"context_lines":[{"line_number":37,"context_line":"Proposed change"},{"line_number":38,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":39,"context_line":""},{"line_number":40,"context_line":"This spec will import one goal and strategy, which will trigger the"},{"line_number":41,"context_line":"evacuate action, if one compute node is down."},{"line_number":42,"context_line":"* Add one new goal - \"Workload Evacuate\""},{"line_number":43,"context_line":"* Add one new strategy for this goal - \"Workload Evacuate\""},{"line_number":44,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"7f287b81_fb1c5b6b","line":41,"range":{"start_line":40,"start_character":45,"end_line":41,"end_character":45},"in_reply_to":"9f436f4f_2c029870","updated":"2017-08-24 11:03:56.000000000","message":"This spec does\u0027t  involve this problem. It is to evacuate instances after nova-compute down.\nExactly, the solution in the spec is not compute HA. Compute HA is one huge and complicated solution. But this spec is just to solve specific problem.","commit_id":"6bb6b8682c57afed8e2c216407ff40ac50d95804"},{"author":{"_account_id":2394,"name":"Adam Spiers","email":"aspiers@suse.com","username":"adam.spiers"},"change_message_id":"44a30c370db543e235a9132c8733eec286e75c59","unresolved":false,"context_lines":[{"line_number":43,"context_line":"* Add one new strategy for this goal - \"Workload Evacuate\""},{"line_number":44,"context_line":""},{"line_number":45,"context_line":"The new strategy executes as follows"},{"line_number":46,"context_line":"* First, get the parameter \"duration\", it\u0027s a threshold value and represents"},{"line_number":47,"context_line":"  how long the \u0027down\u0027 state could last."},{"line_number":48,"context_line":"* Then, check the down compute nodes. If one node has instances residented"},{"line_number":49,"context_line":"  and has been in \u0027down\u0027 state for more than the given parameter \"duration\","},{"line_number":50,"context_line":"  evacuate all the instances of the node."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"This strategy just consider how to evacuate all instances of the failure"},{"line_number":53,"context_line":"compute node, further optim rely on other strategies."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9f436f4f_acde88d7","line":50,"range":{"start_line":46,"start_character":0,"end_line":50,"end_character":41},"updated":"2017-08-21 22:01:09.000000000","message":"This is an approach to solving an HA problem, but it would not work even for HA.  In order to properly implement HA you would need fencing.  For an evacuation strategy, it should just be triggered as part of the optimisation.","commit_id":"6bb6b8682c57afed8e2c216407ff40ac50d95804"},{"author":{"_account_id":17517,"name":"Andrew Beekhof","email":"abeekhof@redhat.com","username":"beekhof"},"change_message_id":"1f87215d8fc3d0a38ff27dd83d0a4a9c7e223f3e","unresolved":false,"context_lines":[{"line_number":43,"context_line":"* Add one new strategy for this goal - \"Workload Evacuate\""},{"line_number":44,"context_line":""},{"line_number":45,"context_line":"The new strategy executes as follows"},{"line_number":46,"context_line":"* First, get the parameter \"duration\", it\u0027s a threshold value and represents"},{"line_number":47,"context_line":"  how long the \u0027down\u0027 state could last."},{"line_number":48,"context_line":"* Then, check the down compute nodes. If one node has instances residented"},{"line_number":49,"context_line":"  and has been in \u0027down\u0027 state for more than the given parameter \"duration\","},{"line_number":50,"context_line":"  evacuate all the instances of the node."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"This strategy just consider how to evacuate all instances of the failure"},{"line_number":53,"context_line":"compute node, further optim rely on other strategies."}],"source_content_type":"text/x-rst","patch_set":2,"id":"7f287b81_d9418b3b","line":50,"range":{"start_line":46,"start_character":0,"end_line":50,"end_character":41},"in_reply_to":"9f436f4f_acde88d7","updated":"2017-08-25 04:31:13.000000000","message":"Exactly.\n\nI think we can all agree that it makes sense for Watcher to have a role in moving instances around as part of an optimisation process or in _anticipation_ of a failure.\n\nHowever without fencing it makes no sense for Watcher to be playing in the \"recovery\" space and it is highly concerning to see recovery solutions being proposed that lack this core functionality.\n\nTimeouts are not a replacement for fencing and relying on them alone makes it trivial to construct a scenario in which an instance was active on two computes - potentially leading to billing issues, inconsistent application behaviour, data corruption, or all three.","commit_id":"6bb6b8682c57afed8e2c216407ff40ac50d95804"},{"author":{"_account_id":24501,"name":"sue","display_name":"suzhengwei","email":"sugar-2008@163.com","username":"suzhengwei"},"change_message_id":"5ff0e0445d16379fc1eb800906336a1a617e4b9d","unresolved":false,"context_lines":[{"line_number":43,"context_line":"* Add one new strategy for this goal - \"Workload Evacuate\""},{"line_number":44,"context_line":""},{"line_number":45,"context_line":"The new strategy executes as follows"},{"line_number":46,"context_line":"* First, get the parameter \"duration\", it\u0027s a threshold value and represents"},{"line_number":47,"context_line":"  how long the \u0027down\u0027 state could last."},{"line_number":48,"context_line":"* Then, check the down compute nodes. If one node has instances residented"},{"line_number":49,"context_line":"  and has been in \u0027down\u0027 state for more than the given parameter \"duration\","},{"line_number":50,"context_line":"  evacuate all the instances of the node."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"This strategy just consider how to evacuate all instances of the failure"},{"line_number":53,"context_line":"compute node, further optim rely on other strategies."}],"source_content_type":"text/x-rst","patch_set":2,"id":"7f287b81_2d1d060f","line":50,"range":{"start_line":46,"start_character":0,"end_line":50,"end_character":41},"in_reply_to":"9f436f4f_acde88d7","updated":"2017-08-24 11:03:56.000000000","message":"ditto\nExactly, this is not compute HA. It is  just to solve this specific problem, what to do for workload after nova-compute down. And I admit it is not useful to the users who have integrated solutions of compute HA.","commit_id":"6bb6b8682c57afed8e2c216407ff40ac50d95804"},{"author":{"_account_id":2394,"name":"Adam Spiers","email":"aspiers@suse.com","username":"adam.spiers"},"change_message_id":"44a30c370db543e235a9132c8733eec286e75c59","unresolved":false,"context_lines":[{"line_number":50,"context_line":"  evacuate all the instances of the node."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"This strategy just consider how to evacuate all instances of the failure"},{"line_number":53,"context_line":"compute node, further optim rely on other strategies."},{"line_number":54,"context_line":""},{"line_number":55,"context_line":""},{"line_number":56,"context_line":"Alternatives"}],"source_content_type":"text/x-rst","patch_set":2,"id":"9f436f4f_ccaaec17","line":53,"range":{"start_line":53,"start_character":22,"end_line":53,"end_character":27},"updated":"2017-08-21 22:01:09.000000000","message":"\"optimisation\"\n\nbut I agree with this sentence.  It would work in conjunction with the strategy for powering off compute hosts.","commit_id":"6bb6b8682c57afed8e2c216407ff40ac50d95804"},{"author":{"_account_id":2394,"name":"Adam Spiers","email":"aspiers@suse.com","username":"adam.spiers"},"change_message_id":"979c01a86f37752a33a8b3046744767ed84d5d51","unresolved":false,"context_lines":[{"line_number":50,"context_line":"  evacuate all the instances of the node."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"This strategy just consider how to evacuate all instances of the failure"},{"line_number":53,"context_line":"compute node, further optim rely on other strategies."},{"line_number":54,"context_line":""},{"line_number":55,"context_line":""},{"line_number":56,"context_line":"Alternatives"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7f287b81_d7372bb0","line":53,"range":{"start_line":53,"start_character":22,"end_line":53,"end_character":27},"in_reply_to":"7f287b81_add69626","updated":"2017-08-24 14:16:11.000000000","message":"Yes there is: https://blueprints.launchpad.net/watcher/+spec/strategy-to-trigger-power-on-and-power-off-actions","commit_id":"6bb6b8682c57afed8e2c216407ff40ac50d95804"},{"author":{"_account_id":24501,"name":"sue","display_name":"suzhengwei","email":"sugar-2008@163.com","username":"suzhengwei"},"change_message_id":"5ff0e0445d16379fc1eb800906336a1a617e4b9d","unresolved":false,"context_lines":[{"line_number":50,"context_line":"  evacuate all the instances of the node."},{"line_number":51,"context_line":""},{"line_number":52,"context_line":"This strategy just consider how to evacuate all instances of the failure"},{"line_number":53,"context_line":"compute node, further optim rely on other strategies."},{"line_number":54,"context_line":""},{"line_number":55,"context_line":""},{"line_number":56,"context_line":"Alternatives"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7f287b81_add69626","line":53,"range":{"start_line":53,"start_character":22,"end_line":53,"end_character":27},"in_reply_to":"9f436f4f_ccaaec17","updated":"2017-08-24 11:03:56.000000000","message":"yes, but there is no strategy for powering off compute nodes. You can submit such atrategy.","commit_id":"6bb6b8682c57afed8e2c216407ff40ac50d95804"}]}