)]}'
{"specs/liberty/approved/platform-health-aware-scheduling.rst":[{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"44b1888f2d0e5ab772dc7a25d98bd810bc2ec27d","unresolved":false,"context_lines":[{"line_number":20,"context_line":""},{"line_number":21,"context_line":"The proposed solution is about creating a general framework to detect these"},{"line_number":22,"context_line":"conditions, account for them in a systematic way, and to expose platform"},{"line_number":23,"context_line":"health status information to nova scheduler."},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"Problem description"},{"line_number":26,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":4,"id":"9a0c5dc1_2734dadc","line":23,"updated":"2015-05-12 00:52:15.000000000","message":"We already have the ability to handle the above use cases. The `nova-manage service disable $HOST \u0027nova-compute\u0027` command can be executed by an external monitoring system script (such as an NRPE script) after that monitoring system has been notified of the failure condition by whatever watchdog or IPMI status heartbeat.\n\nOnce nova-manage service disable is executed, that host is immediately taken out of the scheduler\u0027s decision-making processes and no instances will be launched against that disabled host.\n\nAn example of such an NRPE script is this:\n\nhttp://exchange.nagios.org/components/com_mtree/attachment.php?link_id\u003d3064\u0026cf_id\u003d24\n\nThe script could be trivially adapted to just call nova-manage service disable for the host on some condition. That would immediately take the node out of scheduling considerations.","commit_id":"b781e91eb86b464ccb3faf133fb5426184c25cb8"},{"author":{"_account_id":7,"name":"Jay Pipes","email":"jaypipes@gmail.com","username":"jaypipes"},"change_message_id":"44b1888f2d0e5ab772dc7a25d98bd810bc2ec27d","unresolved":false,"context_lines":[{"line_number":45,"context_line":""},{"line_number":46,"context_line":"The main use case behind this spec is that user expects VMs to be reliable."},{"line_number":47,"context_line":"By allowing Nova to detect problems in hardware, via IPMI sensor readings,"},{"line_number":48,"context_line":"and avoiding scheduling on \u0027unhealthy\u0027 nodes this requirement can be met."},{"line_number":49,"context_line":""},{"line_number":50,"context_line":"An example is a case where servers have single or dual fan failures, or"},{"line_number":51,"context_line":"voltage or temperature sensor issues."}],"source_content_type":"text/x-rst","patch_set":4,"id":"9a0c5dc1_c7517e10","line":48,"updated":"2015-05-12 00:52:15.000000000","message":"A virtual machine already running on a host isn\u0027t affected by any future scheduling decisions, though...","commit_id":"b781e91eb86b464ccb3faf133fb5426184c25cb8"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"2462c4effe2267526e79975524767e955babe295","unresolved":false,"context_lines":[{"line_number":60,"context_line":"The idea is to leverage Utilization Aware Scheduler infrastructure"},{"line_number":61,"context_line":"and implement new \u0027platform health\u0027 monitor which would gather, at some"},{"line_number":62,"context_line":"interval, platform health status. On Compute Nodes Linux tool /usr/bin/ipmitool"},{"line_number":63,"context_line":"will be used for gathering locally HW sensors status via IPMI."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"New filter and weigher classes will be created for use in nova-scheduler."},{"line_number":66,"context_line":"The filter logic will be that node for which all sensor in given category: Fan,"}],"source_content_type":"text/x-rst","patch_set":4,"id":"bab921fb_51515261","line":63,"updated":"2015-04-24 13:04:11.000000000","message":"The current design is to report what the hypervisor is telling us, and only that. \n\nThere is no other source of truth than the hypervisor and that\u0027s a finite number of things.","commit_id":"b781e91eb86b464ccb3faf133fb5426184c25cb8"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"2462c4effe2267526e79975524767e955babe295","unresolved":false,"context_lines":[{"line_number":62,"context_line":"interval, platform health status. On Compute Nodes Linux tool /usr/bin/ipmitool"},{"line_number":63,"context_line":"will be used for gathering locally HW sensors status via IPMI."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"New filter and weigher classes will be created for use in nova-scheduler."},{"line_number":66,"context_line":"The filter logic will be that node for which all sensor in given category: Fan,"},{"line_number":67,"context_line":"Voltage, Temperature or Current will report critical will be filtered out."},{"line_number":68,"context_line":"Node for which sensors will not be in Normal state will be assigned lower"}],"source_content_type":"text/x-rst","patch_set":4,"id":"bab921fb_f19626ef","line":65,"updated":"2015-04-24 13:04:11.000000000","message":"That sounds not related to that spec or at least you should provide more than just a one-liner, ie. describe all of your filter implementation","commit_id":"b781e91eb86b464ccb3faf133fb5426184c25cb8"},{"author":{"_account_id":7166,"name":"Sylvain Bauza","email":"sbauza@redhat.com","username":"sbauza"},"change_message_id":"2462c4effe2267526e79975524767e955babe295","unresolved":false,"context_lines":[{"line_number":73,"context_line":"Subsequent blueprints could be created to extend scope of gathered platform"},{"line_number":74,"context_line":"health status. For exapmle, VMs relying heavily on RAM or disk health may"},{"line_number":75,"context_line":"benefit from not being instantiated on Compute Nodes which exhibit HDD/SSD"},{"line_number":76,"context_line":"errors, paging/swapping or high CPI IO wait."},{"line_number":77,"context_line":""},{"line_number":78,"context_line":"Alternatives"},{"line_number":79,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"bab921fb_2337b5e4","line":76,"updated":"2015-04-24 13:04:11.000000000","message":"that also makes me think that Nova could then pull a dependency on some 3rd party tools, which is IMHO a bad thing for 2 reasons :\n - that\u0027s not ubiquous (depending on the OS)\n - we can\u0027t gate that, unless using a 3rd party CI\n\nWe all know that a 3rd party CI on every change coming in Nova as a pre-merge thing is just a huge thing.","commit_id":"b781e91eb86b464ccb3faf133fb5426184c25cb8"}]}