)]}'
{"specs/rpc-health-checks.rst":[{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"d03ebc14bff0261763ab35cbf1b1f2af35a4322c","unresolved":false,"context_lines":[{"line_number":40,"context_line":"success, e.g.::"},{"line_number":41,"context_line":""},{"line_number":42,"context_line":"    def health_check_simple(self, context):"},{"line_number":43,"context_line":"        LOG.debug(\"It\u0027s alive!\")"},{"line_number":44,"context_line":""},{"line_number":45,"context_line":"The initial goal is simply to solve the problem with OpenStack Helm"},{"line_number":46,"context_line":"described above: health probes should do RPC calls with real methods,"}],"source_content_type":"text/x-rst","patch_set":2,"id":"ffb9cba7_f9d68e27","line":43,"updated":"2019-04-26 16:11:09.000000000","message":"We already have this built into all services in the form of the ping() method. It\u0027s used by compute to halt startup until at least one conductor service is running, since compute requires conductor for basic operation:\n\nhttps://github.com/openstack/nova/blob/master/nova/conductor/api.py#L66\n\nYou should be able to ping() any service that is running and listening for RPC calls.\n\nThat said, nova also already kinda does this in the form of service checkins. Meaning, if RPC is working, compute writes its liveness to the DB via conductor. The other services do the same, but directly to do the DB.\n\nTo me, health checks should really be done as described previously, via supporting the normal http-based health checks on each service, which can then report things like whether or not RPC seems to be working, among other things. Doing it in-band with RPC doesn\u0027t really tell us much that the existing service reporting isn\u0027t already telling us. If the rpc in-band check works, then likely the service checkin is working as well.\n\nFurther, the above-mentioned probe tool puts a non-nova component on the private nova rpc bus, which isn\u0027t a very good idea, IMHO.","commit_id":"a7aca4f0575666610526e24f31a41a4a7014e7be"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"6a12564e07b646e08efd9bea6aed559b12356088","unresolved":false,"context_lines":[{"line_number":40,"context_line":"success, e.g.::"},{"line_number":41,"context_line":""},{"line_number":42,"context_line":"    def health_check_simple(self, context):"},{"line_number":43,"context_line":"        LOG.debug(\"It\u0027s alive!\")"},{"line_number":44,"context_line":""},{"line_number":45,"context_line":"The initial goal is simply to solve the problem with OpenStack Helm"},{"line_number":46,"context_line":"described above: health probes should do RPC calls with real methods,"}],"source_content_type":"text/x-rst","patch_set":2,"id":"dfbec78f_cb8d2132","line":43,"in_reply_to":"dfbec78f_2b3cfdc6","updated":"2019-05-07 17:46:14.000000000","message":"\u003e @Dan Firstly, apologies for not managing to grab you at the PTG for\n \u003e a quick hallway discussion on this.  Had too many other things to\n \u003e think about :-/\n \u003e \n \u003e \u003e That said, nova also already kinda does this in the form of\n \u003e service checkins. Meaning, if RPC is working, compute writes its\n \u003e liveness to the DB via conductor. The other services do the same,\n \u003e but directly to do the DB.\n \u003e \n \u003e This is of course useful, but to me it doesn\u0027t feel like a complete\n \u003e solution to the problem of pin-pointing failures during automated\n \u003e root cause analysis or self-healing.  For example, if a service\u0027s\n \u003e liveness updates cease to appear in the DB, that doesn\u0027t give us an\n \u003e accurate read on where the failure is.  For example, it could be an\n \u003e issue with the DB or the network on the DB end, or with the network\n \u003e on the service\u0027s end, or an issue with the whole machine running\n \u003e the service.  All of these are of course significantly different\n \u003e failure modes to \"there\u0027s a problem with the service\".\n\nSure, which is why I\u0027m saying a direct-to-the-node health check that doesn\u0027t rely on any of the other services is likely to give you a lot more detail. For example, if you\u0027re unable to contact it over RPC, that could be because (a) rabbit is down or (b) conductor is down and the threadpool on the compute has become exhausted trying to check in.\n\n \u003e To clarify, are you suggesting that RPC-only services which don\u0027t\n \u003e currently expose an HTTP endpoint should do so?  That\u0027s something\n \u003e the community has been considering for quite a while in the context\n \u003e of the health check APIs discussion:\n \u003e \n \u003e https://storyboard.openstack.org/#!/story/2001439\n\nRight.\n\n \u003e but it has been a somewhat controversial suggestion, with some\n \u003e people concerned about potential bloat and impact on the\n \u003e security/deployment model.  Having said that, Dirk and I did\n \u003e discuss it with Ben Nemec on Saturday just as the PTG finished, and\n \u003e he didn\u0027t seem to have violent objections. (FWIW personally I\n \u003e haven\u0027t made up my mind either way yet.)\n\nI think it\u0027s a lot more useful than relying on RPC, which is the most likely reason for an unhealthy service (IMHO and IME).\n\n \u003e If OTOH you are suggesting that the existing HTTP endpoints should\n \u003e act as proxies for RPC component health, then I\u0027m pretty sure\n \u003e that\u0027s not going to work for the reasons stated above.\n\nNo, I\u0027m talking about individual shared-with-nothing endpoints. Super simple, minimal bloat, based on something generic that other projects are using for similar things.\n\n \u003e \"likely\" is the key word here.  As mentioned above, for effective\n \u003e RCA via the likes of Vitrage, and effective managment by the likes\n \u003e of k8s, we need checks to be as accurate as possible, otherwise we\n \u003e can\u0027t build self-healing workflows which we can trust to do the\n \u003e right thing.  For example, if the conductor dies, we can\u0027t afford\n \u003e our compute liveness probes to trigger false alarms, even\n \u003e momentarily.\n\nIn any real deployment you have many conductors across multiple nodes (even in the smallest of arrangements), so all of them falling over at once is not very \"likely\". Further, if you\u0027re going for RCA, and you find that no conductors are up (or even that some are down) then the dependency graph tells you that worrying about computes being up is pointless. Meaning, just from nova\u0027s service reporting, I can tell a lot:\n\n 1. Are any conductors reporting? If no, then rabbit or conductor. If yes,\n 2. Are any computes reporting? If no, then rabbit, if yes,\n 3. Only some computes are reporting, likely problem with the non-reporting computes (or their networks)\n\n \u003e This is an interesting point.  Any chance you could elaborate on\n \u003e your concerns here?  Of course an RPC liveness probe would have to\n \u003e act responsibly and not spam the bus, but are there other risks\n \u003e here, e.g. security-related ones?\n\nA polling approach to liveness checks over the RPC bus is going to add load to the single most loaded service in openstack, rabbit. Any additional load you add there is going to go unappreciated by large deployments. My biggest concerns, however, is that on our bus, we are in charge of all the schemas, interfaces, and upgrade compatibility. If you introduce an interface for a health check, that\u0027s something nova exposes which it does not consume, and opens us up to the situation where our bus is not intended to be private. It means the external services now have some expectation of stability of that interface, which we manage internally for everything at the moment.\n\nIn general, Nova has external APIs at the REST layer and the metadata service and that\u0027s it. We take extreme care to keep those stable. Our RPC interfaces are private, versioned, and lifecycle managed internally. Altering that would be a pretty big change, and not for a really significant gain, because as I said, I think an http api per node would be a *lot* more useful, as it can continue to report information independent of pretty much everything else.","commit_id":"a7aca4f0575666610526e24f31a41a4a7014e7be"},{"author":{"_account_id":2394,"name":"Adam Spiers","email":"aspiers@suse.com","username":"adam.spiers"},"change_message_id":"aa362da55d20328635fa398cb08952de4312d7e4","unresolved":false,"context_lines":[{"line_number":40,"context_line":"success, e.g.::"},{"line_number":41,"context_line":""},{"line_number":42,"context_line":"    def health_check_simple(self, context):"},{"line_number":43,"context_line":"        LOG.debug(\"It\u0027s alive!\")"},{"line_number":44,"context_line":""},{"line_number":45,"context_line":"The initial goal is simply to solve the problem with OpenStack Helm"},{"line_number":46,"context_line":"described above: health probes should do RPC calls with real methods,"}],"source_content_type":"text/x-rst","patch_set":2,"id":"dfbec78f_3a41a83a","line":43,"in_reply_to":"dfbec78f_cb8d2132","updated":"2019-05-07 19:32:07.000000000","message":"Got it, thanks. Along with the discussion in IRC just now[1] I think you\u0027ve persuaded me that this is the right way to go.  New dedicated endpoints specifically targeted at exposing this health data seem to have several advantages over doing it over RPC.\n\nSo maybe we should abandon this spec and consolidate efforts with https://storyboard.openstack.org/#!/story/2001439\n\nThe third option which was discussed at the PTG but not mentioned on this review until now, was the Guru Meditation Reports offered by oslo.reports.  However these reports are dumped to STDERR by default, so that behaviour would need some tweaks before it would be easily consumable by a container liveness probe or similar.\n\n[1] http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-05-07.log.html#t2019-05-07T17:50:38","commit_id":"a7aca4f0575666610526e24f31a41a4a7014e7be"},{"author":{"_account_id":2394,"name":"Adam Spiers","email":"aspiers@suse.com","username":"adam.spiers"},"change_message_id":"5556509539e4501afd10bb4e2954f94295306a43","unresolved":false,"context_lines":[{"line_number":40,"context_line":"success, e.g.::"},{"line_number":41,"context_line":""},{"line_number":42,"context_line":"    def health_check_simple(self, context):"},{"line_number":43,"context_line":"        LOG.debug(\"It\u0027s alive!\")"},{"line_number":44,"context_line":""},{"line_number":45,"context_line":"The initial goal is simply to solve the problem with OpenStack Helm"},{"line_number":46,"context_line":"described above: health probes should do RPC calls with real methods,"}],"source_content_type":"text/x-rst","patch_set":2,"id":"dfbec78f_2b3cfdc6","line":43,"in_reply_to":"ffb9cba7_f9d68e27","updated":"2019-05-07 17:22:24.000000000","message":"@Dan Firstly, apologies for not managing to grab you at the PTG for a quick hallway discussion on this.  Had too many other things to think about :-/\n\n \u003e That said, nova also already kinda does this in the form of service checkins. Meaning, if RPC is working, compute writes its liveness to the DB via conductor. The other services do the same, but directly to do the DB.\n\nThis is of course useful, but to me it doesn\u0027t feel like a complete solution to the problem of pin-pointing failures during automated root cause analysis or self-healing.  For example, if a service\u0027s liveness updates cease to appear in the DB, that doesn\u0027t give us an accurate read on where the failure is.  For example, it could be an issue with the DB or the network on the DB end, or with the network on the service\u0027s end, or an issue with the whole machine running the service.  All of these are of course significantly different failure modes to \"there\u0027s a problem with the service\".\n\nAdditionally, if another service such as conductor is proxying the liveness, then that adds another layer of failure modes which can potentially muddy the waters.\n\n \u003e To me, health checks should really be done as described previously, via supporting the normal http-based health checks on each service, which can then report things like whether or not RPC seems to be working, among other things.\n\nTo clarify, are you suggesting that RPC-only services which don\u0027t currently expose an HTTP endpoint should do so?  That\u0027s something the community has been considering for quite a while in the context of the health check APIs discussion:\n\n   https://storyboard.openstack.org/#!/story/2001439\n\nbut it has been a somewhat controversial suggestion, with some people concerned about potential bloat and impact on the security/deployment model.  Having said that, Dirk and I did discuss it with Ben Nemec on Saturday just as the PTG finished, and he didn\u0027t seem to have violent objections. (FWIW personally I haven\u0027t made up my mind either way yet.)\n\nIf OTOH you are suggesting that the existing HTTP endpoints should act as proxies for RPC component health, then I\u0027m pretty sure that\u0027s not going to work for the reasons stated above.\n\nAnd even though we *do* potentially want to expand the existing HTTP endpoints to provide not just health-check data but also internal performance metrics (which was the main thrust of the Saturday discussion), this is out of scope for the specific use case we are trying to address here, i.e. allowing accurate liveness probes from k8s to RPC services.  In that use case, k8s doesn\u0027t need anything more than a simple binary dead-or-alive answer.\n\n \u003e Doing it in-band with RPC doesn\u0027t really tell us much that the existing service reporting isn\u0027t already telling us. If the rpc in-band check works, then likely the service checkin is working as well.\n\n\"likely\" is the key word here.  As mentioned above, for effective RCA via the likes of Vitrage, and effective managment by the likes of k8s, we need checks to be as accurate as possible, otherwise we can\u0027t build self-healing workflows which we can trust to do the right thing.  For example, if the conductor dies, we can\u0027t afford our compute liveness probes to trigger false alarms, even momentarily.\n\n \u003e Further, the above-mentioned probe tool puts a non-nova component on the private nova rpc bus, which isn\u0027t a very good idea, IMHO.\n\nThis is an interesting point.  Any chance you could elaborate on your concerns here?  Of course an RPC liveness probe would have to act responsibly and not spam the bus, but are there other risks here, e.g. security-related ones?","commit_id":"a7aca4f0575666610526e24f31a41a4a7014e7be"},{"author":{"_account_id":17068,"name":"Jean-Philippe Evrard","email":"openstack@a.spamming.party","username":"evrardjp"},"change_message_id":"165c8747750cea82ca1f3a6c933effd3b804b957","unresolved":false,"context_lines":[{"line_number":40,"context_line":"success, e.g.::"},{"line_number":41,"context_line":""},{"line_number":42,"context_line":"    def health_check_simple(self, context):"},{"line_number":43,"context_line":"        LOG.debug(\"It\u0027s alive!\")"},{"line_number":44,"context_line":""},{"line_number":45,"context_line":"The initial goal is simply to solve the problem with OpenStack Helm"},{"line_number":46,"context_line":"described above: health probes should do RPC calls with real methods,"}],"source_content_type":"text/x-rst","patch_set":2,"id":"ffb9cba7_a6cf7b54","line":43,"in_reply_to":"ffb9cba7_f9d68e27","updated":"2019-04-28 16:28:49.000000000","message":"Very nice input!\nI guess the next question for that topic is whether or not a common approach should be taken, and how other projects are doing. I suppose the next step would then be to contact other projects to see if they have implemented the same approach, and have a view of what needs achieving over the whole community to establish a common approach.","commit_id":"a7aca4f0575666610526e24f31a41a4a7014e7be"}]}