)]}'
{"monasca_api/common/repositories/influxdb/metrics_repository.py":[{"author":{"_account_id":12542,"name":"Johannes Grassler","email":"jgr-launchpad@btw23.de","username":"jgrassler"},"change_message_id":"7b76dd9d9362f7c77c351bfa619a9c6cf60becf7","unresolved":false,"context_lines":[{"line_number":60,"context_line":"                self._init_serie_builders_from_v0_11_0()"},{"line_number":61,"context_line":"        except (requests.exceptions.ConnectionError, requests.exceptions.ConnectTimeout):"},{"line_number":62,"context_line":"            # these errors mean that the backend is not ready yet"},{"line_number":63,"context_line":"            raise"},{"line_number":64,"context_line":"        except Exception as ex:"},{"line_number":65,"context_line":"            LOG.exception(ex)"},{"line_number":66,"context_line":"            # Initialize the serie builders to v0_11_0. Not sure when SHOW DIAGNOSTICS added"}],"source_content_type":"text/x-python","patch_set":1,"id":"df87a7cf_a14a4852","line":63,"updated":"2017-12-05 17:06:17.000000000","message":"Please turn this into a retry loop that logs a warning every time it fails. There\u0027s no reason to make this a fatal error, for the root cause for this is usually transient (influxdb not started, yet or temporarily unreachable).","commit_id":"f01eb3f0be4933176ac78c450397a066dbff688d"},{"author":{"_account_id":13478,"name":"Boris Bobrov","email":"b.bobrov@sap.com","username":"bbobrov"},"change_message_id":"c3a841e3e5c564e8dfc89491a2a3abdb0ba0e380","unresolved":false,"context_lines":[{"line_number":60,"context_line":"                self._init_serie_builders_from_v0_11_0()"},{"line_number":61,"context_line":"        except (requests.exceptions.ConnectionError, requests.exceptions.ConnectTimeout):"},{"line_number":62,"context_line":"            # these errors mean that the backend is not ready yet"},{"line_number":63,"context_line":"            raise"},{"line_number":64,"context_line":"        except Exception as ex:"},{"line_number":65,"context_line":"            LOG.exception(ex)"},{"line_number":66,"context_line":"            # Initialize the serie builders to v0_11_0. Not sure when SHOW DIAGNOSTICS added"}],"source_content_type":"text/x-python","patch_set":1,"id":"df87a7cf_7c6387b2","line":63,"in_reply_to":"df87a7cf_216dd803","updated":"2017-12-05 17:55:10.000000000","message":"\u003e I\u0027d go for retrying indefinitely because one should never have to restart monasca-api manually once influxdb is operational once more\n\nI don\u0027t think this is a good idea for several reasons.\n\n1. Indefinite waiting time will break many things with unproper error messages because in many places default timeout is 60 seconds. Afaik in a typical configuration Horizon waits for 60 seconds for reply from monasca AND the browser waits 60 seconds for reply from Horizon. Timeout errors are bad because we are not sure what happened to our action. It led to hardly-diagnosed errors in keystone some time ago.\n\nIt will also mean the panel in Horizon will be \"loading\" for $timeout seconds. Do we really want it?\n\n2. My expectation is that even if monasca-api fails to connect to influxdb on the first attempt, the first request after influxdb is up will spin up a new thread/process where the connection will be re-inited successfully. At least this is what i expect from a wsgi application.\n\nWhat do you think?","commit_id":"f01eb3f0be4933176ac78c450397a066dbff688d"},{"author":{"_account_id":12542,"name":"Johannes Grassler","email":"jgr-launchpad@btw23.de","username":"jgrassler"},"change_message_id":"f88e98df97028fca37fab77c4916da7d998b8f14","unresolved":false,"context_lines":[{"line_number":60,"context_line":"                self._init_serie_builders_from_v0_11_0()"},{"line_number":61,"context_line":"        except (requests.exceptions.ConnectionError, requests.exceptions.ConnectTimeout):"},{"line_number":62,"context_line":"            # these errors mean that the backend is not ready yet"},{"line_number":63,"context_line":"            raise"},{"line_number":64,"context_line":"        except Exception as ex:"},{"line_number":65,"context_line":"            LOG.exception(ex)"},{"line_number":66,"context_line":"            # Initialize the serie builders to v0_11_0. Not sure when SHOW DIAGNOSTICS added"}],"source_content_type":"text/x-python","patch_set":1,"id":"df87a7cf_ea9337a2","line":63,"in_reply_to":"df87a7cf_7c6387b2","updated":"2017-12-06 08:33:50.000000000","message":"\u003e 1. Indefinite waiting time will break many things with unproper error messages because in many places default timeout is 60 seconds. Afaik in a typical configuration Horizon waits for 60 seconds for reply from monasca AND the browser waits 60 seconds for reply from Horizon. Timeout errors are bad because we are not sure what happened to our action. It led to hardly-diagnosed errors in keystone some time ago.\n\u003e It will also mean the panel in Horizon will be \"loading\" for $timeout seconds. Do we really want it?\n\nSo the real problem with retrying is getting into a situation where we cause a timeout here. What I don\u0027t want is monasca-api ending up in an unrecoverable state (and crashing is an unrecoverable state) That\u0027s always a bad thing to happen in a system that\u0027s distributed to this extent. Maybe we could introduce an influxdb_version attribute and set this to None if version retrieval fails. Then we check the state of this attribute in list_dimension_{names,values} for every request that uses it and raise a dedicated exception if influxdb_version indicates a previous failed attempt at retrieving the version. We catch this exception if we encounter it during a request and respond with a 501 and a clear error message as long as that situation persists. That way we get indefinite retrying.\n\n\u003e 2. My expectation is that even if monasca-api fails to connect to influxdb on the first attempt, the first request after influxdb is up will spin up a new thread/process where the connection will be re-inited successfully. At least this is what i expect from a wsgi application.\n\nI\u0027d have to check again, but as far as I remember the main process ended up in a broken state and kept launching broken worker processes (with bad influxdb version state). So crashing the main process hard would probably solve this problem. It\u0027s still not pretty, though (and even in that case we should have a dedicated exception we catch and log before exiting so we do not spam the log file with stack traces). I\u0027d be ok with it, though if the net result is the sort of retrying behaviour I outlined above - just be sure to test it to destruction :-)","commit_id":"f01eb3f0be4933176ac78c450397a066dbff688d"},{"author":{"_account_id":13478,"name":"Boris Bobrov","email":"b.bobrov@sap.com","username":"bbobrov"},"change_message_id":"ad9d3af1253a302763ac2d7f493c80e2aecaf712","unresolved":false,"context_lines":[{"line_number":60,"context_line":"                self._init_serie_builders_from_v0_11_0()"},{"line_number":61,"context_line":"        except (requests.exceptions.ConnectionError, requests.exceptions.ConnectTimeout):"},{"line_number":62,"context_line":"            # these errors mean that the backend is not ready yet"},{"line_number":63,"context_line":"            raise"},{"line_number":64,"context_line":"        except Exception as ex:"},{"line_number":65,"context_line":"            LOG.exception(ex)"},{"line_number":66,"context_line":"            # Initialize the serie builders to v0_11_0. Not sure when SHOW DIAGNOSTICS added"}],"source_content_type":"text/x-python","patch_set":1,"id":"df87a7cf_c143ccae","line":63,"in_reply_to":"df87a7cf_a14a4852","updated":"2017-12-05 17:31:05.000000000","message":"How many times should the loop retry? What should be the sleep time? Today my influxdb took 5 minutes to do something with wal, and my setup wasn\u0027t that big.","commit_id":"f01eb3f0be4933176ac78c450397a066dbff688d"},{"author":{"_account_id":12542,"name":"Johannes Grassler","email":"jgr-launchpad@btw23.de","username":"jgrassler"},"change_message_id":"26e5558c38aef77d44aafa1637aa079d77c1adda","unresolved":false,"context_lines":[{"line_number":60,"context_line":"                self._init_serie_builders_from_v0_11_0()"},{"line_number":61,"context_line":"        except (requests.exceptions.ConnectionError, requests.exceptions.ConnectTimeout):"},{"line_number":62,"context_line":"            # these errors mean that the backend is not ready yet"},{"line_number":63,"context_line":"            raise"},{"line_number":64,"context_line":"        except Exception as ex:"},{"line_number":65,"context_line":"            LOG.exception(ex)"},{"line_number":66,"context_line":"            # Initialize the serie builders to v0_11_0. Not sure when SHOW DIAGNOSTICS added"}],"source_content_type":"text/x-python","patch_set":1,"id":"df87a7cf_216dd803","line":63,"in_reply_to":"df87a7cf_c143ccae","updated":"2017-12-05 17:41:18.000000000","message":"I\u0027d go for retrying indefinitely because one should never have to restart monasca-api manually once influxdb is operational once more (there may after all be a whole cluster of monasca-api instances to deal with, along with poor configuration management that requires waiting a long time to run or that even requires you to restart things manually...). As for sleep time...hmm, good question. I\u0027d go for something on the order of 10 to 30 seconds. That will create a fair amount of visible spam in the log file. This will stand out to whoever debugs the problem but it won\u0027t be so much spam that it ends up overflowing the disk any time soon.\n\nAlso, definitely nothing dynamic/fancy such as exponential backoff: while that may be gentle in terms of load caused by failed attempts, it will also render monasca-api inoperable for a good while longer than influxdb\u0027s actual down time after a couple of failed attempts.","commit_id":"f01eb3f0be4933176ac78c450397a066dbff688d"},{"author":{"_account_id":2419,"name":"Roland Hochmuth","email":"roland.hochmuth@hpe.com","username":"roland-hochmuth-s"},"change_message_id":"c1776a0004ea6d75e1e92576c2af116cb4d4f9a7","unresolved":false,"context_lines":[{"line_number":65,"context_line":"                    requests.exceptions.ConnectTimeout):"},{"line_number":66,"context_line":"                # these errors mean that the backend is not ready yet"},{"line_number":67,"context_line":"                LOG.error(\u0027influxdb is not ready yet, retrying in 10 seconds\u0027)"},{"line_number":68,"context_line":"                time.sleep(10)"},{"line_number":69,"context_line":"            except Exception as ex:"},{"line_number":70,"context_line":"                LOG.exception(ex)"},{"line_number":71,"context_line":"                # Initialize the serie builders to v0_11_0. Not sure when SHOW DIAGNOSTICS added"}],"source_content_type":"text/x-python","patch_set":2,"id":"df87a7cf_c3b658fb","line":68,"updated":"2017-12-06 14:12:08.000000000","message":"The model for Monasca so far has been to fail on start-up if it can\u0027t initialize. So, while I agree adding the additional tests for connectionerror and connecttimeout might be good ideas, looping forever with a 10 second sleep in between is not so good. I would have exited with an error status code and let systemd or kubernetes handle restarting.","commit_id":"518c64706b1489b6d28338dbe2481b819b4967ab"},{"author":{"_account_id":13478,"name":"Boris Bobrov","email":"b.bobrov@sap.com","username":"bbobrov"},"change_message_id":"bcb144eab5312e82712dcc2f7b17137e3d9cc5d2","unresolved":false,"context_lines":[{"line_number":65,"context_line":"                    requests.exceptions.ConnectTimeout):"},{"line_number":66,"context_line":"                # these errors mean that the backend is not ready yet"},{"line_number":67,"context_line":"                LOG.error(\u0027influxdb is not ready yet, retrying in 10 seconds\u0027)"},{"line_number":68,"context_line":"                time.sleep(10)"},{"line_number":69,"context_line":"            except Exception as ex:"},{"line_number":70,"context_line":"                LOG.exception(ex)"},{"line_number":71,"context_line":"                # Initialize the serie builders to v0_11_0. Not sure when SHOW DIAGNOSTICS added"}],"source_content_type":"text/x-python","patch_set":2,"id":"df87a7cf_69e08b97","line":68,"in_reply_to":"df87a7cf_c3b658fb","updated":"2017-12-06 15:23:01.000000000","message":"\u003e I would have exited with an error status code and let systemd or kubernetes handle restarting.\n\nHow would you suggest to do it in case of wsgi app?","commit_id":"518c64706b1489b6d28338dbe2481b819b4967ab"},{"author":{"_account_id":12542,"name":"Johannes Grassler","email":"jgr-launchpad@btw23.de","username":"jgrassler"},"change_message_id":"636bbd205cfe183e5804268544608da89a580957","unresolved":false,"context_lines":[{"line_number":65,"context_line":"                    requests.exceptions.ConnectTimeout):"},{"line_number":66,"context_line":"                # these errors mean that the backend is not ready yet"},{"line_number":67,"context_line":"                LOG.error(\u0027influxdb is not ready yet, retrying in 10 seconds\u0027)"},{"line_number":68,"context_line":"                time.sleep(10)"},{"line_number":69,"context_line":"            except Exception as ex:"},{"line_number":70,"context_line":"                LOG.exception(ex)"},{"line_number":71,"context_line":"                # Initialize the serie builders to v0_11_0. Not sure when SHOW DIAGNOSTICS added"}],"source_content_type":"text/x-python","patch_set":2,"id":"df87a7cf_63a74c51","line":68,"in_reply_to":"df87a7cf_c3b658fb","updated":"2017-12-06 14:29:19.000000000","message":"How about sleeping 10s and then exiting with an error status code upon timeout? Otherwise monasca-api will restart a lot, potentially triggering a \"respawning too fast\" safety mechanism systemd, Kubernetes or Apache (depending on where it\u0027s run). For if there\u0027s a nontrivial number of agents using that monasca-api instance it is going to restart at a very high frequency.","commit_id":"518c64706b1489b6d28338dbe2481b819b4967ab"},{"author":{"_account_id":13478,"name":"Boris Bobrov","email":"b.bobrov@sap.com","username":"bbobrov"},"change_message_id":"d3575dd7366c3b654a9f96cb5d66e06a184f33c9","unresolved":false,"context_lines":[{"line_number":855,"context_line":"                time_clause +\u003d \" and time \u003e\u003d \" + str(int(start_timestamp *"},{"line_number":856,"context_line":"                                                         1000000)) + \"u \""},{"line_number":857,"context_line":""},{"line_number":858,"context_line":"            0if end_timestamp:"},{"line_number":859,"context_line":"                time_clause +\u003d \" and time \u003c\u003d \" + str(int(end_timestamp *"},{"line_number":860,"context_line":"                                                         1000000)) + \"u \""},{"line_number":861,"context_line":""}],"source_content_type":"text/x-python","patch_set":4,"id":"df87a7cf_84b35b98","line":858,"updated":"2017-12-07 13:47:47.000000000","message":"oh wow, how","commit_id":"8b4151b9347161ebc44b2b1f1e60978a7e798e4f"}]}