)]}'
{"oslo_cache/_opts.py":[{"author":{"_account_id":28522,"name":"Hervé Beraud","email":"herveberaud.pro@gmail.com","username":"hberaud"},"change_message_id":"6ef101f2c55b324618a6083ec17e7adcf0527c21","unresolved":false,"context_lines":[{"line_number":111,"context_line":"                   help\u003d\u0027Number of seconds that an operation will wait to get \u0027"},{"line_number":112,"context_line":"                        \u0027a memcache client connection.\u0027),"},{"line_number":113,"context_line":"        cfg.BoolOpt(\u0027memcache_pool_flush_on_reconnect\u0027,"},{"line_number":114,"context_line":"                    default\u003dFalse,"},{"line_number":115,"context_line":"                    help\u003d\u0027Global toggle if memcache will be flushed\u0027"},{"line_number":116,"context_line":"                    \u0027 on reconnect.\u0027"},{"line_number":117,"context_line":"                    \u0027 (oslo_cache.memcache_pool backend only).\u0027),"}],"source_content_type":"text/x-python","patch_set":2,"id":"bf51134e_d55a4e81","line":114,"range":{"start_line":114,"start_character":28,"end_line":114,"end_character":33},"updated":"2020-07-22 09:59:31.000000000","message":"I wondering if it should set to `True` instead, to keep the original behavior as it was before these changes and avoid side effects on user side especially with users who are not aware of these changes and not impacted by this bug or also users who expects that `flush` will happen.\n\nI personally prefer sitting me on the other side of the table, I mean not `flush` only when this really requirered (like in your case).\n\nI know that the goal of these changes is to make this behavior optional, but it could be seen as the opposite too.\n\nIn your case you could fix your issue by passing false to this option in your config, and other users will not see difference on their side.\n\nit would be interesting to debate a bit about this point. I want grab more feedbacks from other core developers especially from Moises.\n\nIn one hand we fix a bug and on the other hand we reintroduce an old bug, so I think it\u0027s worth to speak a bit about it.\n\nThoughts?","commit_id":"e43817046cd2174be28c4f02c7721890bc247dd3"},{"author":{"_account_id":28522,"name":"Hervé Beraud","email":"herveberaud.pro@gmail.com","username":"hberaud"},"change_message_id":"615d2ee856abcfbcba8b6285a26d785bb1560dd2","unresolved":false,"context_lines":[{"line_number":111,"context_line":"                   help\u003d\u0027Number of seconds that an operation will wait to get \u0027"},{"line_number":112,"context_line":"                        \u0027a memcache client connection.\u0027),"},{"line_number":113,"context_line":"        cfg.BoolOpt(\u0027memcache_pool_flush_on_reconnect\u0027,"},{"line_number":114,"context_line":"                    default\u003dFalse,"},{"line_number":115,"context_line":"                    help\u003d\u0027Global toggle if memcache will be flushed\u0027"},{"line_number":116,"context_line":"                    \u0027 on reconnect.\u0027"},{"line_number":117,"context_line":"                    \u0027 (oslo_cache.memcache_pool backend only).\u0027),"}],"source_content_type":"text/x-python","patch_set":2,"id":"bf51134e_062014f7","line":114,"range":{"start_line":114,"start_character":28,"end_line":114,"end_character":33},"in_reply_to":"bf51134e_06abf474","updated":"2020-07-22 16:23:54.000000000","message":"In the case we decide to fix the root cause then I think we could either turn the default option value to True or revert these changes and remove config, but not convinced that back and forth with options are a good idea in a user point of view, I mean it could disturb users and operators to add and remove options every 5 minutes and so it could be preferable to keep the option even if python-memcached will be fixed.","commit_id":"e43817046cd2174be28c4f02c7721890bc247dd3"},{"author":{"_account_id":28522,"name":"Hervé Beraud","email":"herveberaud.pro@gmail.com","username":"hberaud"},"change_message_id":"9d0d56723bf6e028648fefcaa301ebef907ed013","unresolved":false,"context_lines":[{"line_number":111,"context_line":"                   help\u003d\u0027Number of seconds that an operation will wait to get \u0027"},{"line_number":112,"context_line":"                        \u0027a memcache client connection.\u0027),"},{"line_number":113,"context_line":"        cfg.BoolOpt(\u0027memcache_pool_flush_on_reconnect\u0027,"},{"line_number":114,"context_line":"                    default\u003dFalse,"},{"line_number":115,"context_line":"                    help\u003d\u0027Global toggle if memcache will be flushed\u0027"},{"line_number":116,"context_line":"                    \u0027 on reconnect.\u0027"},{"line_number":117,"context_line":"                    \u0027 (oslo_cache.memcache_pool backend only).\u0027),"}],"source_content_type":"text/x-python","patch_set":2,"id":"bf51134e_268b981e","line":114,"range":{"start_line":114,"start_character":28,"end_line":114,"end_character":33},"in_reply_to":"bf51134e_1b498d5f","updated":"2020-07-22 16:16:09.000000000","message":"Anyway in all the case we risk to face an issue (the stale data or the rally), and I think you are right and I think that we should give priority to the race as this is a scenario more easily to observe and as the flush on reconnect is mostly dedicated to keystone first. So in other words I agree with you and I\u0027m OK to continue with the current behavior.\n\nAlso it could be intesting to dive deeper in python-memcached to seen the root cause of this issue, I mean oslo.cache is based on python-memcached and connections are managed by him, so in this case I think the root cause is hidden there.\n\nI think this patch is a good workaround by waiting to find more clues about the root cause.\n\nAlso I need to continue to work on the migration from python-memcached to pymemcache [1], which could help us to solve this issue, as pymemcache handle connections in a different way.\n\nThanks for your feedback and your opinion on this topic :)\n\n[1] https://review.opendev.org/#/c/711220/","commit_id":"e43817046cd2174be28c4f02c7721890bc247dd3"},{"author":{"_account_id":28522,"name":"Hervé Beraud","email":"herveberaud.pro@gmail.com","username":"hberaud"},"change_message_id":"4436b0e45f9845a5f452877c815ddebf9a4f33aa","unresolved":false,"context_lines":[{"line_number":111,"context_line":"                   help\u003d\u0027Number of seconds that an operation will wait to get \u0027"},{"line_number":112,"context_line":"                        \u0027a memcache client connection.\u0027),"},{"line_number":113,"context_line":"        cfg.BoolOpt(\u0027memcache_pool_flush_on_reconnect\u0027,"},{"line_number":114,"context_line":"                    default\u003dFalse,"},{"line_number":115,"context_line":"                    help\u003d\u0027Global toggle if memcache will be flushed\u0027"},{"line_number":116,"context_line":"                    \u0027 on reconnect.\u0027"},{"line_number":117,"context_line":"                    \u0027 (oslo_cache.memcache_pool backend only).\u0027),"}],"source_content_type":"text/x-python","patch_set":2,"id":"bf51134e_06abf474","line":114,"range":{"start_line":114,"start_character":28,"end_line":114,"end_character":33},"in_reply_to":"bf51134e_268b981e","updated":"2020-07-22 16:18:22.000000000","message":"\u003e Anyway in all the case we risk to face an issue (the stale data or\n \u003e the rally), and I think you are right and I think that we should\n \u003e give priority to the race as this is a scenario more easily to\n \u003e observe and as the flush on reconnect is mostly dedicated to\n \u003e keystone first. So in other words I agree with you and I\u0027m OK to\n \u003e continue with the current behavior.\n \u003e \n \u003e Also it could be intesting to dive deeper in python-memcached to\n \u003e seen the root cause of this issue, I mean oslo.cache is based on\n \u003e python-memcached and connections are managed by him, so in this\n \u003e case I think the root cause is hidden there.\n \u003e \n \u003e I think this patch is a good workaround by waiting to find more\n \u003e clues about the root cause.\n \u003e \n \u003e Also I need to continue to work on the migration from\n \u003e python-memcached to pymemcache [1], which could help us to solve\n \u003e this issue, as pymemcache handle connections in a different way.\n\npymemcache doesn\u0027t need to flush on reconnect and so the both bugs doesn\u0027t exist with pymemcache.\n\n \u003e \n \u003e Thanks for your feedback and your opinion on this topic :)\n \u003e \n \u003e [1] https://review.opendev.org/#/c/711220/","commit_id":"e43817046cd2174be28c4f02c7721890bc247dd3"},{"author":{"_account_id":27339,"name":"Michal Arbet","email":"michal.arbet@ultimum.io","username":"michalarbet"},"change_message_id":"c8c17a7acaecde1b5ee13577e0b1a9205e8fe707","unresolved":false,"context_lines":[{"line_number":111,"context_line":"                   help\u003d\u0027Number of seconds that an operation will wait to get \u0027"},{"line_number":112,"context_line":"                        \u0027a memcache client connection.\u0027),"},{"line_number":113,"context_line":"        cfg.BoolOpt(\u0027memcache_pool_flush_on_reconnect\u0027,"},{"line_number":114,"context_line":"                    default\u003dFalse,"},{"line_number":115,"context_line":"                    help\u003d\u0027Global toggle if memcache will be flushed\u0027"},{"line_number":116,"context_line":"                    \u0027 on reconnect.\u0027"},{"line_number":117,"context_line":"                    \u0027 (oslo_cache.memcache_pool backend only).\u0027),"}],"source_content_type":"text/x-python","patch_set":2,"id":"bf51134e_1b498d5f","line":114,"range":{"start_line":114,"start_character":28,"end_line":114,"end_character":33},"in_reply_to":"bf51134e_d55a4e81","updated":"2020-07-22 15:26:32.000000000","message":"I don\u0027t agree : \n\n1. Switch default value to True - original behaviour was to have memcache_pool_connection_get_timeout turned off (turned on only in master branch) always\n2. Avoid side effects on user side - Well, if we switch default value from False to True directly in oslo.cache, users will have issues\n3. Flush_on_reconnect is optional arg, so we should provide way to change this option, by default it is False also in python-memcached\n\n4. If we consider that this option can make openstack unresponsive because dead caches (very easily - just try to run rally against OS , and you will see), I think this option should be False be default.","commit_id":"e43817046cd2174be28c4f02c7721890bc247dd3"},{"author":{"_account_id":6928,"name":"Ben Nemec","email":"openstack@nemebean.com","username":"bnemec"},"change_message_id":"ef8352bb1fd05ca5ddb8e705bf4ba149a40224ea","unresolved":false,"context_lines":[{"line_number":110,"context_line":"                   default\u003d10,"},{"line_number":111,"context_line":"                   help\u003d\u0027Number of seconds that an operation will wait to get \u0027"},{"line_number":112,"context_line":"                        \u0027a memcache client connection.\u0027),"},{"line_number":113,"context_line":"        cfg.BoolOpt(\u0027memcache_pool_flush_on_reconnect\u0027,"},{"line_number":114,"context_line":"                    default\u003dFalse,"},{"line_number":115,"context_line":"                    help\u003d\u0027Global toggle if memcache will be flushed\u0027"},{"line_number":116,"context_line":"                    \u0027 on reconnect.\u0027"}],"source_content_type":"text/x-python","patch_set":9,"id":"9f560f44_b7cc5289","line":113,"updated":"2020-08-10 15:41:48.000000000","message":"I think we need to expose this with a set_defaults API for this library that allows Keystone to override the default. See https://opendev.org/openstack/oslo.policy/src/commit/a626ad12fe5a3abd49d70e3e5b95589d279ab578/oslo_policy/opts.py#L121 for example.\n\nThen we\u0027ll need a followup for Keystone to opt in to this behavior by default.","commit_id":"d9a9a0ef53dc4cdb8e0541d86f5881040c1ff91a"},{"author":{"_account_id":6928,"name":"Ben Nemec","email":"openstack@nemebean.com","username":"bnemec"},"change_message_id":"ef8352bb1fd05ca5ddb8e705bf4ba149a40224ea","unresolved":false,"context_lines":[{"line_number":112,"context_line":"                        \u0027a memcache client connection.\u0027),"},{"line_number":113,"context_line":"        cfg.BoolOpt(\u0027memcache_pool_flush_on_reconnect\u0027,"},{"line_number":114,"context_line":"                    default\u003dFalse,"},{"line_number":115,"context_line":"                    help\u003d\u0027Global toggle if memcache will be flushed\u0027"},{"line_number":116,"context_line":"                    \u0027 on reconnect.\u0027"},{"line_number":117,"context_line":"                    \u0027 (oslo_cache.memcache_pool backend only).\u0027),"},{"line_number":118,"context_line":"        cfg.BoolOpt(\u0027tls_enabled\u0027,"}],"source_content_type":"text/x-python","patch_set":9,"id":"9f560f44_d7e8e6e2","line":115,"updated":"2020-08-10 15:41:48.000000000","message":"I would like to see some discussion of the potential security implications of turning this off.","commit_id":"d9a9a0ef53dc4cdb8e0541d86f5881040c1ff91a"}],"oslo_cache/core.py":[{"author":{"_account_id":5046,"name":"Lance Bragstad","email":"lbragstad@redhat.com","username":"ldbragst"},"change_message_id":"85d7d13fb587969a1ef3bc71d9755addf079fa62","unresolved":false,"context_lines":[{"line_number":140,"context_line":"    # backends. Use setdefault for url to support old-style setting through"},{"line_number":141,"context_line":"    # backend_argument\u003durl:127.0.0.1:11211"},{"line_number":142,"context_line":"    #"},{"line_number":143,"context_line":"    # NOTE(morgan): Explicitly set flush_on_reconnect for pooled"},{"line_number":144,"context_line":"    # connections. This should ensure that stale data is never consumed"},{"line_number":145,"context_line":"    # from a server that pops in/out due to a network partition"},{"line_number":146,"context_line":"    # or disconnect."},{"line_number":147,"context_line":"    #"},{"line_number":148,"context_line":"    # See the help from python-memcached:"},{"line_number":149,"context_line":"    #"}],"source_content_type":"text/x-python","patch_set":8,"id":"9f560f44_280d603e","line":146,"range":{"start_line":143,"start_character":0,"end_line":146,"end_character":20},"updated":"2020-07-27 16:08:42.000000000","message":"This comment isn\u0027t consistent with the new default.\n\nThis information would be great to have in the configuration help text since it describes how the option works and the behavior.","commit_id":"54dcb281ed343d6c2db0cf3a0ca04e7b59deb290"},{"author":{"_account_id":27339,"name":"Michal Arbet","email":"michal.arbet@ultimum.io","username":"michalarbet"},"change_message_id":"41e26a8e3c552564fcd08a743cf3b7e593a287ce","unresolved":false,"context_lines":[{"line_number":140,"context_line":"    # backends. Use setdefault for url to support old-style setting through"},{"line_number":141,"context_line":"    # backend_argument\u003durl:127.0.0.1:11211"},{"line_number":142,"context_line":"    #"},{"line_number":143,"context_line":"    # NOTE(morgan): Explicitly set flush_on_reconnect for pooled"},{"line_number":144,"context_line":"    # connections. This should ensure that stale data is never consumed"},{"line_number":145,"context_line":"    # from a server that pops in/out due to a network partition"},{"line_number":146,"context_line":"    # or disconnect."},{"line_number":147,"context_line":"    #"},{"line_number":148,"context_line":"    # See the help from python-memcached:"},{"line_number":149,"context_line":"    #"}],"source_content_type":"text/x-python","patch_set":8,"id":"9f560f44_863bbd9a","line":146,"range":{"start_line":143,"start_character":0,"end_line":146,"end_character":20},"in_reply_to":"9f560f44_280d603e","updated":"2020-07-27 17:20:42.000000000","message":"Yes, you are right, I\u0027ve just copied this text from original change from \"morgan\" , I will add it to config help.","commit_id":"54dcb281ed343d6c2db0cf3a0ca04e7b59deb290"},{"author":{"_account_id":6928,"name":"Ben Nemec","email":"openstack@nemebean.com","username":"bnemec"},"change_message_id":"702edb7acb143acb68cd23970167202ebd7c31a5","unresolved":true,"context_lines":[{"line_number":140,"context_line":"    # backends. Use setdefault for url to support old-style setting through"},{"line_number":141,"context_line":"    # backend_argument\u003durl:127.0.0.1:11211"},{"line_number":142,"context_line":"    #"},{"line_number":143,"context_line":"    # NOTE(morgan): Explicitly set flush_on_reconnect for pooled"},{"line_number":144,"context_line":"    # connections. This should ensure that stale data is never consumed"},{"line_number":145,"context_line":"    # from a server that pops in/out due to a network partition"},{"line_number":146,"context_line":"    # or disconnect."}],"source_content_type":"text/x-python","patch_set":12,"id":"16d0154f_1a7879d6","line":143,"updated":"2021-01-25 16:26:57.000000000","message":"This comment is a little bit confusing in this context since we\u0027re no longer always setting it (and in fact are defaulting to not set it). It might be nice to add a note somewhere in here that links to the bug and explain that we\u0027re conditionally setting the option now.","commit_id":"a437b219acbf3ad092ecfb6f293a3eee133ec755"},{"author":{"_account_id":28522,"name":"Hervé Beraud","email":"herveberaud.pro@gmail.com","username":"hberaud"},"change_message_id":"34ebaf9eb6080dc04156a68871f45ce9776882b0","unresolved":true,"context_lines":[{"line_number":140,"context_line":"    # backends. Use setdefault for url to support old-style setting through"},{"line_number":141,"context_line":"    # backend_argument\u003durl:127.0.0.1:11211"},{"line_number":142,"context_line":"    #"},{"line_number":143,"context_line":"    # NOTE(morgan): Explicitly set flush_on_reconnect for pooled"},{"line_number":144,"context_line":"    # connections. This should ensure that stale data is never consumed"},{"line_number":145,"context_line":"    # from a server that pops in/out due to a network partition"},{"line_number":146,"context_line":"    # or disconnect."}],"source_content_type":"text/x-python","patch_set":12,"id":"7e9cc1ce_b721ee85","line":143,"in_reply_to":"16d0154f_1a7879d6","updated":"2021-01-25 16:31:01.000000000","message":"Yes you\u0027re right I\u0027ll address that through a follow up patch","commit_id":"a437b219acbf3ad092ecfb6f293a3eee133ec755"}],"releasenotes/notes/bug-1888394-5a53e7a9cb25375b.yaml":[{"author":{"_account_id":28522,"name":"Hervé Beraud","email":"herveberaud.pro@gmail.com","username":"hberaud"},"change_message_id":"6ef101f2c55b324618a6083ec17e7adcf0527c21","unresolved":false,"context_lines":[{"line_number":1,"context_line":"---"},{"line_number":2,"context_line":"fixes:"},{"line_number":3,"context_line":"  - |"},{"line_number":4,"context_line":"    [`bug 1888394 \u003chttps://bugs.launchpad.net/oslo.cache/+bug/1888394\u003e`_]"},{"line_number":5,"context_line":"    If a memcache server disappears and then reconnects when multiple memcache"}],"source_content_type":"text/x-yaml","patch_set":2,"id":"bf51134e_15d966f3","line":2,"range":{"start_line":2,"start_character":0,"end_line":2,"end_character":5},"updated":"2020-07-22 09:59:31.000000000","message":"I wondering if we should  add a `features` section too to more highlight the new option.","commit_id":"e43817046cd2174be28c4f02c7721890bc247dd3"},{"author":{"_account_id":28522,"name":"Hervé Beraud","email":"herveberaud.pro@gmail.com","username":"hberaud"},"change_message_id":"2eb827e15ec8eb6298a2c486d98debfe6556331b","unresolved":false,"context_lines":[{"line_number":18,"context_line":"    This change is moving optional parameter `flush_on_reconnect` to oslo.cache config."},{"line_number":19,"context_line":""},{"line_number":20,"context_line":"features:"},{"line_number":21,"context_line":"  - Configuration option ``flush_on_reconnectt`` added to control"},{"line_number":22,"context_line":"    if flush will be send to memcached server after reconnect."}],"source_content_type":"text/x-yaml","patch_set":3,"id":"bf51134e_db10b552","line":21,"range":{"start_line":21,"start_character":27,"end_line":21,"end_character":45},"updated":"2020-07-22 15:23:53.000000000","message":"isn\u0027t `memcache_pool_flush_on_reconnect`?","commit_id":"cadee925dceb54c1f93c36f37eb2511f48e757b1"},{"author":{"_account_id":28522,"name":"Hervé Beraud","email":"herveberaud.pro@gmail.com","username":"hberaud"},"change_message_id":"2eb827e15ec8eb6298a2c486d98debfe6556331b","unresolved":false,"context_lines":[{"line_number":18,"context_line":"    This change is moving optional parameter `flush_on_reconnect` to oslo.cache config."},{"line_number":19,"context_line":""},{"line_number":20,"context_line":"features:"},{"line_number":21,"context_line":"  - Configuration option ``flush_on_reconnectt`` added to control"},{"line_number":22,"context_line":"    if flush will be send to memcached server after reconnect."}],"source_content_type":"text/x-yaml","patch_set":3,"id":"bf51134e_f8277fc6","line":21,"range":{"start_line":21,"start_character":45,"end_line":21,"end_character":46},"updated":"2020-07-22 15:23:53.000000000","message":"nits","commit_id":"cadee925dceb54c1f93c36f37eb2511f48e757b1"},{"author":{"_account_id":27339,"name":"Michal Arbet","email":"michal.arbet@ultimum.io","username":"michalarbet"},"change_message_id":"450168e002ea9b69d54d16b16bf5f890c7676954","unresolved":false,"context_lines":[{"line_number":18,"context_line":"    This change is moving optional parameter `flush_on_reconnect` to oslo.cache config."},{"line_number":19,"context_line":""},{"line_number":20,"context_line":"features:"},{"line_number":21,"context_line":"  - Configuration option ``flush_on_reconnectt`` added to control"},{"line_number":22,"context_line":"    if flush will be send to memcached server after reconnect."}],"source_content_type":"text/x-yaml","patch_set":3,"id":"bf51134e_7b424911","line":21,"range":{"start_line":21,"start_character":27,"end_line":21,"end_character":45},"in_reply_to":"bf51134e_db10b552","updated":"2020-07-22 15:31:56.000000000","message":"Done","commit_id":"cadee925dceb54c1f93c36f37eb2511f48e757b1"},{"author":{"_account_id":28522,"name":"Hervé Beraud","email":"herveberaud.pro@gmail.com","username":"hberaud"},"change_message_id":"66f3bf1757dbf078fe25ea29ad4e474f969421cb","unresolved":false,"context_lines":[{"line_number":18,"context_line":"    This change is moving optional parameter `flush_on_reconnect` to oslo.cache config."},{"line_number":19,"context_line":""},{"line_number":20,"context_line":"features:"},{"line_number":21,"context_line":"  - Configuration option ``memcache_pool_flush_on_reconnectt`` added to control"},{"line_number":22,"context_line":"    if flush will be send to memcached server after reconnect."}],"source_content_type":"text/x-yaml","patch_set":4,"id":"bf51134e_267718a2","line":21,"range":{"start_line":21,"start_character":59,"end_line":21,"end_character":60},"updated":"2020-07-22 16:01:07.000000000","message":"Sorry I forgot the save my previous comment about this typo.","commit_id":"f9623853c3e38900308b3b4155fbb44ad9fbc4fb"},{"author":{"_account_id":28522,"name":"Hervé Beraud","email":"herveberaud.pro@gmail.com","username":"hberaud"},"change_message_id":"446d39df41d351f93a4845d486d615ead42f763a","unresolved":false,"context_lines":[{"line_number":18,"context_line":"    This change is moving optional parameter `flush_on_reconnect` to oslo.cache config."},{"line_number":19,"context_line":""},{"line_number":20,"context_line":"features:"},{"line_number":21,"context_line":"  - Configuration option ``memcache_pool_flush_on_reconnectt`` added to control"},{"line_number":22,"context_line":"    if flush will be send to memcached server after reconnect."}],"source_content_type":"text/x-yaml","patch_set":4,"id":"bf51134e_06901495","line":21,"range":{"start_line":21,"start_character":59,"end_line":21,"end_character":60},"in_reply_to":"bf51134e_267718a2","updated":"2020-07-22 16:01:48.000000000","message":"\u003e Sorry I forgot the save my previous comment about this typo.\n\ns/the save/to save/","commit_id":"f9623853c3e38900308b3b4155fbb44ad9fbc4fb"},{"author":{"_account_id":28522,"name":"Hervé Beraud","email":"herveberaud.pro@gmail.com","username":"hberaud"},"change_message_id":"4ea765c859a598f22ddaac1007cfb61bf7307360","unresolved":false,"context_lines":[{"line_number":18,"context_line":"    This change is moving optional parameter `flush_on_reconnect` to oslo.cache config."},{"line_number":19,"context_line":""},{"line_number":20,"context_line":"features:"},{"line_number":21,"context_line":"  - Configuration option ``memcache_pool_flush_on_reconnectt`` added to control"},{"line_number":22,"context_line":"    if flush will be send to memcached server after reconnect."}],"source_content_type":"text/x-yaml","patch_set":6,"id":"9f560f44_18742f35","line":21,"range":{"start_line":21,"start_character":59,"end_line":21,"end_character":60},"updated":"2020-07-27 11:23:17.000000000","message":"Typo be back","commit_id":"a4b1f24e45a58f7f705f31c3b641ac4bb64e5b23"},{"author":{"_account_id":27339,"name":"Michal Arbet","email":"michal.arbet@ultimum.io","username":"michalarbet"},"change_message_id":"7fc08b52b037224e40c50e23a5b626180e497b33","unresolved":false,"context_lines":[{"line_number":18,"context_line":"    This change is moving optional parameter `flush_on_reconnect` to oslo.cache config."},{"line_number":19,"context_line":""},{"line_number":20,"context_line":"features:"},{"line_number":21,"context_line":"  - Configuration option ``memcache_pool_flush_on_reconnectt`` added to control"},{"line_number":22,"context_line":"    if flush will be send to memcached server after reconnect."}],"source_content_type":"text/x-yaml","patch_set":6,"id":"9f560f44_23a9087b","line":21,"range":{"start_line":21,"start_character":59,"end_line":21,"end_character":60},"in_reply_to":"9f560f44_18742f35","updated":"2020-07-27 12:21:17.000000000","message":"Done","commit_id":"a4b1f24e45a58f7f705f31c3b641ac4bb64e5b23"},{"author":{"_account_id":11904,"name":"Sean McGinnis","email":"sean.mcginnis@gmail.com","username":"SeanM"},"change_message_id":"a1a4c7db18f53bad3c9f9cb37539dfffbeb848c9","unresolved":false,"context_lines":[{"line_number":5,"context_line":"    If a memcache server disappears and then reconnects when multiple memcache"},{"line_number":6,"context_line":"    servers are used (specific to the python-memcached based backends) it is"},{"line_number":7,"context_line":"    possible that the server will contain stale data. To avoid this, param"},{"line_number":8,"context_line":"    flush_on_reconnect was used in a code."},{"line_number":9,"context_line":""},{"line_number":10,"context_line":"    But unfortunatelly this option is causing another issue."},{"line_number":11,"context_line":"    If memcache server disappears, or client had broken connection to memcache server,"}],"source_content_type":"text/x-yaml","patch_set":7,"id":"9f560f44_48fff4a9","line":8,"range":{"start_line":8,"start_character":32,"end_line":8,"end_character":41},"updated":"2020-07-27 15:27:26.000000000","message":"in code","commit_id":"bd229ad7bc64f0b50611513f69d4e6b700147307"},{"author":{"_account_id":27339,"name":"Michal Arbet","email":"michal.arbet@ultimum.io","username":"michalarbet"},"change_message_id":"23f003b7e16350509606e9829fe62fa39f0b5d8e","unresolved":false,"context_lines":[{"line_number":5,"context_line":"    If a memcache server disappears and then reconnects when multiple memcache"},{"line_number":6,"context_line":"    servers are used (specific to the python-memcached based backends) it is"},{"line_number":7,"context_line":"    possible that the server will contain stale data. To avoid this, param"},{"line_number":8,"context_line":"    flush_on_reconnect was used in a code."},{"line_number":9,"context_line":""},{"line_number":10,"context_line":"    But unfortunatelly this option is causing another issue."},{"line_number":11,"context_line":"    If memcache server disappears, or client had broken connection to memcache server,"}],"source_content_type":"text/x-yaml","patch_set":7,"id":"9f560f44_48c6d469","line":8,"range":{"start_line":8,"start_character":32,"end_line":8,"end_character":41},"in_reply_to":"9f560f44_48fff4a9","updated":"2020-07-27 15:30:46.000000000","message":"Done","commit_id":"bd229ad7bc64f0b50611513f69d4e6b700147307"},{"author":{"_account_id":11904,"name":"Sean McGinnis","email":"sean.mcginnis@gmail.com","username":"SeanM"},"change_message_id":"a1a4c7db18f53bad3c9f9cb37539dfffbeb848c9","unresolved":false,"context_lines":[{"line_number":12,"context_line":"    or whatever, clients start to flush server on reconnect."},{"line_number":13,"context_line":""},{"line_number":14,"context_line":"    This means that network connections will go UP and can cause server to be overloaded"},{"line_number":15,"context_line":"    until memcache will be unresponsible."},{"line_number":16,"context_line":""},{"line_number":17,"context_line":"    Simply said this option can cause loop of flushs and overloaded memcached servers."},{"line_number":18,"context_line":"    This change is moving optional parameter `flush_on_reconnect` to oslo.cache config."}],"source_content_type":"text/x-yaml","patch_set":7,"id":"9f560f44_6802b8b6","line":15,"range":{"start_line":15,"start_character":27,"end_line":15,"end_character":40},"updated":"2020-07-27 15:27:26.000000000","message":"responsive?","commit_id":"bd229ad7bc64f0b50611513f69d4e6b700147307"},{"author":{"_account_id":27339,"name":"Michal Arbet","email":"michal.arbet@ultimum.io","username":"michalarbet"},"change_message_id":"23f003b7e16350509606e9829fe62fa39f0b5d8e","unresolved":false,"context_lines":[{"line_number":12,"context_line":"    or whatever, clients start to flush server on reconnect."},{"line_number":13,"context_line":""},{"line_number":14,"context_line":"    This means that network connections will go UP and can cause server to be overloaded"},{"line_number":15,"context_line":"    until memcache will be unresponsible."},{"line_number":16,"context_line":""},{"line_number":17,"context_line":"    Simply said this option can cause loop of flushs and overloaded memcached servers."},{"line_number":18,"context_line":"    This change is moving optional parameter `flush_on_reconnect` to oslo.cache config."}],"source_content_type":"text/x-yaml","patch_set":7,"id":"9f560f44_a8a050a7","line":15,"range":{"start_line":15,"start_character":27,"end_line":15,"end_character":40},"in_reply_to":"9f560f44_6802b8b6","updated":"2020-07-27 15:30:46.000000000","message":"Done","commit_id":"bd229ad7bc64f0b50611513f69d4e6b700147307"},{"author":{"_account_id":5046,"name":"Lance Bragstad","email":"lbragstad@redhat.com","username":"ldbragst"},"change_message_id":"85d7d13fb587969a1ef3bc71d9755addf079fa62","unresolved":false,"context_lines":[{"line_number":5,"context_line":"    If a memcache server disappears and then reconnects when multiple memcache"},{"line_number":6,"context_line":"    servers are used (specific to the python-memcached based backends) it is"},{"line_number":7,"context_line":"    possible that the server will contain stale data. To avoid this, param"},{"line_number":8,"context_line":"    flush_on_reconnect was used in a code."},{"line_number":9,"context_line":""},{"line_number":10,"context_line":"    But unfortunatelly this option is causing another issue."},{"line_number":11,"context_line":"    If memcache server disappears, or client had broken connection to memcache server,"}],"source_content_type":"text/x-yaml","patch_set":8,"id":"9f560f44_c87144e3","line":8,"range":{"start_line":8,"start_character":35,"end_line":8,"end_character":36},"updated":"2020-07-27 16:08:42.000000000","message":"nit: remove","commit_id":"54dcb281ed343d6c2db0cf3a0ca04e7b59deb290"},{"author":{"_account_id":27339,"name":"Michal Arbet","email":"michal.arbet@ultimum.io","username":"michalarbet"},"change_message_id":"41e26a8e3c552564fcd08a743cf3b7e593a287ce","unresolved":false,"context_lines":[{"line_number":5,"context_line":"    If a memcache server disappears and then reconnects when multiple memcache"},{"line_number":6,"context_line":"    servers are used (specific to the python-memcached based backends) it is"},{"line_number":7,"context_line":"    possible that the server will contain stale data. To avoid this, param"},{"line_number":8,"context_line":"    flush_on_reconnect was used in a code."},{"line_number":9,"context_line":""},{"line_number":10,"context_line":"    But unfortunatelly this option is causing another issue."},{"line_number":11,"context_line":"    If memcache server disappears, or client had broken connection to memcache server,"}],"source_content_type":"text/x-yaml","patch_set":8,"id":"9f560f44_a689a127","line":8,"range":{"start_line":8,"start_character":35,"end_line":8,"end_character":36},"in_reply_to":"9f560f44_c87144e3","updated":"2020-07-27 17:20:42.000000000","message":"Done","commit_id":"54dcb281ed343d6c2db0cf3a0ca04e7b59deb290"},{"author":{"_account_id":5046,"name":"Lance Bragstad","email":"lbragstad@redhat.com","username":"ldbragst"},"change_message_id":"85d7d13fb587969a1ef3bc71d9755addf079fa62","unresolved":false,"context_lines":[{"line_number":9,"context_line":""},{"line_number":10,"context_line":"    But unfortunatelly this option is causing another issue."},{"line_number":11,"context_line":"    If memcache server disappears, or client had broken connection to memcache server,"},{"line_number":12,"context_line":"    or whatever, clients start to flush server on reconnect."},{"line_number":13,"context_line":""},{"line_number":14,"context_line":"    This means that network connections will go UP and can cause server to be overloaded"},{"line_number":15,"context_line":"    until memcache will be unresponsible."}],"source_content_type":"text/x-yaml","patch_set":8,"id":"9f560f44_48a8941e","line":12,"range":{"start_line":12,"start_character":4,"end_line":12,"end_character":16},"updated":"2020-07-27 16:08:42.000000000","message":"nit: I suggest removing this since it\u0027s ambiguous.","commit_id":"54dcb281ed343d6c2db0cf3a0ca04e7b59deb290"},{"author":{"_account_id":27339,"name":"Michal Arbet","email":"michal.arbet@ultimum.io","username":"michalarbet"},"change_message_id":"41e26a8e3c552564fcd08a743cf3b7e593a287ce","unresolved":false,"context_lines":[{"line_number":9,"context_line":""},{"line_number":10,"context_line":"    But unfortunatelly this option is causing another issue."},{"line_number":11,"context_line":"    If memcache server disappears, or client had broken connection to memcache server,"},{"line_number":12,"context_line":"    or whatever, clients start to flush server on reconnect."},{"line_number":13,"context_line":""},{"line_number":14,"context_line":"    This means that network connections will go UP and can cause server to be overloaded"},{"line_number":15,"context_line":"    until memcache will be unresponsible."}],"source_content_type":"text/x-yaml","patch_set":8,"id":"9f560f44_868e1d2c","line":12,"range":{"start_line":12,"start_character":4,"end_line":12,"end_character":16},"in_reply_to":"9f560f44_48a8941e","updated":"2020-07-27 17:20:42.000000000","message":"Done","commit_id":"54dcb281ed343d6c2db0cf3a0ca04e7b59deb290"},{"author":{"_account_id":5046,"name":"Lance Bragstad","email":"lbragstad@redhat.com","username":"ldbragst"},"change_message_id":"85d7d13fb587969a1ef3bc71d9755addf079fa62","unresolved":false,"context_lines":[{"line_number":12,"context_line":"    or whatever, clients start to flush server on reconnect."},{"line_number":13,"context_line":""},{"line_number":14,"context_line":"    This means that network connections will go UP and can cause server to be overloaded"},{"line_number":15,"context_line":"    until memcache will be unresponsible."},{"line_number":16,"context_line":""},{"line_number":17,"context_line":"    Simply said this option can cause loop of flushs and overloaded memcached servers."},{"line_number":18,"context_line":"    This change is moving optional parameter `flush_on_reconnect` to oslo.cache config."}],"source_content_type":"text/x-yaml","patch_set":8,"id":"9f560f44_085cdc49","line":15,"updated":"2020-07-27 16:08:42.000000000","message":"Are network connections directly related to flush events? This sentence implies flushing the cache causes network disconnects. Is that the case or am I misreading this?","commit_id":"54dcb281ed343d6c2db0cf3a0ca04e7b59deb290"},{"author":{"_account_id":27339,"name":"Michal Arbet","email":"michal.arbet@ultimum.io","username":"michalarbet"},"change_message_id":"1a0cc397f5b9aaee8f3ba4f539312985dc257f54","unresolved":false,"context_lines":[{"line_number":12,"context_line":"    or whatever, clients start to flush server on reconnect."},{"line_number":13,"context_line":""},{"line_number":14,"context_line":"    This means that network connections will go UP and can cause server to be overloaded"},{"line_number":15,"context_line":"    until memcache will be unresponsible."},{"line_number":16,"context_line":""},{"line_number":17,"context_line":"    Simply said this option can cause loop of flushs and overloaded memcached servers."},{"line_number":18,"context_line":"    This change is moving optional parameter `flush_on_reconnect` to oslo.cache config."}],"source_content_type":"text/x-yaml","patch_set":8,"id":"9f560f44_61b8132a","line":15,"in_reply_to":"9f560f44_085cdc49","updated":"2020-08-03 11:20:22.000000000","message":"Well, simply said yes.\n\nImagine that you have huge requests to cache ( in our deployment it is kubernetes who do it)","commit_id":"54dcb281ed343d6c2db0cf3a0ca04e7b59deb290"},{"author":{"_account_id":27339,"name":"Michal Arbet","email":"michal.arbet@ultimum.io","username":"michalarbet"},"change_message_id":"09a5be24a82072b0f09d71d014d54fd0956aaf1f","unresolved":false,"context_lines":[{"line_number":12,"context_line":"    or whatever, clients start to flush server on reconnect."},{"line_number":13,"context_line":""},{"line_number":14,"context_line":"    This means that network connections will go UP and can cause server to be overloaded"},{"line_number":15,"context_line":"    until memcache will be unresponsible."},{"line_number":16,"context_line":""},{"line_number":17,"context_line":"    Simply said this option can cause loop of flushs and overloaded memcached servers."},{"line_number":18,"context_line":"    This change is moving optional parameter `flush_on_reconnect` to oslo.cache config."}],"source_content_type":"text/x-yaml","patch_set":8,"id":"9f560f44_e63a79e5","line":15,"in_reply_to":"9f560f44_085cdc49","updated":"2020-07-27 17:51:12.000000000","message":"Well, yes \n\nIn real when memcached\u0027s client\u0027s connection is broken , whatewer reason it is ( typically some network issue, restart, etc..) client will connect again to memcached server, but will flush memcached on reconnect.\n\nLet\u0027s imagine that we have two memcached servers.\nImagine that one server was unresponsive.\n\nAfter unresponsive node appear, memcached node is trying to cache data, but clients are still flushing cache because they have flush_on_reconnect turned on, which causing memcached to be extremly overkilled, connections are growing until memcache is dead, so it is unresposive again.\n\nWhen it is unresponsive, clients are connecting to another memcached which can be unresponsive in short time, because all requests to cache is now served only from one memcached server.\n\nWhen first memcached node start to be responsive again, process is repeated, because clients are connecting to it and flushes again, because it was unresponsive.\n\nWe met this bug when upgrading openstack via kolla-ansible.\nProblem appeared when big traffic was going to memcache.\n\nIn Closes-Bug report i simulated flushs with while script to flush cashes, check it , maybe it will be more descriptive for you.\n\nReverted oslo.cache fixed issue immediately.\n\nHope that it is more clear.\n\n\nThis problem is visible only under big load, but it\u0027s definitively issue","commit_id":"54dcb281ed343d6c2db0cf3a0ca04e7b59deb290"},{"author":{"_account_id":22072,"name":"Viktor Křivák","email":"viktor.krivak@ultimum.io","username":"viktor-krivak"},"change_message_id":"001eaa59e324712a71f1f5d0d6ac10f558b9d917","unresolved":false,"context_lines":[{"line_number":12,"context_line":"    or whatever, clients start to flush server on reconnect."},{"line_number":13,"context_line":""},{"line_number":14,"context_line":"    This means that network connections will go UP and can cause server to be overloaded"},{"line_number":15,"context_line":"    until memcache will be unresponsible."},{"line_number":16,"context_line":""},{"line_number":17,"context_line":"    Simply said this option can cause loop of flushs and overloaded memcached servers."},{"line_number":18,"context_line":"    This change is moving optional parameter `flush_on_reconnect` to oslo.cache config."}],"source_content_type":"text/x-yaml","patch_set":8,"id":"9f560f44_21cd695d","line":15,"in_reply_to":"9f560f44_7360e4cf","updated":"2020-07-28 14:31:02.000000000","message":"I think that this flag doesn\u0027t mitigate the problem entirely. If I understand it correctly the original attack vector is:\n- Issue token\n- Somehow stop memcached server that stores this token\n- Revoke token\n- Start memcached server again\n- Token is valid again and can be misused\nBut even with flush_on_reconnect similar issue is still possible:\n- Issue token\n- Somehow stop memcached server that stores this token\n- Wait before someone is use this token again so it is again pushed to the other memcached (token is still valid)\n- Start memcached server\n- Revoke token\n- Stop the original memcached server again\n- Since the other memcache is still holding the old token it is valid again despite the fact that you have flushed the data on the original memcache (that was stopped and started and stopped again)\n\nSo it\u0027s actually doesn\u0027t improve security that much. And add a ton of problem with performance.","commit_id":"54dcb281ed343d6c2db0cf3a0ca04e7b59deb290"},{"author":{"_account_id":5046,"name":"Lance Bragstad","email":"lbragstad@redhat.com","username":"ldbragst"},"change_message_id":"6f9f3093c274e16e89c08ced6f213cd13a26cedc","unresolved":false,"context_lines":[{"line_number":12,"context_line":"    or whatever, clients start to flush server on reconnect."},{"line_number":13,"context_line":""},{"line_number":14,"context_line":"    This means that network connections will go UP and can cause server to be overloaded"},{"line_number":15,"context_line":"    until memcache will be unresponsible."},{"line_number":16,"context_line":""},{"line_number":17,"context_line":"    Simply said this option can cause loop of flushs and overloaded memcached servers."},{"line_number":18,"context_line":"    This change is moving optional parameter `flush_on_reconnect` to oslo.cache config."}],"source_content_type":"text/x-yaml","patch_set":8,"id":"9f560f44_7360e4cf","line":15,"in_reply_to":"9f560f44_e63a79e5","updated":"2020-07-27 20:29:52.000000000","message":"Let me see if I understand this correctly.\n\nThe root cause is a network connectivity problem, but it\u0027s exacerbated because the memecached cluster isn\u0027t big enough, correct?\n\nDid disabling flush_on_reconnect completely fix the issue or did you eventually scale up your memcached deployment? Without scaling up, won\u0027t future network partition events cause the same issue?\n\nMorgan eluded to this in his comment, but my concern about keeping stale data around is primarily due to security. For example, say I authenticate for a token and it\u0027s stored in memcached-1. Sometime between when my token is cached and its expiration, memcache-1 becomes unreachable, at which time my user is disabled, revoking my token. Now, memcached-1 rejoined the cluster, my invalid token is technically cached and still valid according to keystone it if fetches the token from memcache-1, right?","commit_id":"54dcb281ed343d6c2db0cf3a0ca04e7b59deb290"},{"author":{"_account_id":5046,"name":"Lance Bragstad","email":"lbragstad@redhat.com","username":"ldbragst"},"change_message_id":"85d7d13fb587969a1ef3bc71d9755addf079fa62","unresolved":false,"context_lines":[{"line_number":19,"context_line":""},{"line_number":20,"context_line":"features:"},{"line_number":21,"context_line":"  - Configuration option ``memcache_pool_flush_on_reconnect`` added to control"},{"line_number":22,"context_line":"    if flush will be send to memcached server after reconnect."}],"source_content_type":"text/x-yaml","patch_set":8,"id":"9f560f44_4856d42b","line":22,"range":{"start_line":22,"start_character":21,"end_line":22,"end_character":25},"updated":"2020-07-27 16:08:42.000000000","message":"nit: sent","commit_id":"54dcb281ed343d6c2db0cf3a0ca04e7b59deb290"},{"author":{"_account_id":27339,"name":"Michal Arbet","email":"michal.arbet@ultimum.io","username":"michalarbet"},"change_message_id":"41e26a8e3c552564fcd08a743cf3b7e593a287ce","unresolved":false,"context_lines":[{"line_number":19,"context_line":""},{"line_number":20,"context_line":"features:"},{"line_number":21,"context_line":"  - Configuration option ``memcache_pool_flush_on_reconnect`` added to control"},{"line_number":22,"context_line":"    if flush will be send to memcached server after reconnect."}],"source_content_type":"text/x-yaml","patch_set":8,"id":"9f560f44_e6839903","line":22,"range":{"start_line":22,"start_character":21,"end_line":22,"end_character":25},"in_reply_to":"9f560f44_4856d42b","updated":"2020-07-27 17:20:42.000000000","message":"Done","commit_id":"54dcb281ed343d6c2db0cf3a0ca04e7b59deb290"},{"author":{"_account_id":6928,"name":"Ben Nemec","email":"openstack@nemebean.com","username":"bnemec"},"change_message_id":"702edb7acb143acb68cd23970167202ebd7c31a5","unresolved":true,"context_lines":[{"line_number":7,"context_line":"    possible that the server will contain stale data. To avoid this, param"},{"line_number":8,"context_line":"    flush_on_reconnect was used in code."},{"line_number":9,"context_line":""},{"line_number":10,"context_line":"    But unfortunatelly this option is causing another issue."},{"line_number":11,"context_line":"    If memcache server disappears, or client had broken connection to memcache server,"},{"line_number":12,"context_line":"    clients start to flush server on reconnect."},{"line_number":13,"context_line":""}],"source_content_type":"text/x-yaml","patch_set":12,"id":"75bf7ceb_f2044e43","line":10,"range":{"start_line":10,"start_character":8,"end_line":10,"end_character":22},"updated":"2021-01-25 16:26:57.000000000","message":"*unfortunately","commit_id":"a437b219acbf3ad092ecfb6f293a3eee133ec755"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"fa4cc46720a512c41b781aed0de134832ce29e43","unresolved":true,"context_lines":[{"line_number":14,"context_line":"    This means that network connections will go UP and can cause server to be overloaded"},{"line_number":15,"context_line":"    until memcache will be unresponsive."},{"line_number":16,"context_line":""},{"line_number":17,"context_line":"    Simply said this option can cause loop of flushs and overloaded memcached servers."},{"line_number":18,"context_line":"    This change is moving optional parameter `flush_on_reconnect` to oslo.cache config."},{"line_number":19,"context_line":""},{"line_number":20,"context_line":"features:"}],"source_content_type":"text/x-yaml","patch_set":12,"id":"9df4859e_7584fb22","line":17,"range":{"start_line":17,"start_character":46,"end_line":17,"end_character":52},"updated":"2021-02-03 11:21:37.000000000","message":"flushes","commit_id":"a437b219acbf3ad092ecfb6f293a3eee133ec755"},{"author":{"_account_id":15334,"name":"Stephen Finucane","display_name":"stephenfin","email":"stephenfin@redhat.com","username":"sfinucan"},"change_message_id":"fa4cc46720a512c41b781aed0de134832ce29e43","unresolved":true,"context_lines":[{"line_number":15,"context_line":"    until memcache will be unresponsive."},{"line_number":16,"context_line":""},{"line_number":17,"context_line":"    Simply said this option can cause loop of flushs and overloaded memcached servers."},{"line_number":18,"context_line":"    This change is moving optional parameter `flush_on_reconnect` to oslo.cache config."},{"line_number":19,"context_line":""},{"line_number":20,"context_line":"features:"},{"line_number":21,"context_line":"  - Configuration option ``memcache_pool_flush_on_reconnect`` added to control"}],"source_content_type":"text/x-yaml","patch_set":12,"id":"e973c228_ee1f206d","line":18,"range":{"start_line":18,"start_character":45,"end_line":18,"end_character":65},"updated":"2021-02-03 11:21:37.000000000","message":"needs double backticks","commit_id":"a437b219acbf3ad092ecfb6f293a3eee133ec755"}]}