)]}'
{"specs/mitaka/common-transformers-cache.rst":[{"author":{"_account_id":6537,"name":"gordon chung","email":"gord@live.ca","username":"chungg"},"change_message_id":"776ec30cd46ee6916672bf39e3068b06c3f842b4","unresolved":false,"context_lines":[{"line_number":54,"context_line":"It will allow to get rid of additional queues and coordination in general"},{"line_number":55,"context_line":"for notification agents."},{"line_number":56,"context_line":"Note that for central and alarm evaluation (aodh-related service) services"},{"line_number":57,"context_line":"coordination process remains required for scalability."},{"line_number":58,"context_line":""},{"line_number":59,"context_line":"Alternatives"},{"line_number":60,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a8ffd7b_067c8cb3","line":57,"updated":"2015-11-27 15:01:25.000000000","message":"i think we need more details how this works:\n\nsay there is pipeA with two transforms. if agent1 gets res1_sample1 and agent2 gets res1_sample2 and res2_sample1\n\nwhat gets stored to cache? do we have a cache for each pipeline+transformer+transformer_key?\n\nhow do we avoid race and ensure ordering, do we need to lock the cache each time?","commit_id":"ba44b0680b26edc0c8b5d5741814db18a6aa7363"},{"author":{"_account_id":7478,"name":"Nadya Shakhat","email":"nadmi4@gmail.com","username":"nprivalova"},"change_message_id":"1980a5a57c74e536f18c0d9bd83937513399206a","unresolved":false,"context_lines":[{"line_number":54,"context_line":"It will allow to get rid of additional queues and coordination in general"},{"line_number":55,"context_line":"for notification agents."},{"line_number":56,"context_line":"Note that for central and alarm evaluation (aodh-related service) services"},{"line_number":57,"context_line":"coordination process remains required for scalability."},{"line_number":58,"context_line":""},{"line_number":59,"context_line":"Alternatives"},{"line_number":60,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":1,"id":"5a710552_bdc38f56","line":57,"in_reply_to":"9a8ffd7b_067c8cb3","updated":"2015-12-16 15:59:19.000000000","message":"Gordon, please check the next patch. Probably, I\u0027ve answer your questions there. \n\"do we need to lock the cache each time\" - it will happen on Redis lvl. And i believe that not the whole cache will be locked, only sets with the current key","commit_id":"ba44b0680b26edc0c8b5d5741814db18a6aa7363"},{"author":{"_account_id":6537,"name":"gordon chung","email":"gord@live.ca","username":"chungg"},"change_message_id":"776ec30cd46ee6916672bf39e3068b06c3f842b4","unresolved":false,"context_lines":[{"line_number":83,"context_line":""},{"line_number":84,"context_line":"* pipeline_processing_queues config is obsolete"},{"line_number":85,"context_line":""},{"line_number":86,"context_line":"* grouping key for transformers is obsolete"},{"line_number":87,"context_line":""},{"line_number":88,"context_line":"* Pipeline initialisation will be changed. We may refuse from"},{"line_number":89,"context_line":"  \u0027workload_partitioning\u0027 parameter for notification agent"}],"source_content_type":"text/x-rst","patch_set":1,"id":"9a8ffd7b_66347849","line":86,"updated":"2015-11-27 15:01:25.000000000","message":"transformers need to know a key still to know what value to read...","commit_id":"ba44b0680b26edc0c8b5d5741814db18a6aa7363"},{"author":{"_account_id":7478,"name":"Nadya Shakhat","email":"nadmi4@gmail.com","username":"nprivalova"},"change_message_id":"1980a5a57c74e536f18c0d9bd83937513399206a","unresolved":false,"context_lines":[{"line_number":83,"context_line":""},{"line_number":84,"context_line":"* pipeline_processing_queues config is obsolete"},{"line_number":85,"context_line":""},{"line_number":86,"context_line":"* grouping key for transformers is obsolete"},{"line_number":87,"context_line":""},{"line_number":88,"context_line":"* Pipeline initialisation will be changed. We may refuse from"},{"line_number":89,"context_line":"  \u0027workload_partitioning\u0027 parameter for notification agent"}],"source_content_type":"text/x-rst","patch_set":1,"id":"5a710552_5db86bc8","line":86,"in_reply_to":"9a8ffd7b_66347849","updated":"2015-12-16 15:59:19.000000000","message":"I\u0027ve kept this in the next spec version, will definitely revisit it, it\u0027s my TODO","commit_id":"ba44b0680b26edc0c8b5d5741814db18a6aa7363"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"4aeeb5456c0af3ea52941bf7611d61d381fa2d9c","unresolved":false,"context_lines":[{"line_number":143,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":144,"context_line":""},{"line_number":145,"context_line":"Depends on the libraries which will be used for work with the chosen cache"},{"line_number":146,"context_line":"provider (Redis, Mongo, memcache)."},{"line_number":147,"context_line":"TBD: add specific ones"},{"line_number":148,"context_line":""},{"line_number":149,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"7a740942_5735dbf4","line":146,"updated":"2015-12-04 20:06:05.000000000","message":"Probably best to use the abstraction provided by oslo_cache, which abstracts dogpile.cache, which abstracts lots of cache providers.","commit_id":"ba44b0680b26edc0c8b5d5741814db18a6aa7363"},{"author":{"_account_id":7478,"name":"Nadya Shakhat","email":"nadmi4@gmail.com","username":"nprivalova"},"change_message_id":"1980a5a57c74e536f18c0d9bd83937513399206a","unresolved":false,"context_lines":[{"line_number":143,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":144,"context_line":""},{"line_number":145,"context_line":"Depends on the libraries which will be used for work with the chosen cache"},{"line_number":146,"context_line":"provider (Redis, Mongo, memcache)."},{"line_number":147,"context_line":"TBD: add specific ones"},{"line_number":148,"context_line":""},{"line_number":149,"context_line":""}],"source_content_type":"text/x-rst","patch_set":1,"id":"5a710552_9d47b3bb","line":146,"in_reply_to":"7a740942_5735dbf4","updated":"2015-12-16 15:59:19.000000000","message":"I\u0027d love to, but I didn\u0027t find Redis support in oslo.cache. Also, memcached doesn\u0027t have sorted set structure, so I\u0027m afraid we should work to Redis directly","commit_id":"ba44b0680b26edc0c8b5d5741814db18a6aa7363"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"4aeeb5456c0af3ea52941bf7611d61d381fa2d9c","unresolved":false,"context_lines":[{"line_number":150,"context_line":"Testing"},{"line_number":151,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":152,"context_line":""},{"line_number":153,"context_line":"The current test base will be used."},{"line_number":154,"context_line":""},{"line_number":155,"context_line":""},{"line_number":156,"context_line":"Documentation Impact"}],"source_content_type":"text/x-rst","patch_set":1,"id":"7a740942_f73ba7e9","line":153,"updated":"2015-12-04 20:06:05.000000000","message":"I\u0027ve found it is rather difficult to test caching situations and the difficulty grows a great deal when the cache is distributed...","commit_id":"ba44b0680b26edc0c8b5d5741814db18a6aa7363"},{"author":{"_account_id":7478,"name":"Nadya Shakhat","email":"nadmi4@gmail.com","username":"nprivalova"},"change_message_id":"1980a5a57c74e536f18c0d9bd83937513399206a","unresolved":false,"context_lines":[{"line_number":150,"context_line":"Testing"},{"line_number":151,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":152,"context_line":""},{"line_number":153,"context_line":"The current test base will be used."},{"line_number":154,"context_line":""},{"line_number":155,"context_line":""},{"line_number":156,"context_line":"Documentation Impact"}],"source_content_type":"text/x-rst","patch_set":1,"id":"5a710552_dd6f7b29","line":153,"in_reply_to":"7a740942_f73ba7e9","updated":"2015-12-16 15:59:19.000000000","message":"Agreed. Probably, I will create some high-load scenarios for testing this (in Rally or somewhere else)","commit_id":"ba44b0680b26edc0c8b5d5741814db18a6aa7363"},{"author":{"_account_id":6537,"name":"gordon chung","email":"gord@live.ca","username":"chungg"},"change_message_id":"aa20bc8519347d39ee99e300c2cb6e1cbd63188b","unresolved":false,"context_lines":[{"line_number":50,"context_line":"Proposed change"},{"line_number":51,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":52,"context_line":""},{"line_number":53,"context_line":"Use a Redis distributed cache."},{"line_number":54,"context_line":"It will allow to get rid of additional queues and coordination in general"},{"line_number":55,"context_line":"for notification agents."},{"line_number":56,"context_line":"Note that for central and alarm evaluation (aodh-related service) services"}],"source_content_type":"text/x-rst","patch_set":2,"id":"5a710552_6684b45d","line":53,"updated":"2015-12-16 21:18:45.000000000","message":"we seem to be using it fine in ceilometer\n\nhttps://github.com/openstack/ceilometer/blob/master/devstack/settings#L38","commit_id":"480e728360622017c99fd9a8f419e106ab6358b7"},{"author":{"_account_id":6537,"name":"gordon chung","email":"gord@live.ca","username":"chungg"},"change_message_id":"aa20bc8519347d39ee99e300c2cb6e1cbd63188b","unresolved":false,"context_lines":[{"line_number":58,"context_line":""},{"line_number":59,"context_line":"The structure of the cache should be the following:"},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"#. The key is a pair \u003cresource_id+meter_name+transformer_id\u003e"},{"line_number":62,"context_line":"#. The value is a sorted set structure. The key in this sorted set is a timestamp,"},{"line_number":63,"context_line":"   values are values of corresponding samples"},{"line_number":64,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"5a710552_91cebc2c","line":61,"updated":"2015-12-16 21:18:45.000000000","message":"i don\u0027t think this key won\u0027t works. some transformers work across meters, some meters don\u0027t really care meter-agnostic.\n\nwhat\u0027s transformer id?","commit_id":"480e728360622017c99fd9a8f419e106ab6358b7"},{"author":{"_account_id":7478,"name":"Nadya Shakhat","email":"nadmi4@gmail.com","username":"nprivalova"},"change_message_id":"6bc679ec7b8bf2ef5d38aaceaffca6eaf628aca9","unresolved":false,"context_lines":[{"line_number":58,"context_line":""},{"line_number":59,"context_line":"The structure of the cache should be the following:"},{"line_number":60,"context_line":""},{"line_number":61,"context_line":"#. The key is a pair \u003cresource_id+meter_name+transformer_id\u003e"},{"line_number":62,"context_line":"#. The value is a sorted set structure. The key in this sorted set is a timestamp,"},{"line_number":63,"context_line":"   values are values of corresponding samples"},{"line_number":64,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"3a7e1126_65d95ef3","line":61,"in_reply_to":"5a710552_91cebc2c","updated":"2015-12-17 11:06:32.000000000","message":"I\u0027ve found cache usage in arithmetic and in conversions transformers. In conversions, we always use resource_id+meter_name. In arithmetic, we always have \u0027required_meters\u0027. So, we may get all values by iterating through this list. Am I missing anything?\ntransformer_id is an attempt to have as many cached copies as needed. One sample may be transformed several times. If we remove the cached value after its usage, we need to be sure that  other transformers still have this value to operate with. I\u0027m not sure about this solution, probably it\u0027s better not remove any values, it this case we don\u0027t need any transformer_id.  We may clean up the cache from time to time manually or using lua scripts in reds","commit_id":"480e728360622017c99fd9a8f419e106ab6358b7"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"28b9d133914bd6861a134c69b771c73c08e09992","unresolved":false,"context_lines":[{"line_number":65,"context_line":""},{"line_number":66,"context_line":"We will keep only one value per message as it\u0027s done now. Roughly speaking, amount of"},{"line_number":67,"context_line":"keys will be equal to amount of resources multiplied by amount of metrics. It will be ok"},{"line_number":68,"context_line":"for such tools as Redis or memcached."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"We will be able to use oslo.cache because don\u0027t need any specific features of Redis or whatever."},{"line_number":71,"context_line":"But probably, we have to add Redis support to oslo.cache as well."}],"source_content_type":"text/x-rst","patch_set":3,"id":"da6ed579_5550cb31","line":68,"updated":"2016-01-15 14:24:45.000000000","message":"There\u0027s a lot missing from this description.\n\n* What puts information into the cache? How and when?\n* What takes information out of the cache? How and when?\n\nWe\u0027ve got fairly clear info on the structure of the cache, but not very much no the mechanics of the caching and use of the cache? Without that it is impossible to really know what\u0027s going on.\n\nThe above is _clearly_ more palatable than the earlier redis-set handling, but the implementation is unclear.","commit_id":"84a5333f733a62df1e2bd362b4bf9ef90b17b120"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"28b9d133914bd6861a134c69b771c73c08e09992","unresolved":false,"context_lines":[{"line_number":68,"context_line":"for such tools as Redis or memcached."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"We will be able to use oslo.cache because don\u0027t need any specific features of Redis or whatever."},{"line_number":71,"context_line":"But probably, we have to add Redis support to oslo.cache as well."},{"line_number":72,"context_line":""},{"line_number":73,"context_line":""},{"line_number":74,"context_line":"Alternatives"}],"source_content_type":"text/x-rst","patch_set":3,"id":"da6ed579_f5421f67","line":71,"updated":"2016-01-15 14:24:45.000000000","message":"It\u0027s there, it\u0027s just declared weirdly; You say you want to use dogpile.redis as the caching engine, but the oslo interface continues to work and be present.","commit_id":"84a5333f733a62df1e2bd362b4bf9ef90b17b120"}]}