)]}'
{"specs/liberty/resource-metadata-caching.rst":[{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"ef6cba8a7f311a3984984c394ebcba779af9a58a","unresolved":false,"context_lines":[{"line_number":28,"context_line":"poll of a resource. Then the compute agent  won\u0027t poll the resouce for the"},{"line_number":29,"context_line":"meta-data until directed by a notification agent. Notification agents will"},{"line_number":30,"context_line":"react to a `compute.instance.update` and make a RPC call to the compute-agent"},{"line_number":31,"context_line":"or agents on that host to invalidate the cache and poll the resource again to"},{"line_number":32,"context_line":"refresh the cache. This caching mechanism would be disabled by default."},{"line_number":33,"context_line":""},{"line_number":34,"context_line":"This is assuming that a unique identifer is present in the `compute.instance.update`"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5afe65bd_909f275f","line":31,"updated":"2015-06-02 07:39:33.000000000","message":"The \"poll the resource again\" should only occur when the next polling cycle rolls around ... i.e. we don\u0027t want to poll multiple times within a polling interval if the instance happens to be updated several times in quick succession (as happens often in practice).","commit_id":"e37caceabe506cf969e85560f3e128659e4cbfd6"},{"author":{"_account_id":6537,"name":"gordon chung","email":"gord@live.ca","username":"chungg"},"change_message_id":"7a726592efbddacf7ec0b2bb747c97e007111352","unresolved":false,"context_lines":[{"line_number":38,"context_line":"Alternatives"},{"line_number":39,"context_line":"------------"},{"line_number":40,"context_line":""},{"line_number":41,"context_line":"None"},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"Data model impact"},{"line_number":44,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a016987_3b1f4bb9","line":41,"updated":"2015-05-25 21:24:46.000000000","message":"there is a changes-since parameter you can pass in to nova_client.[1] it seems to be a lot simpler than the mechanism proposed here. i think we should try to look at that first.\n\ni think leveraging the above we can cache all the instances... and only update the ones that are changed since last poll.\n\n\n[1] http://developer.openstack.org/api-ref-compute-v2.html#listDetailServers","commit_id":"e37caceabe506cf969e85560f3e128659e4cbfd6"},{"author":{"_account_id":6794,"name":"Simon Pasquier","email":"pasquier.simon@gmail.com","username":"simon-pasquier"},"change_message_id":"8b6231c8928439defeec1c20a51519288fd786a5","unresolved":false,"context_lines":[{"line_number":38,"context_line":"Alternatives"},{"line_number":39,"context_line":"------------"},{"line_number":40,"context_line":""},{"line_number":41,"context_line":"None"},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"Data model impact"},{"line_number":44,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"3afb71cf_d4188120","line":41,"in_reply_to":"3afb71cf_deb69089","updated":"2015-06-09 11:32:28.000000000","message":"My point was precisely that unlike querying the API, leveraging the libvirt metadata would have been scalable and efficient. But after a second thought, this can\u0027t work (at least in the current version) since the libvirt metadata doesn\u0027t contain all the instance\u0027s metadata...","commit_id":"e37caceabe506cf969e85560f3e128659e4cbfd6"},{"author":{"_account_id":14445,"name":"Jason Myers","email":"jason@jasonamyers.com","username":"jasonamyers"},"change_message_id":"ce5377baa55f7e965978737125c333fbc22fe9b7","unresolved":false,"context_lines":[{"line_number":38,"context_line":"Alternatives"},{"line_number":39,"context_line":"------------"},{"line_number":40,"context_line":""},{"line_number":41,"context_line":"None"},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"Data model impact"},{"line_number":44,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"3afb71cf_deb69089","line":41,"in_reply_to":"3afb71cf_eb3789e1","updated":"2015-06-08 13:49:21.000000000","message":"Wouldn\u0027t we prefer to use the API, and leave the direct interaction to nova?","commit_id":"e37caceabe506cf969e85560f3e128659e4cbfd6"},{"author":{"_account_id":6794,"name":"Simon Pasquier","email":"pasquier.simon@gmail.com","username":"simon-pasquier"},"change_message_id":"578be86e1287a71e4f342f610362f209eaaa2501","unresolved":false,"context_lines":[{"line_number":38,"context_line":"Alternatives"},{"line_number":39,"context_line":"------------"},{"line_number":40,"context_line":""},{"line_number":41,"context_line":"None"},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"Data model impact"},{"line_number":44,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"3afb71cf_eb3789e1","line":41,"in_reply_to":"5afe65bd_474b455a","updated":"2015-06-05 14:18:11.000000000","message":"Sorry if this is a naive question or if it\u0027s already been discussed previously but has anyone looked at using libvirtd domain meta data [1]? There are already lots of information in the XML schema generated by nova-compute like instance_id, tenant_id, ...\n\n[1] https://blueprints.launchpad.net/nova/+spec/libvirt-driver-domain-metadata","commit_id":"e37caceabe506cf969e85560f3e128659e4cbfd6"},{"author":{"_account_id":6537,"name":"gordon chung","email":"gord@live.ca","username":"chungg"},"change_message_id":"ca2f3e6758d0df8cb8cf27798ecdce0acccdb47a","unresolved":false,"context_lines":[{"line_number":38,"context_line":"Alternatives"},{"line_number":39,"context_line":"------------"},{"line_number":40,"context_line":""},{"line_number":41,"context_line":"None"},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"Data model impact"},{"line_number":44,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5afe65bd_474b455a","line":41,"in_reply_to":"5afe65bd_84ca5b4e","updated":"2015-06-01 20:17:10.000000000","message":"right. tbh, i don\u0027t think there\u0027s a way to minimise it anymore? right now, we poll nova-api once for list instances, get it\u0027s metadata, and then poll hypervisor for each instance. we will always need to poll for instances and poll hypervisor... there is no way around that (aside from killing polling)","commit_id":"e37caceabe506cf969e85560f3e128659e4cbfd6"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"196d5b2528e1bb018eec20c7dabce1aaef1e31b4","unresolved":false,"context_lines":[{"line_number":38,"context_line":"Alternatives"},{"line_number":39,"context_line":"------------"},{"line_number":40,"context_line":""},{"line_number":41,"context_line":"None"},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"Data model impact"},{"line_number":44,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a016987_ec048618","line":41,"in_reply_to":"7a016987_0615019a","updated":"2015-06-01 13:50:23.000000000","message":"I don\u0027t reckon that using changes-since will address the main concern: talking to the API too much. Yes, it will lower the amount of data being transferred but the presenting problem is that the compute-agent talks to the nova-api too much. If we can skip that, then bonus.\n\nProbably a combination of the in-agent cache with changes-since is useful.","commit_id":"e37caceabe506cf969e85560f3e128659e4cbfd6"},{"author":{"_account_id":6537,"name":"gordon chung","email":"gord@live.ca","username":"chungg"},"change_message_id":"a16ad22808316f39c6edd62271e76ac28b526fb6","unresolved":false,"context_lines":[{"line_number":38,"context_line":"Alternatives"},{"line_number":39,"context_line":"------------"},{"line_number":40,"context_line":""},{"line_number":41,"context_line":"None"},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"Data model impact"},{"line_number":44,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a016987_0615019a","line":41,"in_reply_to":"7a016987_3b1f4bb9","updated":"2015-05-27 19:18:34.000000000","message":"as promised,\n\ni tried using the \u0027changes-since\u0027 param (see api link above) when we call nova-api to gather instances: https://github.com/openstack/ceilometer/blob/master/ceilometer/nova_client.py#L140-L145\n\nit doesn\u0027t change the number of calls we make but it does change the load of what is returned... basically instead of returning all instances, it returns a subset based on what has changed since a certain time.\n\ni was thinking of adding a instance cache to the client or somewhere. basically it would capture all the instances and the timestamp of the poll. the next poll we\u0027ll query for all the instances that changed since the previous poll, and update/delete them accordingly.\n\nso in theory the first poll we\u0027d get say 1000 instances, we\u0027d save those details... the next poll should return significantly less (assuming you aren\u0027t destroying/editing/creating new instances constantly) and we\u0027d update the saved set of instances.\n\ni don\u0027t know what else we can change... without jumping into nova.","commit_id":"e37caceabe506cf969e85560f3e128659e4cbfd6"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"4cb1fd15c2c60cad649bd16dfb5a05005550f475","unresolved":false,"context_lines":[{"line_number":38,"context_line":"Alternatives"},{"line_number":39,"context_line":"------------"},{"line_number":40,"context_line":""},{"line_number":41,"context_line":"None"},{"line_number":42,"context_line":""},{"line_number":43,"context_line":"Data model impact"},{"line_number":44,"context_line":"-----------------"}],"source_content_type":"text/x-rst","patch_set":4,"id":"5afe65bd_84ca5b4e","line":41,"in_reply_to":"7a016987_ec048618","updated":"2015-06-01 20:03:26.000000000","message":"IIUC, the main potential pay-off of using changes-since will be the fewer number of onward calls out to the neutron API.\n\n(It was speculated at summit that the ceilo load on the neutron-api service was induced by indirect calls from nova-\u003eneutron to retrieve network info for instances)","commit_id":"e37caceabe506cf969e85560f3e128659e4cbfd6"},{"author":{"_account_id":11564,"name":"Chris Dent","email":"cdent@anticdent.org","username":"chdent"},"change_message_id":"196d5b2528e1bb018eec20c7dabce1aaef1e31b4","unresolved":false,"context_lines":[{"line_number":106,"context_line":" - Change compute-agent to populate an in-memory cache if the flag."},{"line_number":107,"context_line":" - Change compute-agent to check for an invalid cache prior to polling."},{"line_number":108,"context_line":" - Change notification-agent to make an RPC call to compute-agents when"},{"line_number":109,"context_line":"   a `compute.instance.update` is processed."},{"line_number":110,"context_line":" - Change compute-agent to invalidate the cache in reaction to the"},{"line_number":111,"context_line":"   notification-agents RPC call."},{"line_number":112,"context_line":" - Add a jitter to avoid flooding repoll requests and the compute-agent."}],"source_content_type":"text/x-rst","patch_set":4,"id":"7a016987_477db36b","line":109,"updated":"2015-06-01 13:50:23.000000000","message":"I\u0027m still not entirely happy about this way of doing things. It creates an implicit coupling between the notification-agent and compute-agent which in a perfect universe wouldn\u0027t be necessary.\n\nWe should, at least, make sure that the RPC call is powerfully documented so that something else can call it. We don\u0027t want to require that the notification-agent be installed and running in order to use this new caching mechanism.\n\nAlso, it would be handy if somewhere in this doc the mechanism for dealing with multiple compute-agents is described. Will all agents listen for the same rpc call and invalidate at once? Or will they only hear a call that is targeted for the agents on this host, or just this one agent. It is possible for there to be multiple compute-agents on a single host, using tooz-based coordination. It makes fairly reasonable sense that all of the agents on one host will invalidate in one go, but it would be a shame if all agents everywhere were invalidated at once: Then, even with jitter, we are creating spikes in usage rather than smoothing things out.","commit_id":"e37caceabe506cf969e85560f3e128659e4cbfd6"},{"author":{"_account_id":14445,"name":"Jason Myers","email":"jason@jasonamyers.com","username":"jasonamyers"},"change_message_id":"ce5377baa55f7e965978737125c333fbc22fe9b7","unresolved":false,"context_lines":[{"line_number":106,"context_line":" - Change compute-agent to populate an in-memory cache if the flag."},{"line_number":107,"context_line":" - Change compute-agent to check for an invalid cache prior to polling."},{"line_number":108,"context_line":" - Change notification-agent to make an RPC call to compute-agents when"},{"line_number":109,"context_line":"   a `compute.instance.update` is processed."},{"line_number":110,"context_line":" - Change compute-agent to invalidate the cache in reaction to the"},{"line_number":111,"context_line":"   notification-agents RPC call."},{"line_number":112,"context_line":" - Add a jitter to avoid flooding repoll requests and the compute-agent."}],"source_content_type":"text/x-rst","patch_set":4,"id":"3afb71cf_3e1c84c0","line":109,"in_reply_to":"5afe65bd_55bf53d0","updated":"2015-06-08 13:49:21.000000000","message":"I like the idea of changes-since better. Not having to add the RPC calls into the mix would remove a bit of complexity.","commit_id":"e37caceabe506cf969e85560f3e128659e4cbfd6"},{"author":{"_account_id":6537,"name":"gordon chung","email":"gord@live.ca","username":"chungg"},"change_message_id":"ce4770c5bd63603f7d00052d36dbb41e28e52557","unresolved":false,"context_lines":[{"line_number":106,"context_line":" - Change compute-agent to populate an in-memory cache if the flag."},{"line_number":107,"context_line":" - Change compute-agent to check for an invalid cache prior to polling."},{"line_number":108,"context_line":" - Change notification-agent to make an RPC call to compute-agents when"},{"line_number":109,"context_line":"   a `compute.instance.update` is processed."},{"line_number":110,"context_line":" - Change compute-agent to invalidate the cache in reaction to the"},{"line_number":111,"context_line":"   notification-agents RPC call."},{"line_number":112,"context_line":" - Add a jitter to avoid flooding repoll requests and the compute-agent."}],"source_content_type":"text/x-rst","patch_set":4,"id":"5afe65bd_55bf53d0","line":109,"in_reply_to":"5afe65bd_90c84776","updated":"2015-06-02 13:58:16.000000000","message":"hmm.. that\u0027s what i was looking at as well. from what i can see, the flavour/image cache is cleared every single poll cycle.\n\nif we were to cache the instances (which i assume is needed since we pass a list of instances to do hypervisor polling) -- i\u0027m not sure how the update notifications differ from changes-since param in addition to the latency of capturing and rebroadcasting notifications. also, are there update notifications on a delete as i think we need that as well (assuming the instance cache assumption)","commit_id":"e37caceabe506cf969e85560f3e128659e4cbfd6"},{"author":{"_account_id":6537,"name":"gordon chung","email":"gord@live.ca","username":"chungg"},"change_message_id":"ca2f3e6758d0df8cb8cf27798ecdce0acccdb47a","unresolved":false,"context_lines":[{"line_number":106,"context_line":" - Change compute-agent to populate an in-memory cache if the flag."},{"line_number":107,"context_line":" - Change compute-agent to check for an invalid cache prior to polling."},{"line_number":108,"context_line":" - Change notification-agent to make an RPC call to compute-agents when"},{"line_number":109,"context_line":"   a `compute.instance.update` is processed."},{"line_number":110,"context_line":" - Change compute-agent to invalidate the cache in reaction to the"},{"line_number":111,"context_line":"   notification-agents RPC call."},{"line_number":112,"context_line":" - Add a jitter to avoid flooding repoll requests and the compute-agent."}],"source_content_type":"text/x-rst","patch_set":4,"id":"5afe65bd_a7a9d9e1","line":109,"in_reply_to":"5afe65bd_a48797d6","updated":"2015-06-01 20:17:10.000000000","message":"eglynn, do you know where in the code we are retrieve metadata? i couldn\u0027t find the place where we\u0027d actually require using compute.instance.update notification. the changes-since param seem to cover all the \u0027metadata\u0027 building i could find.","commit_id":"e37caceabe506cf969e85560f3e128659e4cbfd6"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"ef6cba8a7f311a3984984c394ebcba779af9a58a","unresolved":false,"context_lines":[{"line_number":106,"context_line":" - Change compute-agent to populate an in-memory cache if the flag."},{"line_number":107,"context_line":" - Change compute-agent to check for an invalid cache prior to polling."},{"line_number":108,"context_line":" - Change notification-agent to make an RPC call to compute-agents when"},{"line_number":109,"context_line":"   a `compute.instance.update` is processed."},{"line_number":110,"context_line":" - Change compute-agent to invalidate the cache in reaction to the"},{"line_number":111,"context_line":"   notification-agents RPC call."},{"line_number":112,"context_line":" - Add a jitter to avoid flooding repoll requests and the compute-agent."}],"source_content_type":"text/x-rst","patch_set":4,"id":"5afe65bd_90c84776","line":109,"in_reply_to":"5afe65bd_a7a9d9e1","updated":"2015-06-02 07:39:33.000000000","message":"@gordc: the metadata retrieval is essentially done here[1] as part of the compute agent polling cycle discovery (i.e. just a straight all-tenants GET on the nova servers API, constrained by host, with the addition of cached flavor and image details). \n\nWe currently don\u0027t use the compute.instance.update notification for the purpose of retrieving metadata to add to the samples created by the compute agent polling. The idea at summit was to switch to using the occurance of a compute.instance.update to trigger a refresh on the metadata cache entry for that instance - so this would replace the once-per-cycle nova-api polling, allowing us to skip the nova-api call when there was no update of any instance owned by an agent during the preceding polling interval. (I\u0027m probably stating the obvious there, apologies).\n\n[1] https://github.com/openstack/ceilometer/blob/master/ceilometer/nova_client.py#L139","commit_id":"e37caceabe506cf969e85560f3e128659e4cbfd6"},{"author":{"_account_id":2284,"name":"Eoghan Glynn","email":"eglynn@redhat.com","username":"eglynn"},"change_message_id":"4cb1fd15c2c60cad649bd16dfb5a05005550f475","unresolved":false,"context_lines":[{"line_number":106,"context_line":" - Change compute-agent to populate an in-memory cache if the flag."},{"line_number":107,"context_line":" - Change compute-agent to check for an invalid cache prior to polling."},{"line_number":108,"context_line":" - Change notification-agent to make an RPC call to compute-agents when"},{"line_number":109,"context_line":"   a `compute.instance.update` is processed."},{"line_number":110,"context_line":" - Change compute-agent to invalidate the cache in reaction to the"},{"line_number":111,"context_line":"   notification-agents RPC call."},{"line_number":112,"context_line":" - Add a jitter to avoid flooding repoll requests and the compute-agent."}],"source_content_type":"text/x-rst","patch_set":4,"id":"5afe65bd_a48797d6","line":109,"in_reply_to":"7a016987_477db36b","updated":"2015-06-01 20:03:26.000000000","message":"Per-host invalidation (of the caches in the one or more compute agents running on a single compute node) is the way to go here IMO. Assuming the compute.instance.update notification payload carries sufficient info to determine the node that the instance is currently scheduled to.\n\nOn the RPC call creating a coupling, I\u0027m not sure that this would be a show-stopper for split out, as long as it\u0027s properly versioned and contractualized  - there\u0027s a RPC interaction between the nova-{api|compute|conductor} and nova-scheduler services while efforts are afoot to split the latter service out also.","commit_id":"e37caceabe506cf969e85560f3e128659e4cbfd6"},{"author":{"_account_id":6537,"name":"gordon chung","email":"gord@live.ca","username":"chungg"},"change_message_id":"be2fb5366507636b9e22c1b48f9433bc2485c0be","unresolved":false,"context_lines":[{"line_number":32,"context_line":"of this call will update the cache, and the cache will be used to return"},{"line_number":33,"context_line":"the data. Also, we should populate the instance flavor and image data in"},{"line_number":34,"context_line":"the cache as well. There caching and query changes would be disabled by"},{"line_number":35,"context_line":"default via a configuration option."},{"line_number":36,"context_line":""},{"line_number":37,"context_line":"Alternatives"},{"line_number":38,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":6,"id":"3afb71cf_22020970","line":35,"updated":"2015-06-09 03:58:16.000000000","message":"just curious why we will disable? or make it configurable? is there a concern the memory cache will be too large?","commit_id":"afe4052eab0360618a69020f5c276feb1a458313"},{"author":{"_account_id":6537,"name":"gordon chung","email":"gord@live.ca","username":"chungg"},"change_message_id":"0ed8e045647bae64a30dafbad7f4a63f08f4392e","unresolved":false,"context_lines":[{"line_number":32,"context_line":"of this call will update the cache, and the cache will be used to return"},{"line_number":33,"context_line":"the data. Also, we should populate the instance flavor and image data in"},{"line_number":34,"context_line":"the cache as well. There caching and query changes would be disabled by"},{"line_number":35,"context_line":"default via a configuration option."},{"line_number":36,"context_line":""},{"line_number":37,"context_line":"Alternatives"},{"line_number":38,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":6,"id":"3afb71cf_fb842cd3","line":35,"in_reply_to":"3afb71cf_14285999","updated":"2015-06-11 03:38:58.000000000","message":"let\u0027s default this sucker... we already implicitly cache other stuff.\n\ni can\u0027t imagine the number of instances on a single host being that large although someone can correct me.","commit_id":"afe4052eab0360618a69020f5c276feb1a458313"},{"author":{"_account_id":14445,"name":"Jason Myers","email":"jason@jasonamyers.com","username":"jasonamyers"},"change_message_id":"432921ba313c21a4fb54e6450444e763af576e7e","unresolved":false,"context_lines":[{"line_number":32,"context_line":"of this call will update the cache, and the cache will be used to return"},{"line_number":33,"context_line":"the data. Also, we should populate the instance flavor and image data in"},{"line_number":34,"context_line":"the cache as well. There caching and query changes would be disabled by"},{"line_number":35,"context_line":"default via a configuration option."},{"line_number":36,"context_line":""},{"line_number":37,"context_line":"Alternatives"},{"line_number":38,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":6,"id":"3afb71cf_14285999","line":35,"in_reply_to":"3afb71cf_22020970","updated":"2015-06-09 11:48:42.000000000","message":"this was brought up by eglynn while we were in YVR; however, this doesnt have all the rpc changes etc. So I\u0027d love to make it default.","commit_id":"afe4052eab0360618a69020f5c276feb1a458313"}]}