)]}'
{"specs/in_progress/notifications.rst":[{"author":{"_account_id":4608,"name":"Kota Tsuyuzaki","email":"bloodeagle40234@gmail.com","username":"tsuyuzaki-kota"},"change_message_id":"4d382f5df8bc6955570b92d552b939be1d90fa0b","unresolved":false,"context_lines":[{"line_number":11,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"Swift should be able to send out notifications if new objects are uploaded,"},{"line_number":14,"context_line":"metadata has been changed or data has been deleted."},{"line_number":15,"context_line":""},{"line_number":16,"context_line":"Problem Description"},{"line_number":17,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a016987_9da06992","line":14,"updated":"2015-05-14 12:25:29.000000000","message":"Sounds good. For instance, I have an idea if we could use this for expired notification (i.e. the request notification from object-expirer) :)","commit_id":"9a9f429f93af267e701a3d9f38881a55391e339e"},{"author":{"_account_id":5600,"name":"Brian Cline","email":"bcline@softlayer.com","username":"briancline"},"change_message_id":"7ee95d9ab0407b162202f30d63aedecc4dee5743","unresolved":false,"context_lines":[{"line_number":19,"context_line":"Currently there is no way to detect changes in a given container except listing"},{"line_number":20,"context_line":"it\u0027s contents and comparing timestamps. This makes it difficult and slow in case"},{"line_number":21,"context_line":"there are a lot of objects stored, and it is not very efficient at all."},{"line_number":22,"context_line":"Some external services might be interested when an object got uploaded, updated"},{"line_number":23,"context_line":"or deleted; for example to store the metadata in an external database for"},{"line_number":24,"context_line":"searching or to trigger specific events like computing on object data."},{"line_number":25,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a016987_cdcafd1a","line":22,"updated":"2015-05-21 19:06:59.000000000","message":"I think it\u0027d be valuable to issue notifications at account and container level, in addition to the object notifications discussed here.","commit_id":"9a9f429f93af267e701a3d9f38881a55391e339e"},{"author":{"_account_id":5600,"name":"Brian Cline","email":"bcline@softlayer.com","username":"briancline"},"change_message_id":"7ee95d9ab0407b162202f30d63aedecc4dee5743","unresolved":false,"context_lines":[{"line_number":35,"context_line":"into notifications per account and sending batches of updates to an external"},{"line_number":36,"context_line":"service. However, performance is most likely worse since there is a lot of"},{"line_number":37,"context_line":"string parsing involved, and a central logging service might be required to send"},{"line_number":38,"context_line":"notifications in order."},{"line_number":39,"context_line":""},{"line_number":40,"context_line":"Implementation"},{"line_number":41,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a016987_f82e998a","line":38,"updated":"2015-05-21 19:06:59.000000000","message":"Yes, I can say for sure performance is worse in this scenario. We have had scenarios where we\u0027ve had to analyze logs for small periods where there were metadata indexing issues. It doesn\u0027t take long because we already have supporting code/infra to handle this quickly. But the problem arises if you need more than the a/c/o path (like if you want metadata, etc.), because then you have to go HEAD each object, and end up hammering your object nodes quite a bit if you need to backfill a lot of objects. In some cases it is the only option, though. We also log a warning each time we can\u0027t emit an indexing notification for something, so that we can more easily pick up from the logs a list of things that we are certain need to be rechecked.","commit_id":"9a9f429f93af267e701a3d9f38881a55391e339e"},{"author":{"_account_id":5600,"name":"Brian Cline","email":"bcline@softlayer.com","username":"briancline"},"change_message_id":"7ee95d9ab0407b162202f30d63aedecc4dee5743","unresolved":false,"context_lines":[{"line_number":43,"context_line":"Sending out notifications should happen when an object got modified. That means"},{"line_number":44,"context_line":"every successful object change (PUT, POST, DELETE) should trigger an action and"},{"line_number":45,"context_line":"send out an event notification."},{"line_number":46,"context_line":"It should be configurable on either an account or container level that"},{"line_number":47,"context_line":"notifications should be sent; this leaves it up to the user to decide where they"},{"line_number":48,"context_line":"end up and if a possible performance impact is acceptable."},{"line_number":49,"context_line":"An implementation should be developed as an additional middleware inside the Swift"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a016987_6db8a975","line":46,"updated":"2015-05-21 19:06:59.000000000","message":"My vote would be at the container level, with a configurable default if the option is not found. This way if we want all notifications by default, we can change a config value to treat no value at the container level to mean \"always emit notification\". Whereas the out of box default should be off by default.\n\nOur current implementation has this toggleable at the account level but we are also seeing cases where it\u0027s very valuable to control at the container level.","commit_id":"9a9f429f93af267e701a3d9f38881a55391e339e"},{"author":{"_account_id":5600,"name":"Brian Cline","email":"bcline@softlayer.com","username":"briancline"},"change_message_id":"7ee95d9ab0407b162202f30d63aedecc4dee5743","unresolved":false,"context_lines":[{"line_number":49,"context_line":"An implementation should be developed as an additional middleware inside the Swift"},{"line_number":50,"context_line":"proxy, and make use of existing queuing implementations within OpenStack,"},{"line_number":51,"context_line":"namely Zaqar (https://wiki.openstack.org/wiki/Zaqar)."},{"line_number":52,"context_line":"It needs to be discussed if metadata that is stored along the object should be"},{"line_number":53,"context_line":"included in the notification or not; if there is a lot of metadata the"},{"line_number":54,"context_line":"notifications are getting quite large. A possible trade off might be a threshold"},{"line_number":55,"context_line":"for included metadata, for example only the first X bytes. Or send no metadata"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a016987_6d00a966","line":52,"updated":"2015-05-21 19:06:59.000000000","message":"I think this is very valuable to do - we do this for metadata search today. Perhaps this should then be toggleable since getting the full picture of metadata on an existing object you\u0027re POSTing to can incur some additional overhead.","commit_id":"9a9f429f93af267e701a3d9f38881a55391e339e"},{"author":{"_account_id":4608,"name":"Kota Tsuyuzaki","email":"bloodeagle40234@gmail.com","username":"tsuyuzaki-kota"},"change_message_id":"4d382f5df8bc6955570b92d552b939be1d90fa0b","unresolved":false,"context_lines":[{"line_number":49,"context_line":"An implementation should be developed as an additional middleware inside the Swift"},{"line_number":50,"context_line":"proxy, and make use of existing queuing implementations within OpenStack,"},{"line_number":51,"context_line":"namely Zaqar (https://wiki.openstack.org/wiki/Zaqar)."},{"line_number":52,"context_line":"It needs to be discussed if metadata that is stored along the object should be"},{"line_number":53,"context_line":"included in the notification or not; if there is a lot of metadata the"},{"line_number":54,"context_line":"notifications are getting quite large. A possible trade off might be a threshold"},{"line_number":55,"context_line":"for included metadata, for example only the first X bytes. Or send no metadata"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a016987_7d6c8d9c","line":52,"updated":"2015-05-14 12:25:29.000000000","message":"If you\u0027ve already expected which metadata would be needed, it would be good to write it down somewhere as examples.\n\nFor now, I think if some kinds of metadata which have fixed length (e.g. X-Timestamp, X-Delete-At or something like as Etag?) would be better but we would not want user meta (like X-Object-Meta-xxx) in the stored object for the notification because the user meta generally *uncontrollable* for operators though perhaps, the *requested* user metadata(e.g. POSTed metadata) might be needed there. I\u0027m not sure.\n\nAnyways, agreed we need more discussion which way is suitable for Swift.","commit_id":"9a9f429f93af267e701a3d9f38881a55391e339e"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"f741c361366392fd38e5e7770cb56117e700c73a","unresolved":false,"context_lines":[{"line_number":53,"context_line":"included in the notification or not; if there is a lot of metadata the"},{"line_number":54,"context_line":"notifications are getting quite large. A possible trade off might be a threshold"},{"line_number":55,"context_line":"for included metadata, for example only the first X bytes. Or send no metadata"},{"line_number":56,"context_line":"at all, but only the account/container/objectname."},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"Assignee(s)"},{"line_number":59,"context_line":"-----------"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a016987_9a0d1d3a","line":56,"updated":"2015-05-14 01:51:03.000000000","message":"Do we want messages to be connectionless, i.e. like UDP? We send, but the middleware makes no guarantees. Keep the messages short. This way it would have minimal impact to the proxy pipeline.","commit_id":"9a9f429f93af267e701a3d9f38881a55391e339e"},{"author":{"_account_id":5600,"name":"Brian Cline","email":"bcline@softlayer.com","username":"briancline"},"change_message_id":"7ee95d9ab0407b162202f30d63aedecc4dee5743","unresolved":false,"context_lines":[{"line_number":53,"context_line":"included in the notification or not; if there is a lot of metadata the"},{"line_number":54,"context_line":"notifications are getting quite large. A possible trade off might be a threshold"},{"line_number":55,"context_line":"for included metadata, for example only the first X bytes. Or send no metadata"},{"line_number":56,"context_line":"at all, but only the account/container/objectname."},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"Assignee(s)"},{"line_number":59,"context_line":"-----------"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7a016987_ed001904","line":56,"updated":"2015-05-21 19:06:59.000000000","message":"The option should definitely be there, yes. It seems that if this is pluggable and someone doesn\u0027t use the reference Zaqar implementation, that this would be more of a decision at the individual driver level, right?","commit_id":"9a9f429f93af267e701a3d9f38881a55391e339e"}]}
