)]}'
{"/COMMIT_MSG":[{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"7fc7987b0270070d4294a6f0205156cf9d27c4e5","unresolved":false,"context_lines":[{"line_number":7,"context_line":"New middleware for Openstack Swift allowing to dynamically add"},{"line_number":8,"context_line":"compute functions along the pipeline, without the need to epxlicitly"},{"line_number":9,"context_line":"add new middleware. Code reference can be found here:"},{"line_number":10,"context_line":"https://github.com/Open-I-Beam/swift-storlets"},{"line_number":11,"context_line":""},{"line_number":12,"context_line":"Change-Id: I273d23db6e9373a71115e4b685cc1dfe9536f063"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":5,"id":"7a016987_7a4b812c","line":10,"updated":"2015-05-14 02:49:35.000000000","message":"This paragraph should be broken up into a summary line and then the more detailed description, see: https://wiki.openstack.org/wiki/GitCommitMessages#Summary_of_Git_commit_message_structure\n\nSo could be something like:\n\n  Swift-Storlets - allowing dynamic compute functions\n\n  A new middleware for Openstack Swift allowing to dynamically add\n  compute functions along the pipeline, without the need to epxlicitly\n  add new middleware.\n\n  Code reference can be found here:\n      https://github.com/Open-I-Beam/swift-storlets","commit_id":"001dac6d9b2f66007e66193ef61f33f62ce9593c"}],"specs/in_progress/storlets.rst":[{"author":{"_account_id":12193,"name":"Hisashi Osanai","email":"osanai.hisashi@jp.fujitsu.com","username":"ho"},"change_message_id":"faf3f37a9d031b57e492696ddaa747d6354670f7","unresolved":false,"context_lines":[{"line_number":53,"context_line":"storage and computation by performing certain computational tasks close to the data,"},{"line_number":54,"context_line":"thus avoiding unnecessary data movements over the network. While these trends are well"},{"line_number":55,"context_line":"established in the analytics world, we only start to see them in object stores, e.g."},{"line_number":56,"context_line":"`ZeorVM \u003chttp://www.zerovm.org/\u003e`_ and `Joyent/Manta \u003chttps://www.joyent.com/object-storage\u003e`_"},{"line_number":57,"context_line":""},{"line_number":58,"context_line":"2. High Level Architecture"},{"line_number":59,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":5,"id":"1aca2d91_dec45060","line":56,"updated":"2015-05-08 07:41:34.000000000","message":"s/ZeorVM/ZeroVM/","commit_id":"001dac6d9b2f66007e66193ef61f33f62ce9593c"},{"author":{"_account_id":866,"name":"Chmouel Boudjnah","email":"chmouel@chmouel.com","username":"chmouel"},"change_message_id":"8abf8c7b67c6d5b0599af72b170e0b2276ec1dbd","unresolved":false,"context_lines":[{"line_number":86,"context_line":"----------------------------"},{"line_number":87,"context_line":"The Storlet middleware is responsible to handle the logic of storlet"},{"line_number":88,"context_line":"activation with the compute engine. When a storlet is executed"},{"line_number":89,"context_line":"the middleware extracts the data stream from the request and routes it to the engine."},{"line_number":90,"context_line":"The middleware then gets back the result and wraps it in an iterator that the Swift"},{"line_number":91,"context_line":"WSGI framework can work with."},{"line_number":92,"context_line":"Focusing on object\u0027s GET and PUT flows the middleware is added to the proxy"}],"source_content_type":"text/x-rst","patch_set":5,"id":"1aca2d91_98196c7c","line":89,"updated":"2015-05-07 14:22:43.000000000","message":"I haven\u0027t looked at the reference implementation so apologies if that\u0027s all answered.\n\nI assume that the middleware extract the data stream and it seems to say that this data stream is internally inside the Swift cluster which get streamed to a compute job inside an external compute cluster (be it containers or others)\n\nWould not make sense to have as well a way to do this as well at *initial* GET/PUT time ? I mean for example the user would say I am going to upload this object and with a header I want to apply this storlet to it and not have to upload and then run the job after.\n\nBecause if the storlets job are applied after the initial upload, what happen in case of conflicts? on which version of the object the job would get applied if when it streams an object to the compute platform and there is a new verison coming up?\n\nAgain apologies if that\u0027s already answered on the GitHub ref implementation.","commit_id":"001dac6d9b2f66007e66193ef61f33f62ce9593c"},{"author":{"_account_id":11317,"name":"Eran Rom","email":"eran@itsonlyme.name","username":"eranr"},"change_message_id":"f0f2028aa9d094d304f9fa7aad5b1a7231f83c0e","unresolved":false,"context_lines":[{"line_number":86,"context_line":"----------------------------"},{"line_number":87,"context_line":"The Storlet middleware is responsible to handle the logic of storlet"},{"line_number":88,"context_line":"activation with the compute engine. When a storlet is executed"},{"line_number":89,"context_line":"the middleware extracts the data stream from the request and routes it to the engine."},{"line_number":90,"context_line":"The middleware then gets back the result and wraps it in an iterator that the Swift"},{"line_number":91,"context_line":"WSGI framework can work with."},{"line_number":92,"context_line":"Focusing on object\u0027s GET and PUT flows the middleware is added to the proxy"}],"source_content_type":"text/x-rst","patch_set":5,"id":"1aca2d91_2561033a","line":89,"in_reply_to":"1aca2d91_98196c7c","updated":"2015-05-09 07:52:27.000000000","message":"Actually all computation is done synchronously. The main point here is to have the computation where the data is: Thus, e.g. on GET \"extracting the data stream\" means take the fd of the local file ans pass it over UDS to the compute part that runs on the same node. On the other side of the coin this is not really the lambda case :-)\nHope I did not miss your point.","commit_id":"001dac6d9b2f66007e66193ef61f33f62ce9593c"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"7fc7987b0270070d4294a6f0205156cf9d27c4e5","unresolved":false,"context_lines":[{"line_number":204,"context_line":"To deploy a new Storlet in Swift the user needs to upload the Storlet binary"},{"line_number":205,"context_line":"object to the designated Swift container (as configured by storlet_container)."},{"line_number":206,"context_line":"The name of the Storlet needs to be of the following format: storlet name.verion"},{"line_number":207,"context_line":"The upload must include the following metadata:"},{"line_number":208,"context_line":""},{"line_number":209,"context_line":"::"},{"line_number":210,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"7a016987_ba6c5909","line":207,"updated":"2015-05-14 02:49:35.000000000","message":"Are all of the headers below a must? or are some optional?\n\nWhat if you don\u0027t need to pass any additional parameters, and couldn\u0027t the default Storlet-Dependency be \u0027\u0027, and therefore not mandatory?","commit_id":"001dac6d9b2f66007e66193ef61f33f62ce9593c"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"7fc7987b0270070d4294a6f0205156cf9d27c4e5","unresolved":false,"context_lines":[{"line_number":255,"context_line":""},{"line_number":256,"context_line":"  StorletsGateway(account, conf)"},{"line_number":257,"context_line":"  StorletResponse invoke_get(StoreltGetRequest)"},{"line_number":258,"context_line":"  StorletResponse invoke_put(StoreltPutRequest)"},{"line_number":259,"context_line":""},{"line_number":260,"context_line":"Few notes:"},{"line_number":261,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"7a016987_7a0ca142","line":258,"updated":"2015-05-14 02:49:35.000000000","message":"s/Storelt/Storlet/","commit_id":"001dac6d9b2f66007e66193ef61f33f62ce9593c"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"7fc7987b0270070d4294a6f0205156cf9d27c4e5","unresolved":false,"context_lines":[{"line_number":339,"context_line":"#. The StorletResponse from the invoke_get API call consists of the returned"},{"line_number":340,"context_line":"   metadata and output file descriptor. The metadata is translated to Swift"},{"line_number":341,"context_line":"   response headers, and the file descriptor is wrapped in an iterator that is"},{"line_number":342,"context_line":"   returned as the Swift response app_iter field."},{"line_number":343,"context_line":""},{"line_number":344,"context_line":"5.10.2 PUT"},{"line_number":345,"context_line":"~~~~~~~~~~"}],"source_content_type":"text/x-rst","patch_set":5,"id":"7a016987_359f5a61","line":342,"updated":"2015-05-14 02:49:35.000000000","message":"Do we need to think about a way of notifying the proxy making the call that the result is coming and not time out? I.E if the computation takes longer then recoverable_node_timeout/node_timeout? Some 100 continue magic or something. \n\nOr will these timeouts just have to be adjusted?","commit_id":"001dac6d9b2f66007e66193ef61f33f62ce9593c"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"46f90d3d961c552acb2fc2f27820653ad403a339","unresolved":false,"context_lines":[{"line_number":206,"context_line":"5.2 Storlet Deployment"},{"line_number":207,"context_line":"----------------------"},{"line_number":208,"context_line":"To deploy a new storlet in Swift the user needs to upload the storlet binary"},{"line_number":209,"context_line":"object to a designated Swift container (as configured by storlet_container)."},{"line_number":210,"context_line":"Using a designated container the storlet middleware can intercept a PUT request"},{"line_number":211,"context_line":"of a storlet and invoke an API to validate that the storlet upload is valid."},{"line_number":212,"context_line":"In the below we describe the validation done by our reference implementation."}],"source_content_type":"text/x-rst","patch_set":6,"id":"7a016987_15ccec68","line":209,"updated":"2015-05-18 14:51:40.000000000","message":"I\u0027m not understanding why it needs to be a designated container - is the presence of storlet headers not enough to indicate that validation should be done, regardless of the destination container?","commit_id":"0a6e3ccb786cff17f94ffe5212c3573174119570"},{"author":{"_account_id":11317,"name":"Eran Rom","email":"eran@itsonlyme.name","username":"eranr"},"change_message_id":"86cc1836ca1082ec805f96d90d4c0b281b4859ea","unresolved":false,"context_lines":[{"line_number":206,"context_line":"5.2 Storlet Deployment"},{"line_number":207,"context_line":"----------------------"},{"line_number":208,"context_line":"To deploy a new storlet in Swift the user needs to upload the storlet binary"},{"line_number":209,"context_line":"object to a designated Swift container (as configured by storlet_container)."},{"line_number":210,"context_line":"Using a designated container the storlet middleware can intercept a PUT request"},{"line_number":211,"context_line":"of a storlet and invoke an API to validate that the storlet upload is valid."},{"line_number":212,"context_line":"In the below we describe the validation done by our reference implementation."}],"source_content_type":"text/x-rst","patch_set":6,"id":"7a016987_802d3548","line":209,"in_reply_to":"7a016987_15ccec68","updated":"2015-05-18 18:40:31.000000000","message":"At some point we need to bring the storlet code (kept as object) to each of the nodes, where the Storlet needs to get executed. It was easy to be looking at a predefined container. Another option is to state the container in the X-RunStorlet header, e.g. X-RunStorlet: \u003ccontainer\u003e/\u003cstorlet\u003e.\nNot sure which is better.","commit_id":"0a6e3ccb786cff17f94ffe5212c3573174119570"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"46f90d3d961c552acb2fc2f27820653ad403a339","unresolved":false,"context_lines":[{"line_number":220,"context_line":"  \u0027X-Object-Meta-Storlet-Dependency\u0027: \u0027\u0027"},{"line_number":221,"context_line":"  \u0027X-Object-Meta-Storlet-ID: storlet main invocation class\u0027"},{"line_number":222,"context_line":"  \u0027X-Object-Meta-Storlet-Key1\u0027"},{"line_number":223,"context_line":"  \u0027X-Object-Meta-Storlet-Key2\u0027"},{"line_number":224,"context_line":""},{"line_number":225,"context_line":"Where:"},{"line_number":226,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"7a016987_9589fc28","line":223,"updated":"2015-05-18 14:51:40.000000000","message":"The X-Object-Meta-* header namespace should be left for users\u0027 custom metadata and not effectively reserved for a middleware feature.\n\nI would recommend clients passing in headers using names X-Object-Storlet-\u003ckey\u003e and the storlet middleare translating these to headers X-Object-Sysmeta-Storlet-\u003ckey\u003e which will be persisted with the object.\n\nSee http://docs.openstack.org/developer/swift/development_middleware.html#swift-metadata","commit_id":"0a6e3ccb786cff17f94ffe5212c3573174119570"},{"author":{"_account_id":11317,"name":"Eran Rom","email":"eran@itsonlyme.name","username":"eranr"},"change_message_id":"86cc1836ca1082ec805f96d90d4c0b281b4859ea","unresolved":false,"context_lines":[{"line_number":220,"context_line":"  \u0027X-Object-Meta-Storlet-Dependency\u0027: \u0027\u0027"},{"line_number":221,"context_line":"  \u0027X-Object-Meta-Storlet-ID: storlet main invocation class\u0027"},{"line_number":222,"context_line":"  \u0027X-Object-Meta-Storlet-Key1\u0027"},{"line_number":223,"context_line":"  \u0027X-Object-Meta-Storlet-Key2\u0027"},{"line_number":224,"context_line":""},{"line_number":225,"context_line":"Where:"},{"line_number":226,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"7a016987_e01ce914","line":223,"in_reply_to":"7a016987_9589fc28","updated":"2015-05-18 18:40:31.000000000","message":"Agree. Will update accordingly. Note that this is not really part of the spec in the sense that the headers here are an examples coming from our gateway implentation.","commit_id":"0a6e3ccb786cff17f94ffe5212c3573174119570"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"46f90d3d961c552acb2fc2f27820653ad403a339","unresolved":false,"context_lines":[{"line_number":282,"context_line":"#. The invoke_put function takes the request as the input, as it acts on the uploaded data"},{"line_number":283,"context_line":"   which is part of the request."},{"line_number":284,"context_line":"#. All functions return a pair consisting metadata and data stream which are the"},{"line_number":285,"context_line":"   result of the computation."},{"line_number":286,"context_line":""},{"line_number":287,"context_line":"5.5 SLO / DLO"},{"line_number":288,"context_line":"-------------"}],"source_content_type":"text/x-rst","patch_set":6,"id":"7a016987_15562c5c","line":285,"updated":"2015-05-18 14:51:40.000000000","message":"could you clarify, are request headers passed to the storlet compute engine, and can they be modified by the compute engine?\n\nIf so, do you propose to filter those headers so that for example auth token is not passed to the storlet engine?","commit_id":"0a6e3ccb786cff17f94ffe5212c3573174119570"},{"author":{"_account_id":11317,"name":"Eran Rom","email":"eran@itsonlyme.name","username":"eranr"},"change_message_id":"86cc1836ca1082ec805f96d90d4c0b281b4859ea","unresolved":false,"context_lines":[{"line_number":282,"context_line":"#. The invoke_put function takes the request as the input, as it acts on the uploaded data"},{"line_number":283,"context_line":"   which is part of the request."},{"line_number":284,"context_line":"#. All functions return a pair consisting metadata and data stream which are the"},{"line_number":285,"context_line":"   result of the computation."},{"line_number":286,"context_line":""},{"line_number":287,"context_line":"5.5 SLO / DLO"},{"line_number":288,"context_line":"-------------"}],"source_content_type":"text/x-rst","patch_set":6,"id":"7a016987_a0edd1f0","line":285,"in_reply_to":"7a016987_15562c5c","updated":"2015-05-18 18:40:31.000000000","message":"Good point, Some comments:\n1. Today we pass only user defined metadata.\n2. In general this is a gateway implementation detail. If this is problematic, then we need not to pass all the request but rather filter this before we call the gateway API.","commit_id":"0a6e3ccb786cff17f94ffe5212c3573174119570"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"46f90d3d961c552acb2fc2f27820653ad403a339","unresolved":false,"context_lines":[{"line_number":305,"context_line":"------------------"},{"line_number":306,"context_line":"The question with EC and encryption is how do we run a storlet over an encoded"},{"line_number":307,"context_line":"or an encrypted object. A naive solution for that problem is using the internal"},{"line_number":308,"context_line":"client as done with container sync."},{"line_number":309,"context_line":""},{"line_number":310,"context_line":"5.8 ETAG"},{"line_number":311,"context_line":"--------"}],"source_content_type":"text/x-rst","patch_set":6,"id":"7a016987_b51cf8c1","line":308,"updated":"2015-05-18 14:51:40.000000000","message":"For an encrypted object it might be better to treat it the same way as SLO (if I have understood that) i.e. to always handle GET storlet on the proxy - we should not assume that object servers will have access to a keymaster.\n\nFor EC I wonder what the benefit is of executing the storlet on an object server - fragments will need to be fetched from other object servers, so why not process the storlet on the proxy? (I *assume* the benefit of object server storlet execution with a replicated object is avoiding shipping the object to the proxy if for example the storlet reduces effective object size?? Or is there another benefit?)","commit_id":"0a6e3ccb786cff17f94ffe5212c3573174119570"},{"author":{"_account_id":11317,"name":"Eran Rom","email":"eran@itsonlyme.name","username":"eranr"},"change_message_id":"86cc1836ca1082ec805f96d90d4c0b281b4859ea","unresolved":false,"context_lines":[{"line_number":305,"context_line":"------------------"},{"line_number":306,"context_line":"The question with EC and encryption is how do we run a storlet over an encoded"},{"line_number":307,"context_line":"or an encrypted object. A naive solution for that problem is using the internal"},{"line_number":308,"context_line":"client as done with container sync."},{"line_number":309,"context_line":""},{"line_number":310,"context_line":"5.8 ETAG"},{"line_number":311,"context_line":"--------"}],"source_content_type":"text/x-rst","patch_set":6,"id":"7a016987_e078692d","line":308,"in_reply_to":"7a016987_b51cf8c1","updated":"2015-05-18 18:40:31.000000000","message":"Regarding encrypted object as SLO - Agree. will change accordingly.\nRegarding EC. Basically, we need to run the storlet on the proxy. An optimization can be done assuming systematic code and access to the information which are the non encoded chunks.","commit_id":"0a6e3ccb786cff17f94ffe5212c3573174119570"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"46f90d3d961c552acb2fc2f27820653ad403a339","unresolved":false,"context_lines":[{"line_number":365,"context_line":""},{"line_number":366,"context_line":"#. A GET request with X-Run-Storlet hits the proxy server."},{"line_number":367,"context_line":"#. The storlet middleware at the proxy server lets the request processing continue,"},{"line_number":368,"context_line":"   until a response is retunred. While the request is being processed it gets to the"},{"line_number":369,"context_line":"   SLO middleware, which in turn issues requests to the object servers, where they"},{"line_number":370,"context_line":"   will be ignored by the storlets middleware as they involve SLO."},{"line_number":371,"context_line":"#. As the response gets back to the storlet middleware in the proxy it calls the"}],"source_content_type":"text/x-rst","patch_set":6,"id":"7a016987_b5d758ec","line":368,"updated":"2015-05-18 14:51:40.000000000","message":"s/retunred/returned/","commit_id":"0a6e3ccb786cff17f94ffe5212c3573174119570"}]}
