)]}'
{"specs/juno/taskflow-improved-scoping.rst":[{"author":{"_account_id":4190,"name":"lifeless","email":"robertc@robertcollins.net","username":"lifeless"},"change_message_id":"07d9ad10869947ac564bfcf99eebe08505653bc2","unresolved":false,"context_lines":[{"line_number":35,"context_line":"  and a third atom also produces the same output there is required to be a"},{"line_number":36,"context_line":"  `rebinding`_  application to ensure that the right output goes into the"},{"line_number":37,"context_line":"  right input. This results in a bad user experience (and likely is not what"},{"line_number":38,"context_line":"  users expect)."},{"line_number":39,"context_line":"* Loss of nesting post-compilation. Currently we flatten a flow (which may have"},{"line_number":40,"context_line":"  nested flows) into a single execution graph composed entirely of atoms at"},{"line_number":41,"context_line":"  compile time; this makes it more easy (still not that easy) to execute"}],"source_content_type":"text/x-rst","patch_set":6,"id":"1ae5cdf2_01b4ee33","line":38,"updated":"2014-06-15 20:43:31.000000000","message":"Let me paraphase to check understanding. \u0027names are global in nature today, there is no local scope\u0027.","commit_id":"098331b5679cd8fae9a40b872bac8d7164964378"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"71b08d3309184fe4671707e563d231c6addd7244","unresolved":false,"context_lines":[{"line_number":35,"context_line":"  and a third atom also produces the same output there is required to be a"},{"line_number":36,"context_line":"  `rebinding`_  application to ensure that the right output goes into the"},{"line_number":37,"context_line":"  right input. This results in a bad user experience (and likely is not what"},{"line_number":38,"context_line":"  users expect)."},{"line_number":39,"context_line":"* Loss of nesting post-compilation. Currently we flatten a flow (which may have"},{"line_number":40,"context_line":"  nested flows) into a single execution graph composed entirely of atoms at"},{"line_number":41,"context_line":"  compile time; this makes it more easy (still not that easy) to execute"}],"source_content_type":"text/x-rst","patch_set":6,"id":"1ae5cdf2_be088cda","line":38,"in_reply_to":"1ae5cdf2_01b4ee33","updated":"2014-06-16 19:48:41.000000000","message":"Not exactly, names are somewhat global in nature, there is some limited local scope. I think a few examples will help here to make this clear for all involved to show what exists and what doesn\u0027t work out so well right. Will follow up with that to help those reading.","commit_id":"098331b5679cd8fae9a40b872bac8d7164964378"},{"author":{"_account_id":4190,"name":"lifeless","email":"robertc@robertcollins.net","username":"lifeless"},"change_message_id":"07d9ad10869947ac564bfcf99eebe08505653bc2","unresolved":false,"context_lines":[{"line_number":88,"context_line":""},{"line_number":89,"context_line":"Overall it is expected this will more closely follows what programmers have"},{"line_number":90,"context_line":"come to expect, where a function provides its own local scoping and a nested"},{"line_number":91,"context_line":"function can override that scoping with its own variable names as needed."},{"line_number":92,"context_line":""},{"line_number":93,"context_line":"Alternatives"},{"line_number":94,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":6,"id":"1ae5cdf2_21800a49","line":91,"updated":"2014-06-15 20:43:31.000000000","message":"You find the explanation for this follows more directly if you expressed in compiler speak. E.g. we\u0027ll use each node in the input hiearchy as a symbol table and the scope of a symbol will be the using node, with resolution traversing the hierarchy in a bottom up fashion.","commit_id":"098331b5679cd8fae9a40b872bac8d7164964378"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"71b08d3309184fe4671707e563d231c6addd7244","unresolved":false,"context_lines":[{"line_number":88,"context_line":""},{"line_number":89,"context_line":"Overall it is expected this will more closely follows what programmers have"},{"line_number":90,"context_line":"come to expect, where a function provides its own local scoping and a nested"},{"line_number":91,"context_line":"function can override that scoping with its own variable names as needed."},{"line_number":92,"context_line":""},{"line_number":93,"context_line":"Alternatives"},{"line_number":94,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":6,"id":"1ae5cdf2_1ef7c0d5","line":91,"in_reply_to":"1ae5cdf2_21800a49","updated":"2014-06-16 19:48:41.000000000","message":"Agreed, I wasn\u0027t sure how much compiler speak I wanted to bring out though (didn\u0027t want to add on a new set of terms if I could avoid it) but let me change and see how this goes :)","commit_id":"098331b5679cd8fae9a40b872bac8d7164964378"},{"author":{"_account_id":2472,"name":"Doug Hellmann","email":"dhellmann@redhat.com","username":"doug-hellmann"},"change_message_id":"b58a23adf2978ba68ca2698a4ee12feabfbe1210","unresolved":false,"context_lines":[{"line_number":23,"context_line":"`persistence strategies`_ (it also allows the engine to notify others about"},{"line_number":24,"context_line":"state transitions and other nifty features...)."},{"line_number":25,"context_line":""},{"line_number":26,"context_line":"**Note:** In a way this externalized state is equivalent to a"},{"line_number":27,"context_line":"workflows *memory* (without actually retaining that state on the runtime stack"},{"line_number":28,"context_line":"and/or heap). This externalization of workflow execution \u0026 state enables"},{"line_number":29,"context_line":"many innovative strategies that can be explored in the future and is one of the"}],"source_content_type":"text/x-rst","patch_set":10,"id":"1ae5cdf2_e505ca10","line":26,"updated":"2014-06-18 14:17:17.000000000","message":"There\u0027s a \"note\" directive for cases like this. It\u0027s not worth submitting a new draft just for that change.","commit_id":"91000e605873e638dfef67b74b550574723a3a1e"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"6036afd64e63942329a71db093a05c9d8d60c034","unresolved":false,"context_lines":[{"line_number":23,"context_line":"`persistence strategies`_ (it also allows the engine to notify others about"},{"line_number":24,"context_line":"state transitions and other nifty features...)."},{"line_number":25,"context_line":""},{"line_number":26,"context_line":"**Note:** In a way this externalized state is equivalent to a"},{"line_number":27,"context_line":"workflows *memory* (without actually retaining that state on the runtime stack"},{"line_number":28,"context_line":"and/or heap). This externalization of workflow execution \u0026 state enables"},{"line_number":29,"context_line":"many innovative strategies that can be explored in the future and is one of the"}],"source_content_type":"text/x-rst","patch_set":10,"id":"1ae5cdf2_06955232","line":26,"in_reply_to":"1ae5cdf2_e505ca10","updated":"2014-06-18 18:59:20.000000000","message":"Ya, I did almost use that, it just doesn\u0027t seem to render all that well outside of sphinx and I was thinking that most people would not read these rst files via sphinx but via some other plain rst viewer (http://rst.ninjs.org/ for example).","commit_id":"91000e605873e638dfef67b74b550574723a3a1e"},{"author":{"_account_id":2472,"name":"Doug Hellmann","email":"dhellmann@redhat.com","username":"doug-hellmann"},"change_message_id":"b58a23adf2978ba68ca2698a4ee12feabfbe1210","unresolved":false,"context_lines":[{"line_number":205,"context_line":"inputs) when they are located in disjoint subtrees. This also makes it possible"},{"line_number":206,"context_line":"for post-execution extraction of individual tree segements (by allowing for"},{"line_number":207,"context_line":"fetching a tree nodes symbol table and allowing for users to traverse it as"},{"line_number":208,"context_line":"they desire)."},{"line_number":209,"context_line":""},{"line_number":210,"context_line":"This first step solves the problem of ``Loss of nesting"},{"line_number":211,"context_line":"post-compilation/post-runtime`` but it does not completly solve the first"}],"source_content_type":"text/x-rst","patch_set":10,"id":"1ae5cdf2_05ad569b","line":208,"updated":"2014-06-18 14:17:17.000000000","message":"Doesn\u0027t this just move the name resolution problem to the code that is trying to extract the info from a result set? That code now needs to understand the structure of the flow, to ask for the data from the right part of the tree. That makes it more difficult to change flow structures, since anything using the flow potentially needs to be updated.","commit_id":"91000e605873e638dfef67b74b550574723a3a1e"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"6036afd64e63942329a71db093a05c9d8d60c034","unresolved":false,"context_lines":[{"line_number":205,"context_line":"inputs) when they are located in disjoint subtrees. This also makes it possible"},{"line_number":206,"context_line":"for post-execution extraction of individual tree segements (by allowing for"},{"line_number":207,"context_line":"fetching a tree nodes symbol table and allowing for users to traverse it as"},{"line_number":208,"context_line":"they desire)."},{"line_number":209,"context_line":""},{"line_number":210,"context_line":"This first step solves the problem of ``Loss of nesting"},{"line_number":211,"context_line":"post-compilation/post-runtime`` but it does not completly solve the first"}],"source_content_type":"text/x-rst","patch_set":10,"id":"1ae5cdf2_269c0e0e","line":208,"in_reply_to":"1ae5cdf2_05ad569b","updated":"2014-06-18 18:59:20.000000000","message":"Partially correct, it does add some new code in the result set extraction/finding but we already have this disconnection anyway. The compilation routine converts all flow structures into the same underlying structures (right now just a directed graph) so the separation of the original flow structure and the compiled structure is allowed to be different (the compiled structure is what is actually executed). \n\nThis makes it so that the flow structure can be adjusted by the user, but the internal execution structure can be adjusted/changed... by the engine as it desires. This lookup would be a internal engine structure change and won\u0027t be exposed to users (since the lookup is on a compiled structure that retains the constraints that the users flows stated).\n\nThe post-extraction part is just allowing users to extract results from a given nested flow (likely by name of that subflow). The user would not need to know that a tree structure was used internally for the symbols the extraction function returns.\n\nIs that more clear now (or more fuzzy?)?","commit_id":"91000e605873e638dfef67b74b550574723a3a1e"},{"author":{"_account_id":2472,"name":"Doug Hellmann","email":"dhellmann@redhat.com","username":"doug-hellmann"},"change_message_id":"b58a23adf2978ba68ca2698a4ee12feabfbe1210","unresolved":false,"context_lines":[{"line_number":221,"context_line":"  in itself and not try to require any symbols from any prior flow/atom. A"},{"line_number":222,"context_line":"  second constant (these constants can be *ORed* together to combine them in"},{"line_number":223,"context_line":"  various ways) will be ``contain\u003dPROVIDES`` which will denote that the"},{"line_number":224,"context_line":"  symbols this flow *may* produce will not be consumable by an subsequent flow."},{"line_number":225,"context_line":""},{"line_number":226,"context_line":"This new attribute (provided as a flow constructor parameter and an immutable"},{"line_number":227,"context_line":"flow attribute) will allow a flow to influence the symbol table that it"}],"source_content_type":"text/x-rst","patch_set":10,"id":"1ae5cdf2_65d7faee","line":224,"updated":"2014-06-18 14:17:17.000000000","message":"This feels a bit like Java\u0027s \"final\" declaration, where a component flow has to declare how it is able to be used in advance without knowing anything about the code that is going to try to use it. Why do you want to define a flow so other flows can\u0027t look at its outputs?","commit_id":"91000e605873e638dfef67b74b550574723a3a1e"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"6036afd64e63942329a71db093a05c9d8d60c034","unresolved":false,"context_lines":[{"line_number":221,"context_line":"  in itself and not try to require any symbols from any prior flow/atom. A"},{"line_number":222,"context_line":"  second constant (these constants can be *ORed* together to combine them in"},{"line_number":223,"context_line":"  various ways) will be ``contain\u003dPROVIDES`` which will denote that the"},{"line_number":224,"context_line":"  symbols this flow *may* produce will not be consumable by an subsequent flow."},{"line_number":225,"context_line":""},{"line_number":226,"context_line":"This new attribute (provided as a flow constructor parameter and an immutable"},{"line_number":227,"context_line":"flow attribute) will allow a flow to influence the symbol table that it"}],"source_content_type":"text/x-rst","patch_set":10,"id":"1ae5cdf2_d126c2b9","line":224,"in_reply_to":"1ae5cdf2_65d7faee","updated":"2014-06-18 18:59:20.000000000","message":"So this is for provided/required symbol name collisions.\n\nThis would allow a flow to say that it is self-contained (or only produces outputs and requires no inputs). This \u0027contain\u0027 attribute would allow a flow to state whether it should be self-contained and not expose its produced symbols to others (which will avoid symbol conflicts for symbols that are produced by other flows with the same name that may be added into a larger flow).\n\nIt is slightly like the \u0027final\u0027 declaration I suppose in a way.\n\nBasically the idea is that the 2 initial options (PROVIDES, REQUIRES) are basic settings that can be expanded to more in the future (perhaps a flow produces a lot of output symbols but only wants to expose 2 of them, it could be expanded to allow for this in the future).\n\nAnother idea that just popped into my mind is that this could itself be an object that does this hiding that is wrapped around a flow.\n\nFor example:\n\n f \u003d linear_flow.Flow(\u0027root\u0027)\n f.add(linear_flow.Flow(\"something\").add(task that provides [a, b, c] symbols))\n\nTo avoid naming conflicts with other added root tasks/flows (that could be added after the \u0027something\u0027 flow) we could do something like the following:\n\n f \u003d linear_flow.Flow(\u0027root\u0027)\n f.add(\n    NullifyProvided(\n        linear_flow.Flow(\"something\").add(task that provides [a, b, c] symbols))\n\nThis new wrapper \u0027NullifyProvided\u0027 (name can be changed) would be wrapped around flows that the root flow user would like to discard names from (the way this can be done is just an example). \n\nThis might be more natural?","commit_id":"91000e605873e638dfef67b74b550574723a3a1e"},{"author":{"_account_id":2472,"name":"Doug Hellmann","email":"dhellmann@redhat.com","username":"doug-hellmann"},"change_message_id":"b58a23adf2978ba68ca2698a4ee12feabfbe1210","unresolved":false,"context_lines":[{"line_number":254,"context_line":"overall workflow). Without this change it would likely mean that there would be"},{"line_number":255,"context_line":"various functions created by users that would have *messy* and"},{"line_number":256,"context_line":"*complicated* symbol renaming algorithms to resolve the issue that taskflow"},{"line_number":257,"context_line":"should instead resolve itself. This is bad..."},{"line_number":258,"context_line":""},{"line_number":259,"context_line":"Impact on Existing APIs"},{"line_number":260,"context_line":"-----------------------"}],"source_content_type":"text/x-rst","patch_set":10,"id":"1ae5cdf2_40e4ac0c","line":257,"updated":"2014-06-18 14:17:17.000000000","message":"Do flows have names? Would another alternative be to require name scoping using the flow name to access symbols that occur more than once? Could some of that scoping be done when passing values between flows (look backwards through the flows in a defined order, instead of having a single global namespace)?","commit_id":"91000e605873e638dfef67b74b550574723a3a1e"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"fc395d84521657b5777da9c54807bee02978f3e4","unresolved":false,"context_lines":[{"line_number":254,"context_line":"overall workflow). Without this change it would likely mean that there would be"},{"line_number":255,"context_line":"various functions created by users that would have *messy* and"},{"line_number":256,"context_line":"*complicated* symbol renaming algorithms to resolve the issue that taskflow"},{"line_number":257,"context_line":"should instead resolve itself. This is bad..."},{"line_number":258,"context_line":""},{"line_number":259,"context_line":"Impact on Existing APIs"},{"line_number":260,"context_line":"-----------------------"}],"source_content_type":"text/x-rst","patch_set":10,"id":"dab895d6_8d2bd12f","line":257,"in_reply_to":"1ae5cdf2_19042317","updated":"2014-06-29 06:22:51.000000000","message":"So just to clear up the how they are used.\n\nCurrently tasks/atoms in taskflow can declare \u0027symbols\u0027 they want as input (those symbols are bound to some value), these bound symbols are provided to the tasks execute() method automatically. The task can also produce its own output symbols (with bound values) as a result of the execute() method. The taskflow engine is responsible for binding these values and saving them and finding them before calling execute (and after calling execute).\n\nI think we are thinking the same thing here. The symbol tables and tree lookup are the compiled version of the nesting level lookup that u described (each nesting level/nested flow would have its own symbol table).\n\nMaybe just some misunderstanding here, currently we make it impossible to compose flows if they introduce symbol naming conflicts (this error is detected at execution time), so that functionality already exists in the existing taskflow. So this new proposed mechanism would move from the one big namespace to a hierachy of namespaces. The rest as you described it (anything needed by a nested flow has to come in from parent flow) already exists; it\u0027s just the symbol names must be unique in the whole namespace, which is just a naming/namespace problem and is disconnected from how the passing in \u0026 out occurs.\n\nThe idea for taskflow is there is a user representation of the flows, that is they use the pattern concept (http://docs.openstack.org/developer/taskflow/patterns.html) and they compose them however they want, and there is a runtime representation of that same flow (which may not be the same representation, but must obey the same logical constraints). This adjustment would be at the runtime representation, which is enforcing the naming conflict logic. Fixing the naming conflict runtime representation would then make the user representation less restrictive (the user representation would not have to care about symbol naming uniqueness).","commit_id":"91000e605873e638dfef67b74b550574723a3a1e"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"6036afd64e63942329a71db093a05c9d8d60c034","unresolved":false,"context_lines":[{"line_number":254,"context_line":"overall workflow). Without this change it would likely mean that there would be"},{"line_number":255,"context_line":"various functions created by users that would have *messy* and"},{"line_number":256,"context_line":"*complicated* symbol renaming algorithms to resolve the issue that taskflow"},{"line_number":257,"context_line":"should instead resolve itself. This is bad..."},{"line_number":258,"context_line":""},{"line_number":259,"context_line":"Impact on Existing APIs"},{"line_number":260,"context_line":"-----------------------"}],"source_content_type":"text/x-rst","patch_set":10,"id":"1ae5cdf2_71924e58","line":257,"in_reply_to":"1ae5cdf2_40e4ac0c","updated":"2014-06-18 18:59:20.000000000","message":"Flows and tasks are currently required to have names (which is good, names are helpful in knowing whats going on, and if users can\u0027t associate a name then it likely means they aren\u0027t thinking about what the unit of work is that they are creating).\n\nI think your way could work, the \u0027looking backward\u0027 part is similar to the tree structure defined here (since the looking backward would be traversing the tree upwards).\n\nI don\u0027t think though we can force a task/flow to access symbols by declaring what flow produces them; this makes it hard to compose arbitrary workflows from other sources.\n\nFor example:\n\n f \u003d linear_flow.Flow(\u0027root-volume-create\u0027)\n\n if driver \u003d\u003d \u0027nexenta\u0027:\n   f.add(driver.get_create_flow())\n else:\n   ...\n\n f.add(linear_flow.Flow(\u0027verify-created\u0027))\n f.add(linear_flow.Flow(\u0027start-volume\u0027))\n ...\n\nThe above driver.get_create_flow() would be unaware that it is being added to the \u0027root\u0027 flow so it wouldn\u0027t be able to refer to symbols that \u0027root-volume-create\u0027 creates if we forced name scoping to be used.\n\nWe could of course allow for name scoping and default to not using name scoping (allowing both seems feasible), but I think this can be a later change that limits the parts of the tree that are searched upwards for symbols to just the names that are wanted.","commit_id":"91000e605873e638dfef67b74b550574723a3a1e"},{"author":{"_account_id":2472,"name":"Doug Hellmann","email":"dhellmann@redhat.com","username":"doug-hellmann"},"change_message_id":"b06cfce39daa345c7ac8e762e931d0b96a163977","unresolved":false,"context_lines":[{"line_number":254,"context_line":"overall workflow). Without this change it would likely mean that there would be"},{"line_number":255,"context_line":"various functions created by users that would have *messy* and"},{"line_number":256,"context_line":"*complicated* symbol renaming algorithms to resolve the issue that taskflow"},{"line_number":257,"context_line":"should instead resolve itself. This is bad..."},{"line_number":258,"context_line":""},{"line_number":259,"context_line":"Impact on Existing APIs"},{"line_number":260,"context_line":"-----------------------"}],"source_content_type":"text/x-rst","patch_set":10,"id":"1ae5cdf2_19042317","line":257,"in_reply_to":"1ae5cdf2_71924e58","updated":"2014-06-24 14:48:20.000000000","message":"Maybe I don\u0027t understand how these values are being used.\n\nIt feels like there\u0027s one big namespace, into which flows can publish values and out of which they can request values be given to them. The single namespace is causing problems because multiple flows are using the same names for their inputs or outputs and they collide. I think you are proposing to make it impossible to compose flows if they would introduce naming conflicts and/or to use a more complicated symbol table lookup to allow flows to access values exposed by other flows while still sort of having that single namespace.\n\nMy suggestion was to give each flow its own namespace, in a way that hides the flow implementation details from the user of the flow. Anything returned by a nested flow would have to be passed out through the parent flow. Anything needed by a nested flow would have to be passed in by the parent flow. Flows would only have access to values from other flows at their same nesting level, which reduces the namespace issue.\n\nI don\u0027t know if that would work, or how big of a change that would be to existing flows. Is it a realistic approach?","commit_id":"91000e605873e638dfef67b74b550574723a3a1e"},{"author":{"_account_id":1297,"name":"Joshua Harlow","email":"jxharlow@godaddy.com","username":"harlowja"},"change_message_id":"66d00a64830f3d555318ff8e664186e33ef9c008","unresolved":false,"context_lines":[{"line_number":254,"context_line":"overall workflow). Without this change it would likely mean that there would be"},{"line_number":255,"context_line":"various functions created by users that would have *messy* and"},{"line_number":256,"context_line":"*complicated* symbol renaming algorithms to resolve the issue that taskflow"},{"line_number":257,"context_line":"should instead resolve itself. This is bad..."},{"line_number":258,"context_line":""},{"line_number":259,"context_line":"Impact on Existing APIs"},{"line_number":260,"context_line":"-----------------------"}],"source_content_type":"text/x-rst","patch_set":10,"id":"dab895d6_6db4bdf2","line":257,"in_reply_to":"dab895d6_8d2bd12f","updated":"2014-06-29 06:27:42.000000000","message":"Let me adjust the spec and see if it can be made more clear here.","commit_id":"91000e605873e638dfef67b74b550574723a3a1e"}]}