)]}'
{"doc/source/userdoc/edp.rst":[{"author":{"_account_id":8091,"name":"Trevor McKay","email":"tmckay@redhat.com","username":"tmckay"},"change_message_id":"cfa25876a05884bdfdc289280556c8c0c681bac2","unresolved":false,"context_lines":[{"line_number":6,"context_line":""},{"line_number":7,"context_line":"Sahara\u0027s Elastic Data Processing facility or :dfn:`EDP` allows the execution of jobs on clusters created from Sahara. EDP supports:"},{"line_number":8,"context_line":""},{"line_number":9,"context_line":"* Hive, Pig, MapReduce, MapReduce.Streaming, Java, and Oozie Shell job types on Hadoop clusters"},{"line_number":10,"context_line":"* Spark jobs on Spark standalone clusters"},{"line_number":11,"context_line":"* storage of job binaries in Swift or Sahara\u0027s own database"},{"line_number":12,"context_line":"* access to input and output data sources in"}],"source_content_type":"text/x-rst","patch_set":1,"id":"ba7be1f8_5f5f8172","line":9,"updated":"2015-02-27 17:29:31.000000000","message":"I would just change this to \"Shell\" and drop Oozie everywhere. I know Oozie was mentioned in the spec ... maybe it shouldn\u0027t have been.\n\nWe can implement a Shell action for any cluster type most likely, and support that from another EDP engine (probably just a thin wrapper around an ssh launch).\n\nSo, the job type and the associated docs should be neutral on it.  There may well be other places where too much Ooziness has come through.","commit_id":"b6b0b8e2d10841148753e6c6657af0b396608d33"},{"author":{"_account_id":13953,"name":"Elise Gafford","email":"egafford@redhat.com","username":"egafford"},"change_message_id":"ef10df1a667c7178e6d627a492a8163cbfb6fa34","unresolved":false,"context_lines":[{"line_number":6,"context_line":""},{"line_number":7,"context_line":"Sahara\u0027s Elastic Data Processing facility or :dfn:`EDP` allows the execution of jobs on clusters created from Sahara. EDP supports:"},{"line_number":8,"context_line":""},{"line_number":9,"context_line":"* Hive, Pig, MapReduce, MapReduce.Streaming, Java, and Oozie Shell job types on Hadoop clusters"},{"line_number":10,"context_line":"* Spark jobs on Spark standalone clusters"},{"line_number":11,"context_line":"* storage of job binaries in Swift or Sahara\u0027s own database"},{"line_number":12,"context_line":"* access to input and output data sources in"}],"source_content_type":"text/x-rst","patch_set":1,"id":"ba7be1f8_7fafe52b","line":9,"in_reply_to":"ba7be1f8_5f5f8172","updated":"2015-02-27 17:41:36.000000000","message":"Reasonable! Will fix in next patch. My only question there is: in the direct engine case, what will \"config\" type configuration kvps become?","commit_id":"b6b0b8e2d10841148753e6c6657af0b396608d33"},{"author":{"_account_id":8091,"name":"Trevor McKay","email":"tmckay@redhat.com","username":"tmckay"},"change_message_id":"94b077904ddb4f2dc64c468c4909b1e104cb037a","unresolved":false,"context_lines":[{"line_number":254,"context_line":"in the same working directory (on both the filesystem and in HDFS). Command line arguments may be passed to"},{"line_number":255,"context_line":"the script through the ``args`` array, and any ``params`` values will be passed as environment variables."},{"line_number":256,"context_line":""},{"line_number":257,"context_line":"Data Source objects are not used with Shell job types."},{"line_number":258,"context_line":""},{"line_number":259,"context_line":"The ``edp-shell`` example bundled with Sahara contains a script which will output the executing user to"},{"line_number":260,"context_line":"a file specified by the first command line argument."}],"source_content_type":"text/x-rst","patch_set":6,"id":"9a80dd14_0dfc233e","line":257,"updated":"2015-03-04 15:36:53.000000000","message":"Note, we need a section in the EDP docs describing the data_source reference substitution support in general.  But this is fine for now. We can update docs after the freeze.","commit_id":"b135a8eb003c6c9c245398885ccf18078ef8bbfb"}],"etc/edp-examples/json-api-examples/v1.1/README.rst":[{"author":{"_account_id":12707,"name":"Jacob Bin Wang","email":"wangbin@awcloud.com","username":"Jacob"},"change_message_id":"99ac2aa65098c7d0f5c2ed87dd59c179b34ca4a3","unresolved":false,"context_lines":[{"line_number":220,"context_line":""},{"line_number":221,"context_line":"This example assumes the following:"},{"line_number":222,"context_line":""},{"line_number":223,"context_line":"1. Usage of an OpenStack user named \"demo\", with password \"password.\""},{"line_number":224,"context_line":"2. An active Hadoop cluster exists in the demo user\u0027s project."},{"line_number":225,"context_line":""},{"line_number":226,"context_line":"Steps"}],"source_content_type":"text/x-rst","patch_set":2,"id":"ba7be1f8_e4e6da15","line":223,"updated":"2015-03-01 02:20:17.000000000","message":".\" -\u003e \".","commit_id":"b5ce045d17e402667bcd6c914b35f62238b788b6"},{"author":{"_account_id":13953,"name":"Elise Gafford","email":"egafford@redhat.com","username":"egafford"},"change_message_id":"bf8e3a07faa4a46b71f17c920bd3e645a454beb1","unresolved":false,"context_lines":[{"line_number":220,"context_line":""},{"line_number":221,"context_line":"This example assumes the following:"},{"line_number":222,"context_line":""},{"line_number":223,"context_line":"1. Usage of an OpenStack user named \"demo\", with password \"password.\""},{"line_number":224,"context_line":"2. An active Hadoop cluster exists in the demo user\u0027s project."},{"line_number":225,"context_line":""},{"line_number":226,"context_line":"Steps"}],"source_content_type":"text/x-rst","patch_set":2,"id":"ba7be1f8_beba03df","line":223,"in_reply_to":"ba7be1f8_e4e6da15","updated":"2015-03-02 15:22:48.000000000","message":"Done","commit_id":"b5ce045d17e402667bcd6c914b35f62238b788b6"},{"author":{"_account_id":10670,"name":"Michael McCune","email":"elmiko@redhat.com","username":"mimccune"},"change_message_id":"85b2835e0e28fe9fbbfca196c78e381b384300bb","unresolved":false,"context_lines":[{"line_number":225,"context_line":""},{"line_number":226,"context_line":"Steps"},{"line_number":227,"context_line":"-----"},{"line_number":228,"context_line":""},{"line_number":229,"context_line":"1. **Script File**: Put the file at"},{"line_number":230,"context_line":"   ``edp-examples/edp-shell/shell-example.sh`` into your Sahara endpoint at"},{"line_number":231,"context_line":"   path ``job-binary-internals/shell-example.sh``. Note the new object\u0027s id."}],"source_content_type":"text/x-rst","patch_set":2,"id":"ba7be1f8_2bd6fac3","line":228,"updated":"2015-02-28 15:14:30.000000000","message":"i think it would be nice in this section to make the HTTP methods all caps, for example \"PUT the file at ....\". just to be explicit about what is going on.","commit_id":"b5ce045d17e402667bcd6c914b35f62238b788b6"},{"author":{"_account_id":13953,"name":"Elise Gafford","email":"egafford@redhat.com","username":"egafford"},"change_message_id":"bf8e3a07faa4a46b71f17c920bd3e645a454beb1","unresolved":false,"context_lines":[{"line_number":225,"context_line":""},{"line_number":226,"context_line":"Steps"},{"line_number":227,"context_line":"-----"},{"line_number":228,"context_line":""},{"line_number":229,"context_line":"1. **Script File**: Put the file at"},{"line_number":230,"context_line":"   ``edp-examples/edp-shell/shell-example.sh`` into your Sahara endpoint at"},{"line_number":231,"context_line":"   path ``job-binary-internals/shell-example.sh``. Note the new object\u0027s id."}],"source_content_type":"text/x-rst","patch_set":2,"id":"ba7be1f8_3ee7f313","line":228,"in_reply_to":"ba7be1f8_2bd6fac3","updated":"2015-03-02 15:22:48.000000000","message":"Good call; I\u0027ll change those throughout the doc.","commit_id":"b5ce045d17e402667bcd6c914b35f62238b788b6"},{"author":{"_account_id":8090,"name":"Chad Roberts","email":"croberts@redhat.com","username":"croberts"},"change_message_id":"ca1954dce85c69e51cca4a402c6fd97aba430702","unresolved":false,"context_lines":[{"line_number":221,"context_line":"This example assumes the following:"},{"line_number":222,"context_line":""},{"line_number":223,"context_line":"1. Usage of an OpenStack user named \"demo\", with password \"password\"."},{"line_number":224,"context_line":"2. An active Hadoop cluster exists in the demo user\u0027s project."},{"line_number":225,"context_line":""},{"line_number":226,"context_line":"Steps"},{"line_number":227,"context_line":"-----"}],"source_content_type":"text/x-rst","patch_set":4,"id":"ba7be1f8_8fd312d4","line":224,"updated":"2015-03-03 15:22:44.000000000","message":"Should we mention that Oozie is also required? I think it is technically possible to start a vanilla cluster without Oozie.","commit_id":"4722894841ba6fb5fd023d093b1c4069d4dfe52a"},{"author":{"_account_id":13953,"name":"Elise Gafford","email":"egafford@redhat.com","username":"egafford"},"change_message_id":"4e7d8da73b74929275085285f75738d69184f9b0","unresolved":false,"context_lines":[{"line_number":221,"context_line":"This example assumes the following:"},{"line_number":222,"context_line":""},{"line_number":223,"context_line":"1. Usage of an OpenStack user named \"demo\", with password \"password\"."},{"line_number":224,"context_line":"2. An active Hadoop cluster exists in the demo user\u0027s project."},{"line_number":225,"context_line":""},{"line_number":226,"context_line":"Steps"},{"line_number":227,"context_line":"-----"}],"source_content_type":"text/x-rst","patch_set":4,"id":"ba7be1f8_6a232c58","line":224,"in_reply_to":"ba7be1f8_8fd312d4","updated":"2015-03-03 15:46:50.000000000","message":"So in terms of this, I am of two minds. Early in the progression of this review, I was (validly) advised by tmckay to avoid mention of Oozie-specificity in this feature, as the Shell job type may be expanded to other engines.\n\nI believe it\u0027s also true that at present, Oozie is required to run EDP in Hadoop in the general case, so (if we want to act on your comment, which we really might,) then we should put that precondition almost everywhere.\n\nOther thoughts? I\u0027m happy to add or not; no strong opinion here.","commit_id":"4722894841ba6fb5fd023d093b1c4069d4dfe52a"},{"author":{"_account_id":13953,"name":"Elise Gafford","email":"egafford@redhat.com","username":"egafford"},"change_message_id":"31037020692c6aac197df8e9c0a40623ce07d726","unresolved":false,"context_lines":[{"line_number":221,"context_line":"This example assumes the following:"},{"line_number":222,"context_line":""},{"line_number":223,"context_line":"1. Usage of an OpenStack user named \"demo\", with password \"password\"."},{"line_number":224,"context_line":"2. An active Hadoop cluster exists in the demo user\u0027s project."},{"line_number":225,"context_line":""},{"line_number":226,"context_line":"Steps"},{"line_number":227,"context_line":"-----"}],"source_content_type":"text/x-rst","patch_set":4,"id":"9a80dd14_cdae8be7","line":224,"in_reply_to":"ba7be1f8_8fd312d4","updated":"2015-03-04 15:03:42.000000000","message":"To meet your requests for clarification and tmckay\u0027s that Shell not be tied to Oozie overmuch in doc, I added a global note re: Oozie in patch set 6. Thanks Chad.","commit_id":"4722894841ba6fb5fd023d093b1c4069d4dfe52a"}],"sahara/conductor/api.py":[{"author":{"_account_id":7710,"name":"Sergey Reshetnyak","email":"sreshetniak@mirantis.com","username":"sreshetniak"},"change_message_id":"01de8f24f92506a4cfbe08f15c12224439fe90d0","unresolved":false,"context_lines":[{"line_number":358,"context_line":"        :param job: This is expected to be a Job object"},{"line_number":359,"context_line":"        \"\"\""},{"line_number":360,"context_line":"        lib_ids \u003d job.libs or []"},{"line_number":361,"context_line":"        binaries \u003d filter(None, (self.job_binary_get(context, lib_id)"},{"line_number":362,"context_line":"                                 for lib_id in lib_ids))"},{"line_number":363,"context_line":"        return [binary[\"name\"] for binary in binaries]"},{"line_number":364,"context_line":""}],"source_content_type":"text/x-python","patch_set":2,"id":"ba7be1f8_a86749e9","line":361,"updated":"2015-03-01 07:18:37.000000000","message":"maybe should use map?","commit_id":"b5ce045d17e402667bcd6c914b35f62238b788b6"},{"author":{"_account_id":13953,"name":"Elise Gafford","email":"egafford@redhat.com","username":"egafford"},"change_message_id":"bf8e3a07faa4a46b71f17c920bd3e645a454beb1","unresolved":false,"context_lines":[{"line_number":358,"context_line":"        :param job: This is expected to be a Job object"},{"line_number":359,"context_line":"        \"\"\""},{"line_number":360,"context_line":"        lib_ids \u003d job.libs or []"},{"line_number":361,"context_line":"        binaries \u003d filter(None, (self.job_binary_get(context, lib_id)"},{"line_number":362,"context_line":"                                 for lib_id in lib_ids))"},{"line_number":363,"context_line":"        return [binary[\"name\"] for binary in binaries]"},{"line_number":364,"context_line":""}],"source_content_type":"text/x-python","patch_set":2,"id":"ba7be1f8_fe839bc9","line":361,"in_reply_to":"ba7be1f8_a86749e9","updated":"2015-03-02 15:22:48.000000000","message":"I\u0027d already convinced myself that I needed to make this clearer. I don\u0027t love map either (this does have to be a filter operation on the objects\u0027 bool conversions,) but I agree; this will be changed.","commit_id":"b5ce045d17e402667bcd6c914b35f62238b788b6"}],"sahara/service/edp/oozie/engine.py":[{"author":{"_account_id":12038,"name":"Vitaly Gridnev","email":"gridnevvvit@gmail.com","username":"vgridnev"},"change_message_id":"3a05ffa433a273874743d160216a4bff110d5bf7","unresolved":false,"context_lines":[{"line_number":178,"context_line":""},{"line_number":179,"context_line":"    def validate_job_execution(self, cluster, job, data):"},{"line_number":180,"context_line":"        # Shell job type requires no specific fields"},{"line_number":181,"context_line":"        if job.type in [edp.JOB_TYPE_SHELL]:"},{"line_number":182,"context_line":"            return"},{"line_number":183,"context_line":"        # All other types except Java require input and output"},{"line_number":184,"context_line":"        # objects and Java require main class"}],"source_content_type":"text/x-python","patch_set":5,"id":"ba7be1f8_063296a6","line":181,"range":{"start_line":181,"start_character":0,"end_line":181,"end_character":1},"updated":"2015-03-03 22:12:11.000000000","message":"Hm, strange. Why not just job.type \u003d\u003d edp.JOB_TYPE_SHELL?","commit_id":"6cda468718b4c66c7e073956166ecafba3649ca7"},{"author":{"_account_id":8091,"name":"Trevor McKay","email":"tmckay@redhat.com","username":"tmckay"},"change_message_id":"3c823328baa647279a32d9842ea0603f5bc02473","unresolved":false,"context_lines":[{"line_number":178,"context_line":""},{"line_number":179,"context_line":"    def validate_job_execution(self, cluster, job, data):"},{"line_number":180,"context_line":"        # Shell job type requires no specific fields"},{"line_number":181,"context_line":"        if job.type in [edp.JOB_TYPE_SHELL]:"},{"line_number":182,"context_line":"            return"},{"line_number":183,"context_line":"        # All other types except Java require input and output"},{"line_number":184,"context_line":"        # objects and Java require main class"}],"source_content_type":"text/x-python","patch_set":5,"id":"9a80dd14_53ba3ae2","line":181,"in_reply_to":"9a80dd14_dfdab1e8","updated":"2015-03-04 15:36:29.000000000","message":"By the way, I believe this was here because the method was moved in the past.\nAt one point it was common, and the \"in\" applied to Java and Spark.\n\nWhoever did the refactor missed changing the \"in\"","commit_id":"6cda468718b4c66c7e073956166ecafba3649ca7"},{"author":{"_account_id":13953,"name":"Elise Gafford","email":"egafford@redhat.com","username":"egafford"},"change_message_id":"091927a069cc6ae0443db4f0bc00239075ceb4bf","unresolved":false,"context_lines":[{"line_number":178,"context_line":""},{"line_number":179,"context_line":"    def validate_job_execution(self, cluster, job, data):"},{"line_number":180,"context_line":"        # Shell job type requires no specific fields"},{"line_number":181,"context_line":"        if job.type in [edp.JOB_TYPE_SHELL]:"},{"line_number":182,"context_line":"            return"},{"line_number":183,"context_line":"        # All other types except Java require input and output"},{"line_number":184,"context_line":"        # objects and Java require main class"}],"source_content_type":"text/x-python","patch_set":5,"id":"9a80dd14_dfdab1e8","line":181,"in_reply_to":"ba7be1f8_063296a6","updated":"2015-03-04 15:03:52.000000000","message":"Fixing this (in both clauses.) Thanks Vitaly.","commit_id":"6cda468718b4c66c7e073956166ecafba3649ca7"},{"author":{"_account_id":13953,"name":"Elise Gafford","email":"egafford@redhat.com","username":"egafford"},"change_message_id":"5d702b74ffed75ed340b9ec487f9e9d823624398","unresolved":false,"context_lines":[{"line_number":178,"context_line":""},{"line_number":179,"context_line":"    def validate_job_execution(self, cluster, job, data):"},{"line_number":180,"context_line":"        # Shell job type requires no specific fields"},{"line_number":181,"context_line":"        if job.type in [edp.JOB_TYPE_SHELL]:"},{"line_number":182,"context_line":"            return"},{"line_number":183,"context_line":"        # All other types except Java require input and output"},{"line_number":184,"context_line":"        # objects and Java require main class"}],"source_content_type":"text/x-python","patch_set":5,"id":"ba7be1f8_868686b7","line":181,"in_reply_to":"ba7be1f8_063296a6","updated":"2015-03-03 22:20:29.000000000","message":"Yeah, that\u0027s a pretty good question. I was following form from the next block, which I thought was probably trying to leave itself open to extension without changes in form, so it didn\u0027t bother me. If we need another patch set I\u0027ll change this.","commit_id":"6cda468718b4c66c7e073956166ecafba3649ca7"},{"author":{"_account_id":10670,"name":"Michael McCune","email":"elmiko@redhat.com","username":"mimccune"},"change_message_id":"55ce26b5977777f8e8d939ab500161f7dfeefe1a","unresolved":false,"context_lines":[{"line_number":178,"context_line":""},{"line_number":179,"context_line":"    def validate_job_execution(self, cluster, job, data):"},{"line_number":180,"context_line":"        # Shell job type requires no specific fields"},{"line_number":181,"context_line":"        if job.type in [edp.JOB_TYPE_SHELL]:"},{"line_number":182,"context_line":"            return"},{"line_number":183,"context_line":"        # All other types except Java require input and output"},{"line_number":184,"context_line":"        # objects and Java require main class"}],"source_content_type":"text/x-python","patch_set":5,"id":"ba7be1f8_67f509ec","line":181,"in_reply_to":"ba7be1f8_868686b7","updated":"2015-03-04 00:52:29.000000000","message":"agreed that it seems a little odd, but i guess it\u0027s mostly harmless here","commit_id":"6cda468718b4c66c7e073956166ecafba3649ca7"},{"author":{"_account_id":8091,"name":"Trevor McKay","email":"tmckay@redhat.com","username":"tmckay"},"change_message_id":"94b077904ddb4f2dc64c468c4909b1e104cb037a","unresolved":false,"context_lines":[{"line_number":213,"context_line":"        uploaded_paths \u003d []"},{"line_number":214,"context_line":"        hdfs_user \u003d self.get_hdfs_user()"},{"line_number":215,"context_line":"        job_dir_suffix \u003d \u0027/lib\u0027 if job.type !\u003d edp.JOB_TYPE_SHELL else \u0027\u0027"},{"line_number":216,"context_line":"        lib_dir \u003d job_dir + job_dir_suffix"},{"line_number":217,"context_line":""},{"line_number":218,"context_line":"        with remote.get_remote(where) as r:"},{"line_number":219,"context_line":"            for main in mains:"}],"source_content_type":"text/x-python","patch_set":6,"id":"9a80dd14_c8e67916","line":216,"updated":"2015-03-04 15:36:53.000000000","message":"should we use os.path.join() here?","commit_id":"b135a8eb003c6c9c245398885ccf18078ef8bbfb"},{"author":{"_account_id":13953,"name":"Elise Gafford","email":"egafford@redhat.com","username":"egafford"},"change_message_id":"74a94a64854c921ff3d213c6d835df2009ac0b86","unresolved":false,"context_lines":[{"line_number":213,"context_line":"        uploaded_paths \u003d []"},{"line_number":214,"context_line":"        hdfs_user \u003d self.get_hdfs_user()"},{"line_number":215,"context_line":"        job_dir_suffix \u003d \u0027/lib\u0027 if job.type !\u003d edp.JOB_TYPE_SHELL else \u0027\u0027"},{"line_number":216,"context_line":"        lib_dir \u003d job_dir + job_dir_suffix"},{"line_number":217,"context_line":""},{"line_number":218,"context_line":"        with remote.get_remote(where) as r:"},{"line_number":219,"context_line":"            for main in mains:"}],"source_content_type":"text/x-python","patch_set":6,"id":"9a80dd14_ce5da14a","line":216,"in_reply_to":"9a80dd14_c8e67916","updated":"2015-03-04 16:09:06.000000000","message":"Done","commit_id":"b135a8eb003c6c9c245398885ccf18078ef8bbfb"}],"sahara/service/edp/oozie/workflow_creator/shell_workflow.py":[{"author":{"_account_id":12038,"name":"Vitaly Gridnev","email":"gridnevvvit@gmail.com","username":"vgridnev"},"change_message_id":"b28712d35d8de01a27e3f68e1ba21877366c1e99","unresolved":false,"context_lines":[{"line_number":30,"context_line":"        x.add_attributes_to_element(self.doc, self.tag_name, self.SHELL_XMLNS)"},{"line_number":31,"context_line":""},{"line_number":32,"context_line":"        for k in sorted(prepare):"},{"line_number":33,"context_line":"            self._add_to_prepare_element(k, prepare[k])"},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"        self._add_configuration_elements(configuration)"},{"line_number":36,"context_line":""}],"source_content_type":"text/x-python","patch_set":1,"id":"ba7be1f8_7953856c","line":33,"updated":"2015-02-27 16:57:35.000000000","message":"Is it possible to rename variables to make code more clear? What is k here, for example?","commit_id":"b6b0b8e2d10841148753e6c6657af0b396608d33"},{"author":{"_account_id":13953,"name":"Elise Gafford","email":"egafford@redhat.com","username":"egafford"},"change_message_id":"a3e99221dd6895d9388b91fed366e06707542f30","unresolved":false,"context_lines":[{"line_number":30,"context_line":"        x.add_attributes_to_element(self.doc, self.tag_name, self.SHELL_XMLNS)"},{"line_number":31,"context_line":""},{"line_number":32,"context_line":"        for k in sorted(prepare):"},{"line_number":33,"context_line":"            self._add_to_prepare_element(k, prepare[k])"},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"        self._add_configuration_elements(configuration)"},{"line_number":36,"context_line":""}],"source_content_type":"text/x-python","patch_set":1,"id":"ba7be1f8_fc2be330","line":33,"in_reply_to":"ba7be1f8_7953856c","updated":"2015-02-27 17:06:28.000000000","message":"Hi Vitaly,\n\nThis line is present in all current workflow modules, so I went with the prevailing wind. I agree it\u0027s not the clearest thing in the world, though I think that the overall effect is clear enough (sort an unordered dict by key, then add the kvps in that order.)\n\nTherefore, I don\u0027t see this comment as particularly relevant to this changeset, or that this is so unclear that it must be changed (though in general, one-letter vars do bother me too. :) )","commit_id":"b6b0b8e2d10841148753e6c6657af0b396608d33"}],"sahara/service/edp/oozie/workflow_creator/workflow_factory.py":[{"author":{"_account_id":12707,"name":"Jacob Bin Wang","email":"wangbin@awcloud.com","username":"Jacob"},"change_message_id":"99ac2aa65098c7d0f5c2ed87dd59c179b34ca4a3","unresolved":false,"context_lines":[{"line_number":230,"context_line":"        return configs"},{"line_number":231,"context_line":""},{"line_number":232,"context_line":"    def get_workflow_xml(self, cluster, job_configs, *args, **kwargs):"},{"line_number":233,"context_line":""},{"line_number":234,"context_line":"        proxy_configs \u003d job_configs.get(\u0027proxy_configs\u0027)"},{"line_number":235,"context_line":"        job_dict \u003d {\u0027configs\u0027: self.get_configs(proxy_configs\u003dproxy_configs),"},{"line_number":236,"context_line":"                    \u0027args\u0027: []}"}],"source_content_type":"text/x-python","patch_set":2,"id":"ba7be1f8_221298b4","line":233,"updated":"2015-03-01 02:20:17.000000000","message":"Extra newline?","commit_id":"b5ce045d17e402667bcd6c914b35f62238b788b6"},{"author":{"_account_id":13953,"name":"Elise Gafford","email":"egafford@redhat.com","username":"egafford"},"change_message_id":"bf8e3a07faa4a46b71f17c920bd3e645a454beb1","unresolved":false,"context_lines":[{"line_number":230,"context_line":"        return configs"},{"line_number":231,"context_line":""},{"line_number":232,"context_line":"    def get_workflow_xml(self, cluster, job_configs, *args, **kwargs):"},{"line_number":233,"context_line":""},{"line_number":234,"context_line":"        proxy_configs \u003d job_configs.get(\u0027proxy_configs\u0027)"},{"line_number":235,"context_line":"        job_dict \u003d {\u0027configs\u0027: self.get_configs(proxy_configs\u003dproxy_configs),"},{"line_number":236,"context_line":"                    \u0027args\u0027: []}"}],"source_content_type":"text/x-python","patch_set":2,"id":"ba7be1f8_be31230d","line":233,"in_reply_to":"ba7be1f8_221298b4","updated":"2015-03-02 15:22:48.000000000","message":"Done","commit_id":"b5ce045d17e402667bcd6c914b35f62238b788b6"}],"sahara/tests/integration/tests/edp.py":[{"author":{"_account_id":12707,"name":"Jacob Bin Wang","email":"wangbin@awcloud.com","username":"Jacob"},"change_message_id":"99ac2aa65098c7d0f5c2ed87dd59c179b34ca4a3","unresolved":false,"context_lines":[{"line_number":100,"context_line":"        }"},{"line_number":101,"context_line":""},{"line_number":102,"context_line":"    def read_shell_example_script(self):"},{"line_number":103,"context_line":"        return open(self.SHELL_PATH + \u0027shell-example.sh\u0027)"},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"    def read_shell_example_text_file(self):"},{"line_number":106,"context_line":"        return open(self.SHELL_PATH + \u0027shell-example.txt\u0027)"}],"source_content_type":"text/x-python","patch_set":2,"id":"ba7be1f8_e276c005","line":103,"updated":"2015-03-01 02:20:17.000000000","message":"Why without .read()?","commit_id":"b5ce045d17e402667bcd6c914b35f62238b788b6"},{"author":{"_account_id":13953,"name":"Elise Gafford","email":"egafford@redhat.com","username":"egafford"},"change_message_id":"bf8e3a07faa4a46b71f17c920bd3e645a454beb1","unresolved":false,"context_lines":[{"line_number":100,"context_line":"        }"},{"line_number":101,"context_line":""},{"line_number":102,"context_line":"    def read_shell_example_script(self):"},{"line_number":103,"context_line":"        return open(self.SHELL_PATH + \u0027shell-example.sh\u0027)"},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"    def read_shell_example_text_file(self):"},{"line_number":106,"context_line":"        return open(self.SHELL_PATH + \u0027shell-example.txt\u0027)"}],"source_content_type":"text/x-python","patch_set":2,"id":"ba7be1f8_e4d30c5d","line":103,"in_reply_to":"ba7be1f8_c862bdd8","updated":"2015-03-02 15:22:48.000000000","message":"Done","commit_id":"b5ce045d17e402667bcd6c914b35f62238b788b6"},{"author":{"_account_id":7710,"name":"Sergey Reshetnyak","email":"sreshetniak@mirantis.com","username":"sreshetniak"},"change_message_id":"01de8f24f92506a4cfbe08f15c12224439fe90d0","unresolved":false,"context_lines":[{"line_number":100,"context_line":"        }"},{"line_number":101,"context_line":""},{"line_number":102,"context_line":"    def read_shell_example_script(self):"},{"line_number":103,"context_line":"        return open(self.SHELL_PATH + \u0027shell-example.sh\u0027)"},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"    def read_shell_example_text_file(self):"},{"line_number":106,"context_line":"        return open(self.SHELL_PATH + \u0027shell-example.txt\u0027)"}],"source_content_type":"text/x-python","patch_set":2,"id":"ba7be1f8_c862bdd8","line":103,"in_reply_to":"ba7be1f8_e276c005","updated":"2015-03-01 07:18:37.000000000","message":"+1","commit_id":"b5ce045d17e402667bcd6c914b35f62238b788b6"},{"author":{"_account_id":12707,"name":"Jacob Bin Wang","email":"wangbin@awcloud.com","username":"Jacob"},"change_message_id":"99ac2aa65098c7d0f5c2ed87dd59c179b34ca4a3","unresolved":false,"context_lines":[{"line_number":103,"context_line":"        return open(self.SHELL_PATH + \u0027shell-example.sh\u0027)"},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"    def read_shell_example_text_file(self):"},{"line_number":106,"context_line":"        return open(self.SHELL_PATH + \u0027shell-example.txt\u0027)"},{"line_number":107,"context_line":""},{"line_number":108,"context_line":"    def shell_example_configs(self):"},{"line_number":109,"context_line":"        return {"}],"source_content_type":"text/x-python","patch_set":2,"id":"ba7be1f8_0272f410","line":106,"updated":"2015-03-01 02:20:17.000000000","message":"ditto","commit_id":"b5ce045d17e402667bcd6c914b35f62238b788b6"},{"author":{"_account_id":7710,"name":"Sergey Reshetnyak","email":"sreshetniak@mirantis.com","username":"sreshetniak"},"change_message_id":"01de8f24f92506a4cfbe08f15c12224439fe90d0","unresolved":false,"context_lines":[{"line_number":103,"context_line":"        return open(self.SHELL_PATH + \u0027shell-example.sh\u0027)"},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"    def read_shell_example_text_file(self):"},{"line_number":106,"context_line":"        return open(self.SHELL_PATH + \u0027shell-example.txt\u0027)"},{"line_number":107,"context_line":""},{"line_number":108,"context_line":"    def shell_example_configs(self):"},{"line_number":109,"context_line":"        return {"}],"source_content_type":"text/x-python","patch_set":2,"id":"ba7be1f8_e86141d4","line":106,"in_reply_to":"ba7be1f8_0272f410","updated":"2015-03-01 07:18:37.000000000","message":"+1","commit_id":"b5ce045d17e402667bcd6c914b35f62238b788b6"},{"author":{"_account_id":13953,"name":"Elise Gafford","email":"egafford@redhat.com","username":"egafford"},"change_message_id":"bf8e3a07faa4a46b71f17c920bd3e645a454beb1","unresolved":false,"context_lines":[{"line_number":103,"context_line":"        return open(self.SHELL_PATH + \u0027shell-example.sh\u0027)"},{"line_number":104,"context_line":""},{"line_number":105,"context_line":"    def read_shell_example_text_file(self):"},{"line_number":106,"context_line":"        return open(self.SHELL_PATH + \u0027shell-example.txt\u0027)"},{"line_number":107,"context_line":""},{"line_number":108,"context_line":"    def shell_example_configs(self):"},{"line_number":109,"context_line":"        return {"}],"source_content_type":"text/x-python","patch_set":2,"id":"ba7be1f8_04cbc0a5","line":106,"in_reply_to":"ba7be1f8_e86141d4","updated":"2015-03-02 15:22:48.000000000","message":"Done","commit_id":"b5ce045d17e402667bcd6c914b35f62238b788b6"}],"sahara/tests/unit/service/edp/workflow_creator/test_create_workflow.py":[{"author":{"_account_id":8091,"name":"Trevor McKay","email":"tmckay@redhat.com","username":"tmckay"},"change_message_id":"cfa25876a05884bdfdc289280556c8c0c681bac2","unresolved":false,"context_lines":[{"line_number":236,"context_line":""},{"line_number":237,"context_line":"        res \u003d shell_workflow.get_built_workflow_xml()"},{"line_number":238,"context_line":"        shell_action \u003d \"\"\""},{"line_number":239,"context_line":"    \u003cshell xmlns\u003d\"uri:oozie:shell-action:0.1\"\u003e"},{"line_number":240,"context_line":"      \u003cjob-tracker\u003e${jobTracker}\u003c/job-tracker\u003e"},{"line_number":241,"context_line":"      \u003cname-node\u003e${nameNode}\u003c/name-node\u003e"},{"line_number":242,"context_line":"      \u003cprepare\u003e"}],"source_content_type":"text/x-python","patch_set":1,"id":"ba7be1f8_7f3e052b","line":239,"updated":"2015-02-27 17:29:31.000000000","message":"I know there is a legacy of this type of string compare all throughout the unit\ntests for xml, but if we\u0027re adding new tests we should consider a different model:\n\nhttps://review.openstack.org/#/c/141418/\n\nI fully intend to replace more of these as we go","commit_id":"b6b0b8e2d10841148753e6c6657af0b396608d33"},{"author":{"_account_id":13953,"name":"Elise Gafford","email":"egafford@redhat.com","username":"egafford"},"change_message_id":"c4e2dd56e8b18a15954f5e40cb36ec6bd8c8e2fd","unresolved":false,"context_lines":[{"line_number":236,"context_line":""},{"line_number":237,"context_line":"        res \u003d shell_workflow.get_built_workflow_xml()"},{"line_number":238,"context_line":"        shell_action \u003d \"\"\""},{"line_number":239,"context_line":"    \u003cshell xmlns\u003d\"uri:oozie:shell-action:0.1\"\u003e"},{"line_number":240,"context_line":"      \u003cjob-tracker\u003e${jobTracker}\u003c/job-tracker\u003e"},{"line_number":241,"context_line":"      \u003cname-node\u003e${nameNode}\u003c/name-node\u003e"},{"line_number":242,"context_line":"      \u003cprepare\u003e"}],"source_content_type":"text/x-python","patch_set":1,"id":"ba7be1f8_9661dbd5","line":239,"in_reply_to":"ba7be1f8_40bb1d72","updated":"2015-02-27 20:19:53.000000000","message":"What you\u0027re saying is a totally reasonable stance, and one that I\u0027ve often heard. I primarily learned on a team that took the opposite stance, which I\u0027ll summarize as:\n\n1) Unit tests should be used to educate developers on how code should function, and should be written as legibly as possible to that end.\n\n2) Unit tests should be written explicitly, literally, even \"dumbly\" so that code logic problems aren\u0027t duplicated in test logic.\n\n3) \"Breaking\" a unit test shouldn\u0027t be a big deal; they should live and breathe with the codebase, and changed whenever needed and appropriate (whenever output changes.)\n\nBoth stances have a lot to recommend them, and satisfy different goals (\"I want to have as little maintenance burden as possible on extant tests against future changes\" vs. \"I want to verify that my code is always doing precisely and literally what I think it is doing, and don\u0027t mind maintaining that contract\".)\n\nI don\u0027t actually have much of a horse in this race, and can see both arguments as wholly legitimate; still, I think the argument for this type of testing is real. If the community likes minidom better, I\u0027ve no problem with that at all.","commit_id":"b6b0b8e2d10841148753e6c6657af0b396608d33"},{"author":{"_account_id":13953,"name":"Elise Gafford","email":"egafford@redhat.com","username":"egafford"},"change_message_id":"ef10df1a667c7178e6d627a492a8163cbfb6fa34","unresolved":false,"context_lines":[{"line_number":236,"context_line":""},{"line_number":237,"context_line":"        res \u003d shell_workflow.get_built_workflow_xml()"},{"line_number":238,"context_line":"        shell_action \u003d \"\"\""},{"line_number":239,"context_line":"    \u003cshell xmlns\u003d\"uri:oozie:shell-action:0.1\"\u003e"},{"line_number":240,"context_line":"      \u003cjob-tracker\u003e${jobTracker}\u003c/job-tracker\u003e"},{"line_number":241,"context_line":"      \u003cname-node\u003e${nameNode}\u003c/name-node\u003e"},{"line_number":242,"context_line":"      \u003cprepare\u003e"}],"source_content_type":"text/x-python","patch_set":1,"id":"ba7be1f8_aab83d2b","line":239,"in_reply_to":"ba7be1f8_7f3e052b","updated":"2015-02-27 17:41:36.000000000","message":"Hm; I actually don\u0027t know that I agree on that. If we change the contents of an XML payload, in this case, I think it\u0027s very important that we know about it, even if it\u0027s a trivial difference. This might be seen as brittleness, but because of the local nature of the test, I don\u0027t see it as a problem (and think of it more as sensitivity to potential error.)\n\nJust as importantly, the literal also helps folks who don\u0027t know Oozie payloads well understand what\u0027s happening much more directly than a minidom assert sequence.","commit_id":"b6b0b8e2d10841148753e6c6657af0b396608d33"},{"author":{"_account_id":8091,"name":"Trevor McKay","email":"tmckay@redhat.com","username":"tmckay"},"change_message_id":"5f31b73721eff89e4c83f1111fbcf48d2f68c6ab","unresolved":false,"context_lines":[{"line_number":236,"context_line":""},{"line_number":237,"context_line":"        res \u003d shell_workflow.get_built_workflow_xml()"},{"line_number":238,"context_line":"        shell_action \u003d \"\"\""},{"line_number":239,"context_line":"    \u003cshell xmlns\u003d\"uri:oozie:shell-action:0.1\"\u003e"},{"line_number":240,"context_line":"      \u003cjob-tracker\u003e${jobTracker}\u003c/job-tracker\u003e"},{"line_number":241,"context_line":"      \u003cname-node\u003e${nameNode}\u003c/name-node\u003e"},{"line_number":242,"context_line":"      \u003cprepare\u003e"}],"source_content_type":"text/x-python","patch_set":1,"id":"ba7be1f8_40bb1d72","line":239,"in_reply_to":"ba7be1f8_aab83d2b","updated":"2015-02-27 19:50:35.000000000","message":"This is probably a topic for a spec. The CR can stand as it is. \n\nTesting spacing and order is out of bounds, imho.  Way too fragile.  Additionally, there is a lot that can be pulled out and tested against the base class and doesn\u0027t need to be in every job type (job-tracker elements, name-node-elements, prepare elements, etc).\n\nAnd common routines can be added to check things like \"does my resulting doc have all of the configuration elements in \u003cproperty\u003e tags that I specified?\"\n\nUnit tests shouldn\u0027t be concerned with educating readers.  There are lots  of examples out there of Oozie workflows to look at.  What matters here are\n\n1) are the right elements in the right place with the right values? (easily verifiable with the minidom)\n\n2) is the test robust\n\nI think this type of testing persisted because it was quick and easy, but I think using the dom is better. The xml stuff needs an overhaul, imho.","commit_id":"b6b0b8e2d10841148753e6c6657af0b396608d33"}]}