)]}'
{"specs/mitaka/approved/scoring-module.rst":[{"author":{"_account_id":7664,"name":"Joe Cropper","email":"jwcroppe@us.ibm.com","username":"jwcroppe"},"change_message_id":"bbf216439b2fbd67ea3c0b550a83de677293f1e0","unresolved":false,"context_lines":[{"line_number":91,"context_line":"which should be then extended by all Scoring Engine implementations. The"},{"line_number":92,"context_line":"abstract class will include the following abstract methods:"},{"line_number":93,"context_line":""},{"line_number":94,"context_line":"* getModelMetadata"},{"line_number":95,"context_line":"  Method will return a map with metadata information about the data model."},{"line_number":96,"context_line":"  This might include informations about used algorithm, labels for data"},{"line_number":97,"context_line":"  returned by the score method (useful for interpreting the results), etc."}],"source_content_type":"text/x-rst","patch_set":2,"id":"1af94dfe_b495a8e5","line":94,"range":{"start_line":94,"start_character":2,"end_line":94,"end_character":18},"updated":"2016-03-16 05:36:21.000000000","message":"Python doesn\u0027t really use camelCaseNaming for methods... rather use get_model_metadata","commit_id":"3904e48ad7f5b9c26065f5e2999eb7b177adb8e9"},{"author":{"_account_id":7664,"name":"Joe Cropper","email":"jwcroppe@us.ibm.com","username":"jwcroppe"},"change_message_id":"bbf216439b2fbd67ea3c0b550a83de677293f1e0","unresolved":false,"context_lines":[{"line_number":101,"context_line":"* score"},{"line_number":102,"context_line":"  Method responsible for performing the actual scoring, such as classifying"},{"line_number":103,"context_line":"  or predicting data."},{"line_number":104,"context_line":"  * Input: array of doubles (e.g. feature values)"},{"line_number":105,"context_line":"  * Result: array of doubles (e.g. classified values, predicted results)"},{"line_number":106,"context_line":""},{"line_number":107,"context_line":"The Watcher Scoring Module will also provide a helper methods for working with"}],"source_content_type":"text/x-rst","patch_set":2,"id":"1af94dfe_f49f3001","line":104,"range":{"start_line":104,"start_character":11,"end_line":104,"end_character":16},"updated":"2016-03-16 05:36:21.000000000","message":"Lists are more Pythonic.","commit_id":"3904e48ad7f5b9c26065f5e2999eb7b177adb8e9"},{"author":{"_account_id":16495,"name":"Jean-Emile DARTOIS","email":"jean-emile.dartois@b-com.com","username":"JeD"},"change_message_id":"348cbb706fe90e0ec3a3f9687db0d987d2247c3a","unresolved":false,"context_lines":[{"line_number":19,"context_line":""},{"line_number":20,"context_line":"This blueprint aims at providing a generic scoring engine module, which will"},{"line_number":21,"context_line":"standarize interactions with scoring engines through the common API. Also,"},{"line_number":22,"context_line":"it will be possible to use the scoring engine by different `Strategies`_, which"},{"line_number":23,"context_line":"will improve the code and data model re-use."},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"The scoring module will be independent and optional. There is no need for any"}],"source_content_type":"text/x-rst","patch_set":4,"id":"1af94dfe_c8dcd0e6","line":22,"updated":"2016-03-17 16:16:53.000000000","message":"What is the granularity of abstraction that we want to expose to the strategy developer through the scoring engine ? Do we want to expose at the framework level (spark, TAP, ..) or at the algorithms level (kmeans, ...) ?","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"83ff765da294006d96bb823d860b710be86abba3","unresolved":false,"context_lines":[{"line_number":19,"context_line":""},{"line_number":20,"context_line":"This blueprint aims at providing a generic scoring engine module, which will"},{"line_number":21,"context_line":"standarize interactions with scoring engines through the common API. Also,"},{"line_number":22,"context_line":"it will be possible to use the scoring engine by different `Strategies`_, which"},{"line_number":23,"context_line":"will improve the code and data model re-use."},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"The scoring module will be independent and optional. There is no need for any"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa0719c6_cbd70a22","line":22,"in_reply_to":"1af94dfe_c8dcd0e6","updated":"2016-03-21 09:00:03.000000000","message":"From my experience with scoring engines we should treat them as black boxes. They all might be very different in terms of algorithm or framework used, but the plan is to use a single abstraction layer to work with all of them in a similar way. But \"one-click replacement\" in all cases might be hard to achieve due to differences in algorithms used (different inputs/outputs).","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"0fbd6f10ff87ef552a84e1fc523a368c75aa4b0c","unresolved":false,"context_lines":[{"line_number":19,"context_line":""},{"line_number":20,"context_line":"This blueprint aims at providing a generic scoring engine module, which will"},{"line_number":21,"context_line":"standarize interactions with scoring engines through the common API. Also,"},{"line_number":22,"context_line":"it will be possible to use the scoring engine by different `Strategies`_, which"},{"line_number":23,"context_line":"will improve the code and data model re-use."},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"The scoring module will be independent and optional. There is no need for any"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa0719c6_728ef52e","line":22,"in_reply_to":"fa0719c6_c49de65a","updated":"2016-03-24 09:27:37.000000000","message":"Not really. It\u0027s not about algorithms, but scoring engines. \n\nLet me give an example: there is a ML algorithm called Gradient Boosted Machines (https://en.wikipedia.org/wiki/Gradient_boosting) which is widely used by data scientists for creating data models. Using this algorithm AND some learning data, you can get a decision tree which might be used as a classifier to solve some problems (e.g. Watcher optimization).\n\nWrapping this data model into some API or web service gives you a scoring engine, which can be then used by some Watcher Strategy.\n\nAs you see, it doesn\u0027t make sense to expose the GBM algorithm itself, but only it\u0027s outcome after the learning data was applied.\n\nI plan to provide one sample Scoring Engine for each scenario I\u0027m proposing (inside Decision Engine, inside Scoring Module, external), but that\u0027s only how far I can go. From this point, Watcher will need additional work to create some specific scoring engines, which might be then used in some strategies.","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":16495,"name":"Jean-Emile DARTOIS","email":"jean-emile.dartois@b-com.com","username":"JeD"},"change_message_id":"339ced98593d54081db08280f1150ed8cfd81c5e","unresolved":false,"context_lines":[{"line_number":19,"context_line":""},{"line_number":20,"context_line":"This blueprint aims at providing a generic scoring engine module, which will"},{"line_number":21,"context_line":"standarize interactions with scoring engines through the common API. Also,"},{"line_number":22,"context_line":"it will be possible to use the scoring engine by different `Strategies`_, which"},{"line_number":23,"context_line":"will improve the code and data model re-use."},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"The scoring module will be independent and optional. There is no need for any"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa0719c6_c49de65a","line":22,"in_reply_to":"fa0719c6_cbd70a22","updated":"2016-03-23 13:14:35.000000000","message":"I fully agree that we should treat scoring engines as black boxes.\nI\u0027m also agree that the input parameters and the results are very depending of the algorithms.\n\nif I understand well your answer, you will share one by one the algorithms, is it right ?","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":16495,"name":"Jean-Emile DARTOIS","email":"jean-emile.dartois@b-com.com","username":"JeD"},"change_message_id":"348cbb706fe90e0ec3a3f9687db0d987d2247c3a","unresolved":false,"context_lines":[{"line_number":23,"context_line":"will improve the code and data model re-use."},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"The scoring module will be independent and optional. There is no need for any"},{"line_number":26,"context_line":"`Strategy`_ to use it."},{"line_number":27,"context_line":""},{"line_number":28,"context_line":"Problem description"},{"line_number":29,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":4,"id":"1af94dfe_a88504ae","line":26,"updated":"2016-03-17 16:16:53.000000000","message":"I would like to have a schema which show the concerns of the scoring module compared to a typical machine learning process in order to make clear the scope of the scoring module.\n\nIn my opinion a typical flow look like that:\nData Source(reads and selects data) \u003d\u003e \nData Processing(cleaner,checker,..) \u003d\u003e \nAlgorithms(s) \u003d\u003e \nServing(expose the predicted or classification results)\n\u003c\u003d Watcher strategies (used the results to optimize the infrastructure)\n\nIMHO, in case of watcher a common scenario, the scoring module continuously collect (online) or batch data from ceilometer, apache kafka, rabbitmq, ect\n\nA Scoring engine then builds predictive model(s) with one ore more algorithms using the data. After it the scoring module is exposing the results through his REST API. It listens to queries from the strategies and respond with predicted or classification results in real-time.\n\nDo you think that the scoring module is in charge of collecting the data ?","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"83ff765da294006d96bb823d860b710be86abba3","unresolved":false,"context_lines":[{"line_number":23,"context_line":"will improve the code and data model re-use."},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"The scoring module will be independent and optional. There is no need for any"},{"line_number":26,"context_line":"`Strategy`_ to use it."},{"line_number":27,"context_line":""},{"line_number":28,"context_line":"Problem description"},{"line_number":29,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa0719c6_8bd8721f","line":26,"in_reply_to":"1af94dfe_a88504ae","updated":"2016-03-21 09:00:03.000000000","message":"I think it makes a sense to prepare a diagram showing how all the components work together. I will prepare it and include with the new spec iteration.\n\nI think you described well what will be happening.\n\nFor the data collection: the scoring module abstraction layer or the scoring framework is not going to collect or store any data. But at the end of the day, it\u0027s up to the scoring plugin developer. The data will be flowing to the scoring engine and if for whatever reason the particular scoring engine wants to store it (for further analysis or to use as a reinforcement loop for some algorithms), it\u0027s completely up to him.","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":16495,"name":"Jean-Emile DARTOIS","email":"jean-emile.dartois@b-com.com","username":"JeD"},"change_message_id":"339ced98593d54081db08280f1150ed8cfd81c5e","unresolved":false,"context_lines":[{"line_number":23,"context_line":"will improve the code and data model re-use."},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"The scoring module will be independent and optional. There is no need for any"},{"line_number":26,"context_line":"`Strategy`_ to use it."},{"line_number":27,"context_line":""},{"line_number":28,"context_line":"Problem description"},{"line_number":29,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa0719c6_e5fb9dbf","line":26,"in_reply_to":"fa0719c6_8bd8721f","updated":"2016-03-23 13:14:35.000000000","message":"IMHO, we should create a data source component in charge of collecting asynchronously the measurements from various data sources and provide them directly to the scoring engines.\n\nThe component will avoid that the strategies have to querying the data but also avoid to querying the same data from multiple strategies (e.g: case of the the need of the same prediction for two strategies).\n\nThe strategies will not have to wait to have the predicted results  they are pre-calculated.\n\nThis component can also be in charge of adapting the format of the measurements according to scoring engines.\n\nIMHO, we should do that in an another blueprint. Moreover, I think that this blueprint will be link with https://blueprints.launchpad.net/watcher/+spec/metrics-storage-cache","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"0fbd6f10ff87ef552a84e1fc523a368c75aa4b0c","unresolved":false,"context_lines":[{"line_number":23,"context_line":"will improve the code and data model re-use."},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"The scoring module will be independent and optional. There is no need for any"},{"line_number":26,"context_line":"`Strategy`_ to use it."},{"line_number":27,"context_line":""},{"line_number":28,"context_line":"Problem description"},{"line_number":29,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa0719c6_d259c18f","line":26,"in_reply_to":"fa0719c6_e5fb9dbf","updated":"2016-03-24 09:27:37.000000000","message":"I\u0027m not questioning this idea, but it\u0027s definitely out of scope of this BP. This BP is about introducing Scoring Engines to the Watcher system. It makes a lot of sense to use it for future predictions as you are suggesting, but the scoring engines themselves don\u0027t even need to know how they are used. The current design should well support the scenario you are describing.","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":16495,"name":"Jean-Emile DARTOIS","email":"jean-emile.dartois@b-com.com","username":"JeD"},"change_message_id":"348cbb706fe90e0ec3a3f9687db0d987d2247c3a","unresolved":false,"context_lines":[{"line_number":34,"context_line":"between different `Strategies`_."},{"line_number":35,"context_line":""},{"line_number":36,"context_line":"It should be possible to implement scoring engines independently from"},{"line_number":37,"context_line":"`Strategies`_. Once implemented prediction or classifier could be shared and"},{"line_number":38,"context_line":"re-used allowing to implement more advanced `Strategies`_, which could even use"},{"line_number":39,"context_line":"multiple scoring engines."},{"line_number":40,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"1af94dfe_2678058f","line":37,"updated":"2016-03-17 16:16:53.000000000","message":"Do we want to add the possibility to manage the \"versioning\" of the algorithms (e.g:prediction or classifier) shared by the strategies ?\n\ne.g: we are sharing the prediction \u0027A\u0027 with two strategies (\u0027S1\u0027,\u0027S2\u0027) and the strategy \u0027S1\u0027 needs to tune some parameters for his own needs but this modification will change the result need by \u0027S2\u0027. \nFor simplicity, we could say that \u0027S2\u0027 need to create a new prediction ? What do you think ?","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"83ff765da294006d96bb823d860b710be86abba3","unresolved":false,"context_lines":[{"line_number":34,"context_line":"between different `Strategies`_."},{"line_number":35,"context_line":""},{"line_number":36,"context_line":"It should be possible to implement scoring engines independently from"},{"line_number":37,"context_line":"`Strategies`_. Once implemented prediction or classifier could be shared and"},{"line_number":38,"context_line":"re-used allowing to implement more advanced `Strategies`_, which could even use"},{"line_number":39,"context_line":"multiple scoring engines."},{"line_number":40,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa0719c6_2b6de604","line":37,"in_reply_to":"1af94dfe_2678058f","updated":"2016-03-21 09:00:03.000000000","message":"IMO a scoring engine is a data model instance created with particular data used for training. Taking the same data model, but applying different training data creates a different scoring engine with (possibly) different results. \n\nI think that the rules similar to API versioning should apply here - a new version of the scoring engine should be created (with different key/name), so it won\u0027t affect the current use cases. Of course I can imagine that somebody will want to update the existing instance, but then should take the full responsibility for that and understand the fact, that it might change the other use cases.\n\nOn the other hand one might argue that for example improving a classifier should improve all the use cases anyway...","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"0fbd6f10ff87ef552a84e1fc523a368c75aa4b0c","unresolved":false,"context_lines":[{"line_number":34,"context_line":"between different `Strategies`_."},{"line_number":35,"context_line":""},{"line_number":36,"context_line":"It should be possible to implement scoring engines independently from"},{"line_number":37,"context_line":"`Strategies`_. Once implemented prediction or classifier could be shared and"},{"line_number":38,"context_line":"re-used allowing to implement more advanced `Strategies`_, which could even use"},{"line_number":39,"context_line":"multiple scoring engines."},{"line_number":40,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa0719c6_ad92ee49","line":37,"in_reply_to":"fa0719c6_04ffde34","updated":"2016-03-24 09:27:37.000000000","message":"Added.","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":16495,"name":"Jean-Emile DARTOIS","email":"jean-emile.dartois@b-com.com","username":"JeD"},"change_message_id":"339ced98593d54081db08280f1150ed8cfd81c5e","unresolved":false,"context_lines":[{"line_number":34,"context_line":"between different `Strategies`_."},{"line_number":35,"context_line":""},{"line_number":36,"context_line":"It should be possible to implement scoring engines independently from"},{"line_number":37,"context_line":"`Strategies`_. Once implemented prediction or classifier could be shared and"},{"line_number":38,"context_line":"re-used allowing to implement more advanced `Strategies`_, which could even use"},{"line_number":39,"context_line":"multiple scoring engines."},{"line_number":40,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa0719c6_04ffde34","line":37,"in_reply_to":"fa0719c6_2b6de604","updated":"2016-03-23 13:14:35.000000000","message":"I agree, we should write that in the documentation.","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":16495,"name":"Jean-Emile DARTOIS","email":"jean-emile.dartois@b-com.com","username":"JeD"},"change_message_id":"348cbb706fe90e0ec3a3f9687db0d987d2247c3a","unresolved":false,"context_lines":[{"line_number":61,"context_line":"metadata might contain an important information how to work with a specific"},{"line_number":62,"context_line":"Scoring Engine, for example: number of input parameters (e.g. number of"},{"line_number":63,"context_line":"features), the response format (e.g. label order used by classifier)."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"Project Priority"},{"line_number":66,"context_line":"----------------"},{"line_number":67,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"1af94dfe_74b3c9e9","line":64,"updated":"2016-03-17 16:16:53.000000000","message":"As a strategy developer.\nI want to be able to list the available predictive model(s).\nSo that I can quickly identify them and reuse the available predicted result in my strategy.\n\ne.g prediction energy consumption of the VMs, predicted CPU of the VMs,ect","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"83ff765da294006d96bb823d860b710be86abba3","unresolved":false,"context_lines":[{"line_number":61,"context_line":"metadata might contain an important information how to work with a specific"},{"line_number":62,"context_line":"Scoring Engine, for example: number of input parameters (e.g. number of"},{"line_number":63,"context_line":"features), the response format (e.g. label order used by classifier)."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"Project Priority"},{"line_number":66,"context_line":"----------------"},{"line_number":67,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa0719c6_a61dab68","line":64,"in_reply_to":"1af94dfe_74b3c9e9","updated":"2016-03-21 09:00:03.000000000","message":"Interesting use case and makes a lot of sense! I guess that requires to expose that through the watcher CLI? I will add that use case, but need to have a look how to actually implement it.","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":16495,"name":"Jean-Emile DARTOIS","email":"jean-emile.dartois@b-com.com","username":"JeD"},"change_message_id":"339ced98593d54081db08280f1150ed8cfd81c5e","unresolved":false,"context_lines":[{"line_number":61,"context_line":"metadata might contain an important information how to work with a specific"},{"line_number":62,"context_line":"Scoring Engine, for example: number of input parameters (e.g. number of"},{"line_number":63,"context_line":"features), the response format (e.g. label order used by classifier)."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"Project Priority"},{"line_number":66,"context_line":"----------------"},{"line_number":67,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa0719c6_8762d0ab","line":64,"in_reply_to":"fa0719c6_a61dab68","updated":"2016-03-23 13:14:35.000000000","message":"IMHO, it requires that we expose this information through the watcher API.\n\nFYI, we are working on a generic way to return the list of available goals/strategies/strategies_parameters which have been installed on an instance of watcher. \n\nI think you should reuse the same mechanism for this use case. \nYou can take a look to the specification: https://review.openstack.org/#/c/293862/","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":16495,"name":"Jean-Emile DARTOIS","email":"jean-emile.dartois@b-com.com","username":"JeD"},"change_message_id":"348cbb706fe90e0ec3a3f9687db0d987d2247c3a","unresolved":false,"context_lines":[{"line_number":62,"context_line":"Scoring Engine, for example: number of input parameters (e.g. number of"},{"line_number":63,"context_line":"features), the response format (e.g. label order used by classifier)."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"Project Priority"},{"line_number":66,"context_line":"----------------"},{"line_number":67,"context_line":""},{"line_number":68,"context_line":"Not relevant because Watcher is not in the big tent so far."}],"source_content_type":"text/x-rst","patch_set":4,"id":"1af94dfe_d760df03","line":65,"updated":"2016-03-17 16:16:53.000000000","message":"As a strategy developer.\nI want te be able to subscribe to \"online\" results.\nSo that I can have the most updated data as soon as possible.","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"83ff765da294006d96bb823d860b710be86abba3","unresolved":false,"context_lines":[{"line_number":62,"context_line":"Scoring Engine, for example: number of input parameters (e.g. number of"},{"line_number":63,"context_line":"features), the response format (e.g. label order used by classifier)."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"Project Priority"},{"line_number":66,"context_line":"----------------"},{"line_number":67,"context_line":""},{"line_number":68,"context_line":"Not relevant because Watcher is not in the big tent so far."}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa0719c6_46c12f80","line":65,"in_reply_to":"1af94dfe_d760df03","updated":"2016-03-21 09:00:03.000000000","message":"I don\u0027t fully understand this use case. From scoring engine perspective, there is no such thing as \"online\" or \"offline\" results. The strategy will basically call the score method and that\u0027s it - the result should be returned ASAP in all cases. The idea is that the scoring engine should fast and respond in real-time. If for whatever reason it\u0027s not the case (e.g. the calculations take forever), the strategy developer should take this into account and design a system, which will support that (e.g. use score method to \"request\" data and have some additional queue for retrieving the results).","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":16495,"name":"Jean-Emile DARTOIS","email":"jean-emile.dartois@b-com.com","username":"JeD"},"change_message_id":"339ced98593d54081db08280f1150ed8cfd81c5e","unresolved":false,"context_lines":[{"line_number":62,"context_line":"Scoring Engine, for example: number of input parameters (e.g. number of"},{"line_number":63,"context_line":"features), the response format (e.g. label order used by classifier)."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"Project Priority"},{"line_number":66,"context_line":"----------------"},{"line_number":67,"context_line":""},{"line_number":68,"context_line":"Not relevant because Watcher is not in the big tent so far."}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa0719c6_e783cc02","line":65,"in_reply_to":"fa0719c6_46c12f80","updated":"2016-03-23 13:14:35.000000000","message":"I would like something close to the pattern observer that push the information to the observers. In other word, I don\u0027t to want to query to scoring engines to have the information.\ne.g: So that I can use the scoring module to trigger an audit (predicted failure).","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"0fbd6f10ff87ef552a84e1fc523a368c75aa4b0c","unresolved":false,"context_lines":[{"line_number":62,"context_line":"Scoring Engine, for example: number of input parameters (e.g. number of"},{"line_number":63,"context_line":"features), the response format (e.g. label order used by classifier)."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"Project Priority"},{"line_number":66,"context_line":"----------------"},{"line_number":67,"context_line":""},{"line_number":68,"context_line":"Not relevant because Watcher is not in the big tent so far."}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa0719c6_0d6cba33","line":65,"in_reply_to":"fa0719c6_e783cc02","updated":"2016-03-24 09:27:37.000000000","message":"Fair enough, but this happens outside of the scoring engine itself, likely in strategy. It\u0027s more related to the continuous audits I think.","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":16495,"name":"Jean-Emile DARTOIS","email":"jean-emile.dartois@b-com.com","username":"JeD"},"change_message_id":"348cbb706fe90e0ec3a3f9687db0d987d2247c3a","unresolved":false,"context_lines":[{"line_number":63,"context_line":"features), the response format (e.g. label order used by classifier)."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"Project Priority"},{"line_number":66,"context_line":"----------------"},{"line_number":67,"context_line":""},{"line_number":68,"context_line":"Not relevant because Watcher is not in the big tent so far."},{"line_number":69,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"1af94dfe_4a28a686","line":66,"updated":"2016-03-17 16:16:53.000000000","message":"As a strategy developer.\nI want te be able to learn on the entire data set at once.\nSo that i can generate the best prediction.","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"83ff765da294006d96bb823d860b710be86abba3","unresolved":false,"context_lines":[{"line_number":63,"context_line":"features), the response format (e.g. label order used by classifier)."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"Project Priority"},{"line_number":66,"context_line":"----------------"},{"line_number":67,"context_line":""},{"line_number":68,"context_line":"Not relevant because Watcher is not in the big tent so far."},{"line_number":69,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa0719c6_e6b963ee","line":66,"in_reply_to":"1af94dfe_4a28a686","updated":"2016-03-21 09:00:03.000000000","message":"This is up to the scoring plugin developer. If there is a case where a strategy wants to collect some data for some time and then use the whole batch for prediction - that\u0027s fine. But then the interface to the scoring module is the same: the score method with some inputs and outputs. Do you agree?","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":16495,"name":"Jean-Emile DARTOIS","email":"jean-emile.dartois@b-com.com","username":"JeD"},"change_message_id":"339ced98593d54081db08280f1150ed8cfd81c5e","unresolved":false,"context_lines":[{"line_number":63,"context_line":"features), the response format (e.g. label order used by classifier)."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"Project Priority"},{"line_number":66,"context_line":"----------------"},{"line_number":67,"context_line":""},{"line_number":68,"context_line":"Not relevant because Watcher is not in the big tent so far."},{"line_number":69,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa0719c6_8d12bf07","line":66,"in_reply_to":"fa0719c6_e6b963ee","updated":"2016-03-23 13:14:35.000000000","message":"I think that this use case is link with the data source collector.","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":16495,"name":"Jean-Emile DARTOIS","email":"jean-emile.dartois@b-com.com","username":"JeD"},"change_message_id":"348cbb706fe90e0ec3a3f9687db0d987d2247c3a","unresolved":false,"context_lines":[{"line_number":64,"context_line":""},{"line_number":65,"context_line":"Project Priority"},{"line_number":66,"context_line":"----------------"},{"line_number":67,"context_line":""},{"line_number":68,"context_line":"Not relevant because Watcher is not in the big tent so far."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"Proposed change"}],"source_content_type":"text/x-rst","patch_set":4,"id":"1af94dfe_834c549f","line":67,"updated":"2016-03-17 16:16:53.000000000","message":"As a developer.\nI want te able to subscribe to events (predicted)from the scoring module.\nSo that I can trigger a new audit in order for example to react to a predicted critical situation.","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"83ff765da294006d96bb823d860b710be86abba3","unresolved":false,"context_lines":[{"line_number":64,"context_line":""},{"line_number":65,"context_line":"Project Priority"},{"line_number":66,"context_line":"----------------"},{"line_number":67,"context_line":""},{"line_number":68,"context_line":"Not relevant because Watcher is not in the big tent so far."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"Proposed change"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa0719c6_a6776b01","line":67,"in_reply_to":"1af94dfe_834c549f","updated":"2016-03-21 09:00:03.000000000","message":"Interesting use case, I can see that somebody wants to predict in advance some problems. But isn\u0027t it the responsibility of the strategy? If a strategy wants to send some \"theoretical\" data and check \"what would happen if\", that\u0027s completely fine! But scoring engine is a quite dumb service: it\u0027s basically a mathematical function - it\u0027s just calculates the results, but it\u0027s up to the user (strategy in this case) how to use the results, what input data use etc. I would keep the scoring module as simple as possible.","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":16495,"name":"Jean-Emile DARTOIS","email":"jean-emile.dartois@b-com.com","username":"JeD"},"change_message_id":"348cbb706fe90e0ec3a3f9687db0d987d2247c3a","unresolved":false,"context_lines":[{"line_number":120,"context_line":"      :Result:"},{"line_number":121,"context_line":"        unique string ID (must be unique across all Scoring Engines)"},{"line_number":122,"context_line":""},{"line_number":123,"context_line":"    :get_model_metadata:"},{"line_number":124,"context_line":"      Method will return a map with metadata information about the data"},{"line_number":125,"context_line":"      model. This might include informations about used algorithm, labels"},{"line_number":126,"context_line":"      for data returned by the score method (useful for interpreting the"}],"source_content_type":"text/x-rst","patch_set":4,"id":"1af94dfe_6667964d","line":123,"updated":"2016-03-17 16:16:53.000000000","message":"get_description ?","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"83ff765da294006d96bb823d860b710be86abba3","unresolved":false,"context_lines":[{"line_number":120,"context_line":"      :Result:"},{"line_number":121,"context_line":"        unique string ID (must be unique across all Scoring Engines)"},{"line_number":122,"context_line":""},{"line_number":123,"context_line":"    :get_model_metadata:"},{"line_number":124,"context_line":"      Method will return a map with metadata information about the data"},{"line_number":125,"context_line":"      model. This might include informations about used algorithm, labels"},{"line_number":126,"context_line":"      for data returned by the score method (useful for interpreting the"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa0719c6_08baa5e1","line":123,"in_reply_to":"1af94dfe_6667964d","updated":"2016-03-21 09:00:03.000000000","message":"It\u0027s not just the description. It\u0027s a map and can contain a lot of information, some of it might be even critical to use the scoring engine.\n\nTake this example (real case which I already had!):\n- scoring engine is a classifier, which is based on the learning data with these column labels (last column is the result used for learning): [MEM_USAGE, PROC_USAGE, PCI_USAGE, POWER_CONSUMPTION, CLASSIFICATION_ID]\n- during the learning process, the machine learning decides that it actually only needs these columns to calculate the expected CLASSIFICATION_ID: [MEM_USAGE, PROC_USAGE]\n- Also, because the result is a list of doubles, you need to know what it means, e.g. 0.0 \u003d\u003d CLASSIFICATION_ID_2, 1.0 \u003d\u003d CLASSIFICATION_ID_1, etc.\n- As you see there is no guarantee of the order of the columns or even the existence of them\n- You need to pass this information as metadata, so the user of the scoring engine is able to \"understand\" the results.\n- In addition, the metadata might provide some insights what is the algorithm, how many training records were used etc.\n\nAs you see, it\u0027s not the description :)","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":16495,"name":"Jean-Emile DARTOIS","email":"jean-emile.dartois@b-com.com","username":"JeD"},"change_message_id":"339ced98593d54081db08280f1150ed8cfd81c5e","unresolved":false,"context_lines":[{"line_number":120,"context_line":"      :Result:"},{"line_number":121,"context_line":"        unique string ID (must be unique across all Scoring Engines)"},{"line_number":122,"context_line":""},{"line_number":123,"context_line":"    :get_model_metadata:"},{"line_number":124,"context_line":"      Method will return a map with metadata information about the data"},{"line_number":125,"context_line":"      model. This might include informations about used algorithm, labels"},{"line_number":126,"context_line":"      for data returned by the score method (useful for interpreting the"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa0719c6_87bad0be","line":123,"in_reply_to":"fa0719c6_08baa5e1","updated":"2016-03-23 13:14:35.000000000","message":"I understand. Could you please add this example in the specification ?","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"0fbd6f10ff87ef552a84e1fc523a368c75aa4b0c","unresolved":false,"context_lines":[{"line_number":120,"context_line":"      :Result:"},{"line_number":121,"context_line":"        unique string ID (must be unique across all Scoring Engines)"},{"line_number":122,"context_line":""},{"line_number":123,"context_line":"    :get_model_metadata:"},{"line_number":124,"context_line":"      Method will return a map with metadata information about the data"},{"line_number":125,"context_line":"      model. This might include informations about used algorithm, labels"},{"line_number":126,"context_line":"      for data returned by the score method (useful for interpreting the"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa0719c6_08dca871","line":123,"in_reply_to":"fa0719c6_87bad0be","updated":"2016-03-24 09:27:37.000000000","message":"Added.","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":16495,"name":"Jean-Emile DARTOIS","email":"jean-emile.dartois@b-com.com","username":"JeD"},"change_message_id":"348cbb706fe90e0ec3a3f9687db0d987d2247c3a","unresolved":false,"context_lines":[{"line_number":122,"context_line":""},{"line_number":123,"context_line":"    :get_model_metadata:"},{"line_number":124,"context_line":"      Method will return a map with metadata information about the data"},{"line_number":125,"context_line":"      model. This might include informations about used algorithm, labels"},{"line_number":126,"context_line":"      for data returned by the score method (useful for interpreting the"},{"line_number":127,"context_line":"      results)"},{"line_number":128,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"1af94dfe_6032de60","line":125,"updated":"2016-03-17 16:16:53.000000000","message":"How do you plan to define a common namming of results ?","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"83ff765da294006d96bb823d860b710be86abba3","unresolved":false,"context_lines":[{"line_number":122,"context_line":""},{"line_number":123,"context_line":"    :get_model_metadata:"},{"line_number":124,"context_line":"      Method will return a map with metadata information about the data"},{"line_number":125,"context_line":"      model. This might include informations about used algorithm, labels"},{"line_number":126,"context_line":"      for data returned by the score method (useful for interpreting the"},{"line_number":127,"context_line":"      results)"},{"line_number":128,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa0719c6_887cf521","line":125,"in_reply_to":"1af94dfe_6032de60","updated":"2016-03-21 09:00:03.000000000","message":"I don\u0027t plan! It\u0027s up to the scoring engine developer. Strategy must be able to understand the results.","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":16495,"name":"Jean-Emile DARTOIS","email":"jean-emile.dartois@b-com.com","username":"JeD"},"change_message_id":"339ced98593d54081db08280f1150ed8cfd81c5e","unresolved":false,"context_lines":[{"line_number":122,"context_line":""},{"line_number":123,"context_line":"    :get_model_metadata:"},{"line_number":124,"context_line":"      Method will return a map with metadata information about the data"},{"line_number":125,"context_line":"      model. This might include informations about used algorithm, labels"},{"line_number":126,"context_line":"      for data returned by the score method (useful for interpreting the"},{"line_number":127,"context_line":"      results)"},{"line_number":128,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa0719c6_d2a3e84d","line":125,"in_reply_to":"fa0719c6_887cf521","updated":"2016-03-23 13:14:35.000000000","message":"Do you thin that we should provide some guideline to help formalize and promote consistency between the developers ?","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"0fbd6f10ff87ef552a84e1fc523a368c75aa4b0c","unresolved":false,"context_lines":[{"line_number":122,"context_line":""},{"line_number":123,"context_line":"    :get_model_metadata:"},{"line_number":124,"context_line":"      Method will return a map with metadata information about the data"},{"line_number":125,"context_line":"      model. This might include informations about used algorithm, labels"},{"line_number":126,"context_line":"      for data returned by the score method (useful for interpreting the"},{"line_number":127,"context_line":"      results)"},{"line_number":128,"context_line":""}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa0719c6_2876cc6f","line":125,"in_reply_to":"fa0719c6_d2a3e84d","updated":"2016-03-24 09:27:37.000000000","message":"Definitely. That\u0027s why I plan to implement one Scoring Engine for each of three scenarios. They will serve (together with the documentation) as a reference implementation or a guideline.","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":16495,"name":"Jean-Emile DARTOIS","email":"jean-emile.dartois@b-com.com","username":"JeD"},"change_message_id":"348cbb706fe90e0ec3a3f9687db0d987d2247c3a","unresolved":false,"context_lines":[{"line_number":137,"context_line":"      classifying or predicting data"},{"line_number":138,"context_line":""},{"line_number":139,"context_line":"      :Input:"},{"line_number":140,"context_line":"        list of doubles (e.g. feature values)"},{"line_number":141,"context_line":""},{"line_number":142,"context_line":"      :Result:"},{"line_number":143,"context_line":"        list of doubles (e.g. classified values, predicted results)"}],"source_content_type":"text/x-rst","patch_set":4,"id":"1af94dfe_5465ed1b","line":140,"updated":"2016-03-17 16:16:53.000000000","message":"Who is in charge of providing the input data ?\nThe querying of the input data from any sources(ceilometer, rabbitmq,ect) can take lot of time and therefore there should be a background task which is able to produce the expected feature values for a given strategy.\n\nIf I understand well, in your case, every time a strategy is executed the strategy will be in charge of collecting the data and sending it to the scoring module and the wait for the result.","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"83ff765da294006d96bb823d860b710be86abba3","unresolved":false,"context_lines":[{"line_number":137,"context_line":"      classifying or predicting data"},{"line_number":138,"context_line":""},{"line_number":139,"context_line":"      :Input:"},{"line_number":140,"context_line":"        list of doubles (e.g. feature values)"},{"line_number":141,"context_line":""},{"line_number":142,"context_line":"      :Result:"},{"line_number":143,"context_line":"        list of doubles (e.g. classified values, predicted results)"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa0719c6_48fe5d6c","line":140,"in_reply_to":"1af94dfe_5465ed1b","updated":"2016-03-21 09:00:03.000000000","message":"Correct, it\u0027s up to the strategy which is the only place where scoring engine will be used in Watcher.\n\nI fully understand the concerns about scalability, I raised them myself :) There must be an efficient way for implementing the continuous monitoring. In this case the strategy should not query for data each time, it should just process the incoming data (e.g. from some queue).","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":16495,"name":"Jean-Emile DARTOIS","email":"jean-emile.dartois@b-com.com","username":"JeD"},"change_message_id":"348cbb706fe90e0ec3a3f9687db0d987d2247c3a","unresolved":false,"context_lines":[{"line_number":141,"context_line":""},{"line_number":142,"context_line":"      :Result:"},{"line_number":143,"context_line":"        list of doubles (e.g. classified values, predicted results)"},{"line_number":144,"context_line":""},{"line_number":145,"context_line":"* In the `Watcher Decision Engine`_:"},{"line_number":146,"context_line":""},{"line_number":147,"context_line":"  * New scoring package containing:"}],"source_content_type":"text/x-rst","patch_set":4,"id":"1af94dfe_be71a878","line":144,"updated":"2016-03-17 16:16:53.000000000","message":"In order to avoid too many requests to the requests to the scoring engine  we may request several resutls at once.\n\nPredictedResult(\n  Scores: list of TupleScore\n) \n\nclass TupleScore(\n  item: String,\n  score: list of Double\n)\n\nPredictedResult\n{\"Scores\":[\n  {\"item\":vm_1,\"score\":[4.07,4.07,4.08,4,09,]},\n  {\"item\":vm_2,\"score\":[4.05]},\n  {\"item\":vm_4,\"score\":[4.04]},\n  {\"item\":vm_5,\"score\":[3.81]}\n]}","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"83ff765da294006d96bb823d860b710be86abba3","unresolved":false,"context_lines":[{"line_number":141,"context_line":""},{"line_number":142,"context_line":"      :Result:"},{"line_number":143,"context_line":"        list of doubles (e.g. classified values, predicted results)"},{"line_number":144,"context_line":""},{"line_number":145,"context_line":"* In the `Watcher Decision Engine`_:"},{"line_number":146,"context_line":""},{"line_number":147,"context_line":"  * New scoring package containing:"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa0719c6_ab69f38d","line":144,"in_reply_to":"1af94dfe_be71a878","updated":"2016-03-21 09:00:03.000000000","message":"The thing with scoring engines is that they will likely have different characteristics. They might be implemented \"internally\" in plain Python code, they might reside in the different Watcher Scoring service (please note that the current proposal allows to run scoring engines as part of the Watcher Decision Engine as well) or even in the cloud (like TAP). Optimizing it might make things even worse and extend the response times.\n\nI would not do this at this point. I would make things simple for now and observe how it behaves. We might need to iterate on this solution of course, but experience says that usually it goes in different direction than everybody is thinking and problems might be in completely different place. What is your opinion?","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":16495,"name":"Jean-Emile DARTOIS","email":"jean-emile.dartois@b-com.com","username":"JeD"},"change_message_id":"348cbb706fe90e0ec3a3f9687db0d987d2247c3a","unresolved":false,"context_lines":[{"line_number":243,"context_line":"----------"},{"line_number":244,"context_line":""},{"line_number":245,"context_line":"The list of foreseen work items:"},{"line_number":246,"context_line":""},{"line_number":247,"context_line":"* Review this BluePrint, improve it based on feedback received"},{"line_number":248,"context_line":"* Implement generic Watcher Scoring Module"},{"line_number":249,"context_line":"* Implement a sample Scoring Engine to demonstrate Scoring Module design"}],"source_content_type":"text/x-rst","patch_set":4,"id":"1af94dfe_d7b1ff48","line":246,"updated":"2016-03-17 16:16:53.000000000","message":"* Implement scoring engine loader (reuse the common loader watcher/watcher/common/loader/default.py)","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"83ff765da294006d96bb823d860b710be86abba3","unresolved":false,"context_lines":[{"line_number":243,"context_line":"----------"},{"line_number":244,"context_line":""},{"line_number":245,"context_line":"The list of foreseen work items:"},{"line_number":246,"context_line":""},{"line_number":247,"context_line":"* Review this BluePrint, improve it based on feedback received"},{"line_number":248,"context_line":"* Implement generic Watcher Scoring Module"},{"line_number":249,"context_line":"* Implement a sample Scoring Engine to demonstrate Scoring Module design"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa0719c6_6b1d1b0f","line":246,"in_reply_to":"1af94dfe_d7b1ff48","updated":"2016-03-21 09:00:03.000000000","message":"OK.","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":18971,"name":"Vincent Françoise","email":"Vincent.FRANCOISE@b-com.com","username":"v-francoise"},"change_message_id":"5e2b36b9224d2d38b2ff5e156cbdf98217d8a78b","unresolved":false,"context_lines":[{"line_number":250,"context_line":"  and provide a guidance how to use it (no external dependencies)"},{"line_number":251,"context_line":"* Implement a sample `Strategy`_ using sample Scoring Engine from previous"},{"line_number":252,"context_line":"  point"},{"line_number":253,"context_line":"* Provide documentation of the new Scoring Module"},{"line_number":254,"context_line":""},{"line_number":255,"context_line":"Dependencies"},{"line_number":256,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":4,"id":"1af94dfe_82401ca7","line":253,"range":{"start_line":253,"start_character":2,"end_line":253,"end_character":49},"updated":"2016-03-17 09:28:47.000000000","message":"IMO we can explicit this a bit:\n\n* We need to add Watcher Scoring in the glossary\n* We need to explain how to implement a scoring engine plugin","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":16495,"name":"Jean-Emile DARTOIS","email":"jean-emile.dartois@b-com.com","username":"JeD"},"change_message_id":"348cbb706fe90e0ec3a3f9687db0d987d2247c3a","unresolved":false,"context_lines":[{"line_number":250,"context_line":"  and provide a guidance how to use it (no external dependencies)"},{"line_number":251,"context_line":"* Implement a sample `Strategy`_ using sample Scoring Engine from previous"},{"line_number":252,"context_line":"  point"},{"line_number":253,"context_line":"* Provide documentation of the new Scoring Module"},{"line_number":254,"context_line":""},{"line_number":255,"context_line":"Dependencies"},{"line_number":256,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":4,"id":"1af94dfe_142ec5a4","line":253,"range":{"start_line":253,"start_character":2,"end_line":253,"end_character":49},"in_reply_to":"1af94dfe_82401ca7","updated":"2016-03-17 16:16:53.000000000","message":"+1 :)","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"83ff765da294006d96bb823d860b710be86abba3","unresolved":false,"context_lines":[{"line_number":250,"context_line":"  and provide a guidance how to use it (no external dependencies)"},{"line_number":251,"context_line":"* Implement a sample `Strategy`_ using sample Scoring Engine from previous"},{"line_number":252,"context_line":"  point"},{"line_number":253,"context_line":"* Provide documentation of the new Scoring Module"},{"line_number":254,"context_line":""},{"line_number":255,"context_line":"Dependencies"},{"line_number":256,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa0719c6_ab1733ec","line":253,"range":{"start_line":253,"start_character":2,"end_line":253,"end_character":49},"in_reply_to":"1af94dfe_82401ca7","updated":"2016-03-21 09:00:03.000000000","message":"OK.","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":16495,"name":"Jean-Emile DARTOIS","email":"jean-emile.dartois@b-com.com","username":"JeD"},"change_message_id":"348cbb706fe90e0ec3a3f9687db0d987d2247c3a","unresolved":false,"context_lines":[{"line_number":259,"context_line":""},{"line_number":260,"context_line":"However, in the long run Watcher should provide a flexible plugin model, which"},{"line_number":261,"context_line":"would allow to easily integrate Scoring Engines, `Strategies`_ and `Actions`_"},{"line_number":262,"context_line":"with Watcher without needing to reinstall or upgrade it. The ideal scenario"},{"line_number":263,"context_line":"would be that the third party developers would provide implementations in a"},{"line_number":264,"context_line":"separate repository, which could then be included in one of the Watcher"},{"line_number":265,"context_line":"configuration files. Solving this problem is not in scope of this document."}],"source_content_type":"text/x-rst","patch_set":4,"id":"1af94dfe_6d449c1a","line":262,"updated":"2016-03-17 16:16:53.000000000","message":"We can already do that through the entry points.FYI, we are using the thirdparty stevedor in order to load our plugins.\nhttp://docs.openstack.org/developer/stevedore/tutorial/creating_plugins.html\nyou can take a look here:\nhttps://github.com/openstack/watcher/blob/master/watcher/common/loader/default.py\n\nSo, it is possible to a strategy developer to create his separate repository with some actions or strategies. \n\nYou can take a look to the documentation:\nhttps://forge.b-com.com/www/indeed-ccl/doc/watcher/dev/plugin/base-setup.html","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"83ff765da294006d96bb823d860b710be86abba3","unresolved":false,"context_lines":[{"line_number":259,"context_line":""},{"line_number":260,"context_line":"However, in the long run Watcher should provide a flexible plugin model, which"},{"line_number":261,"context_line":"would allow to easily integrate Scoring Engines, `Strategies`_ and `Actions`_"},{"line_number":262,"context_line":"with Watcher without needing to reinstall or upgrade it. The ideal scenario"},{"line_number":263,"context_line":"would be that the third party developers would provide implementations in a"},{"line_number":264,"context_line":"separate repository, which could then be included in one of the Watcher"},{"line_number":265,"context_line":"configuration files. Solving this problem is not in scope of this document."}],"source_content_type":"text/x-rst","patch_set":4,"id":"fa0719c6_2bb963ea","line":262,"in_reply_to":"1af94dfe_6d449c1a","updated":"2016-03-21 09:00:03.000000000","message":"Yes, I was studying it for the last few days :) Fully agreed here, I will of course do the same - use stevedore.","commit_id":"cf64f25449f741a05d95164ffccf7adade99461e"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"b0ef663b27de6c1359bb375095c7cd19e89096d2","unresolved":false,"context_lines":[{"line_number":8,"context_line":" Provide a scoring module for Watcher"},{"line_number":9,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":10,"context_line":""},{"line_number":11,"context_line":"Include the URL of your launchpad blueprint:"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"https://blueprints.launchpad.net/watcher/+spec/scoring-module"},{"line_number":14,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba0121b8_a0028209","line":11,"updated":"2016-03-29 17:14:10.000000000","message":"These instruction lines should be removed when writing the spec.","commit_id":"1d4084ab138da21ae2ec848374dee8a37df43483"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"ab394456a15f5d6e7547fe331f99c84bf1b954c7","unresolved":false,"context_lines":[{"line_number":8,"context_line":" Provide a scoring module for Watcher"},{"line_number":9,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":10,"context_line":""},{"line_number":11,"context_line":"Include the URL of your launchpad blueprint:"},{"line_number":12,"context_line":""},{"line_number":13,"context_line":"https://blueprints.launchpad.net/watcher/+spec/scoring-module"},{"line_number":14,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba0121b8_654b38ca","line":11,"in_reply_to":"ba0121b8_a0028209","updated":"2016-03-30 10:46:17.000000000","message":"Done","commit_id":"1d4084ab138da21ae2ec848374dee8a37df43483"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"b0ef663b27de6c1359bb375095c7cd19e89096d2","unresolved":false,"context_lines":[{"line_number":63,"context_line":"API."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"As a developer."},{"line_number":66,"context_line":"I want to get a metadata information about a given Scoring Engine. The"},{"line_number":67,"context_line":"metadata might contain an important information how to work with a specific"},{"line_number":68,"context_line":"Scoring Engine, for example: number of input parameters (e.g. number of"},{"line_number":69,"context_line":"features), the response format (e.g. label order used by classifier)."}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba0121b8_235b1c30","line":66,"range":{"start_line":66,"start_character":10,"end_line":66,"end_character":24},"updated":"2016-03-29 17:14:10.000000000","message":"get metadata","commit_id":"1d4084ab138da21ae2ec848374dee8a37df43483"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"ab394456a15f5d6e7547fe331f99c84bf1b954c7","unresolved":false,"context_lines":[{"line_number":63,"context_line":"API."},{"line_number":64,"context_line":""},{"line_number":65,"context_line":"As a developer."},{"line_number":66,"context_line":"I want to get a metadata information about a given Scoring Engine. The"},{"line_number":67,"context_line":"metadata might contain an important information how to work with a specific"},{"line_number":68,"context_line":"Scoring Engine, for example: number of input parameters (e.g. number of"},{"line_number":69,"context_line":"features), the response format (e.g. label order used by classifier)."}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba0121b8_e50d8891","line":66,"range":{"start_line":66,"start_character":10,"end_line":66,"end_character":24},"in_reply_to":"ba0121b8_235b1c30","updated":"2016-03-30 10:46:17.000000000","message":"Done","commit_id":"1d4084ab138da21ae2ec848374dee8a37df43483"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"b0ef663b27de6c1359bb375095c7cd19e89096d2","unresolved":false,"context_lines":[{"line_number":71,"context_line":"Project Priority"},{"line_number":72,"context_line":"----------------"},{"line_number":73,"context_line":""},{"line_number":74,"context_line":"Not relevant because Watcher is not in the big tent so far."},{"line_number":75,"context_line":""},{"line_number":76,"context_line":"Proposed change"},{"line_number":77,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba0121b8_6364f4e4","line":74,"updated":"2016-03-29 17:14:10.000000000","message":"This whole section can be removed, as it isn\u0027t used by projects anymore.","commit_id":"1d4084ab138da21ae2ec848374dee8a37df43483"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"ab394456a15f5d6e7547fe331f99c84bf1b954c7","unresolved":false,"context_lines":[{"line_number":71,"context_line":"Project Priority"},{"line_number":72,"context_line":"----------------"},{"line_number":73,"context_line":""},{"line_number":74,"context_line":"Not relevant because Watcher is not in the big tent so far."},{"line_number":75,"context_line":""},{"line_number":76,"context_line":"Proposed change"},{"line_number":77,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba0121b8_a51390b7","line":74,"in_reply_to":"ba0121b8_6364f4e4","updated":"2016-03-30 10:46:17.000000000","message":"Done","commit_id":"1d4084ab138da21ae2ec848374dee8a37df43483"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"86e061eadf3469c86a7717582dca79ab0936f48b","unresolved":false,"context_lines":[{"line_number":71,"context_line":"Project Priority"},{"line_number":72,"context_line":"----------------"},{"line_number":73,"context_line":""},{"line_number":74,"context_line":"Not relevant because Watcher is not in the big tent so far."},{"line_number":75,"context_line":""},{"line_number":76,"context_line":"Proposed change"},{"line_number":77,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba0121b8_c6420710","line":74,"in_reply_to":"ba0121b8_a51390b7","updated":"2016-03-30 11:12:43.000000000","message":"Actually, this section is required. Removing it broke the Jenkins build!","commit_id":"1d4084ab138da21ae2ec848374dee8a37df43483"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"2b464ca9a1a1bcfc8d094ec7bdbe00ed4c408f0d","unresolved":false,"context_lines":[{"line_number":71,"context_line":"Project Priority"},{"line_number":72,"context_line":"----------------"},{"line_number":73,"context_line":""},{"line_number":74,"context_line":"Not relevant because Watcher is not in the big tent so far."},{"line_number":75,"context_line":""},{"line_number":76,"context_line":"Proposed change"},{"line_number":77,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba0121b8_f7499a61","line":74,"in_reply_to":"ba0121b8_c6420710","updated":"2016-03-30 16:18:29.000000000","message":"Hmmm... that must be a Nova thing, then.","commit_id":"1d4084ab138da21ae2ec848374dee8a37df43483"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"b0ef663b27de6c1359bb375095c7cd19e89096d2","unresolved":false,"context_lines":[{"line_number":84,"context_line":""},{"line_number":85,"context_line":"The Scoring Engines will be used by some `Strategies`_, so it should be as"},{"line_number":86,"context_line":"easy as possible to get a reference to a given Scoring Engine without"},{"line_number":87,"context_line":"having any assumtions about their internal implementation. Therefore the"},{"line_number":88,"context_line":"proposal is to define an abstraction layer in the `Decision Engine`_, which"},{"line_number":89,"context_line":"will provide an abstract ScoringEngine class, which will have to be"},{"line_number":90,"context_line":"implemented by all Scoring Engines and a ScoringFactory class, which will be"},{"line_number":91,"context_line":"used by a `Strategy`_ implementation to select a Scoring Engine to work with."},{"line_number":92,"context_line":""},{"line_number":93,"context_line":"Usage scenarios"},{"line_number":94,"context_line":"---------------"}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba0121b8_c31b401a","line":91,"range":{"start_line":87,"start_character":69,"end_line":91,"end_character":77},"updated":"2016-03-29 17:14:10.000000000","message":"The highlighted text is the \u0027proposed change\u0027. The rest can be eliminated.","commit_id":"1d4084ab138da21ae2ec848374dee8a37df43483"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"ab394456a15f5d6e7547fe331f99c84bf1b954c7","unresolved":false,"context_lines":[{"line_number":84,"context_line":""},{"line_number":85,"context_line":"The Scoring Engines will be used by some `Strategies`_, so it should be as"},{"line_number":86,"context_line":"easy as possible to get a reference to a given Scoring Engine without"},{"line_number":87,"context_line":"having any assumtions about their internal implementation. Therefore the"},{"line_number":88,"context_line":"proposal is to define an abstraction layer in the `Decision Engine`_, which"},{"line_number":89,"context_line":"will provide an abstract ScoringEngine class, which will have to be"},{"line_number":90,"context_line":"implemented by all Scoring Engines and a ScoringFactory class, which will be"},{"line_number":91,"context_line":"used by a `Strategy`_ implementation to select a Scoring Engine to work with."},{"line_number":92,"context_line":""},{"line_number":93,"context_line":"Usage scenarios"},{"line_number":94,"context_line":"---------------"}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba0121b8_c5a3244e","line":91,"range":{"start_line":87,"start_character":69,"end_line":91,"end_character":77},"in_reply_to":"ba0121b8_c31b401a","updated":"2016-03-30 10:46:17.000000000","message":"Done","commit_id":"1d4084ab138da21ae2ec848374dee8a37df43483"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"b0ef663b27de6c1359bb375095c7cd19e89096d2","unresolved":false,"context_lines":[{"line_number":155,"context_line":"        none"},{"line_number":156,"context_line":""},{"line_number":157,"context_line":"      :Result:"},{"line_number":158,"context_line":"        map with metadata, both keys and values as strings"},{"line_number":159,"context_line":""},{"line_number":160,"context_line":"        For example, the metadata can contain the following information (real"},{"line_number":161,"context_line":"        world example):"}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba0121b8_a368cc88","line":158,"range":{"start_line":158,"start_character":8,"end_line":158,"end_character":11},"updated":"2016-03-29 17:14:10.000000000","message":"s/map/dict","commit_id":"1d4084ab138da21ae2ec848374dee8a37df43483"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"ab394456a15f5d6e7547fe331f99c84bf1b954c7","unresolved":false,"context_lines":[{"line_number":155,"context_line":"        none"},{"line_number":156,"context_line":""},{"line_number":157,"context_line":"      :Result:"},{"line_number":158,"context_line":"        map with metadata, both keys and values as strings"},{"line_number":159,"context_line":""},{"line_number":160,"context_line":"        For example, the metadata can contain the following information (real"},{"line_number":161,"context_line":"        world example):"}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba0121b8_ab836155","line":158,"range":{"start_line":158,"start_character":8,"end_line":158,"end_character":11},"in_reply_to":"ba0121b8_a368cc88","updated":"2016-03-30 10:46:17.000000000","message":"Done","commit_id":"1d4084ab138da21ae2ec848374dee8a37df43483"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"b0ef663b27de6c1359bb375095c7cd19e89096d2","unresolved":false,"context_lines":[{"line_number":178,"context_line":"          the algorithm used for learning or how many training records were"},{"line_number":179,"context_line":"          used"},{"line_number":180,"context_line":""},{"line_number":181,"context_line":"    :score:"},{"line_number":182,"context_line":"      Method responsible for performing the actual scoring, such as"},{"line_number":183,"context_line":"      classifying or predicting data"},{"line_number":184,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba0121b8_6378742c","line":181,"range":{"start_line":181,"start_character":5,"end_line":181,"end_character":10},"updated":"2016-03-29 17:14:10.000000000","message":"This name is confusing, because in English \u0027score\u0027 is both a noun and a verb. IOW, the \u0027score()\u0027 method would return a score. Maybe \u0027calculate_score\u0027 instead?","commit_id":"1d4084ab138da21ae2ec848374dee8a37df43483"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"4c7056ed966be0fed1b7090ab1f46f5c89f7782a","unresolved":false,"context_lines":[{"line_number":178,"context_line":"          the algorithm used for learning or how many training records were"},{"line_number":179,"context_line":"          used"},{"line_number":180,"context_line":""},{"line_number":181,"context_line":"    :score:"},{"line_number":182,"context_line":"      Method responsible for performing the actual scoring, such as"},{"line_number":183,"context_line":"      classifying or predicting data"},{"line_number":184,"context_line":""}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba0121b8_faa2fafa","line":181,"range":{"start_line":181,"start_character":5,"end_line":181,"end_character":10},"in_reply_to":"ba0121b8_6378742c","updated":"2016-03-31 11:50:43.000000000","message":"Sorry, somehow I missed that comment. Makes sense, I will rename to \"calculate_score\".","commit_id":"1d4084ab138da21ae2ec848374dee8a37df43483"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"b0ef663b27de6c1359bb375095c7cd19e89096d2","unresolved":false,"context_lines":[{"line_number":183,"context_line":"      classifying or predicting data"},{"line_number":184,"context_line":""},{"line_number":185,"context_line":"      :Input:"},{"line_number":186,"context_line":"        list of doubles (e.g. feature values)"},{"line_number":187,"context_line":""},{"line_number":188,"context_line":"      :Result:"},{"line_number":189,"context_line":"        list of doubles (e.g. classified values, predicted results)"}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba0121b8_e3e004cf","line":186,"range":{"start_line":186,"start_character":16,"end_line":186,"end_character":23},"updated":"2016-03-29 17:14:10.000000000","message":"What is a \u0027double\u0027? A double-precision float? A two-tuple?","commit_id":"1d4084ab138da21ae2ec848374dee8a37df43483"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"4c7056ed966be0fed1b7090ab1f46f5c89f7782a","unresolved":false,"context_lines":[{"line_number":183,"context_line":"      classifying or predicting data"},{"line_number":184,"context_line":""},{"line_number":185,"context_line":"      :Input:"},{"line_number":186,"context_line":"        list of doubles (e.g. feature values)"},{"line_number":187,"context_line":""},{"line_number":188,"context_line":"      :Result:"},{"line_number":189,"context_line":"        list of doubles (e.g. classified values, predicted results)"}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba0121b8_9ad65e4d","line":186,"range":{"start_line":186,"start_character":16,"end_line":186,"end_character":23},"in_reply_to":"ba0121b8_17da4e4c","updated":"2016-03-31 11:50:43.000000000","message":"Changed to \"list of float numbers\" :)","commit_id":"1d4084ab138da21ae2ec848374dee8a37df43483"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"2b464ca9a1a1bcfc8d094ec7bdbe00ed4c408f0d","unresolved":false,"context_lines":[{"line_number":183,"context_line":"      classifying or predicting data"},{"line_number":184,"context_line":""},{"line_number":185,"context_line":"      :Input:"},{"line_number":186,"context_line":"        list of doubles (e.g. feature values)"},{"line_number":187,"context_line":""},{"line_number":188,"context_line":"      :Result:"},{"line_number":189,"context_line":"        list of doubles (e.g. classified values, predicted results)"}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba0121b8_17da4e4c","line":186,"range":{"start_line":186,"start_character":16,"end_line":186,"end_character":23},"in_reply_to":"ba0121b8_2bc6319a","updated":"2016-03-30 16:18:29.000000000","message":"Yeah, Python floats are C doubles under the hood. It would be clearer to simply say \u0027list of floats\u0027.","commit_id":"1d4084ab138da21ae2ec848374dee8a37df43483"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"ab394456a15f5d6e7547fe331f99c84bf1b954c7","unresolved":false,"context_lines":[{"line_number":183,"context_line":"      classifying or predicting data"},{"line_number":184,"context_line":""},{"line_number":185,"context_line":"      :Input:"},{"line_number":186,"context_line":"        list of doubles (e.g. feature values)"},{"line_number":187,"context_line":""},{"line_number":188,"context_line":"      :Result:"},{"line_number":189,"context_line":"        list of doubles (e.g. classified values, predicted results)"}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba0121b8_2bc6319a","line":186,"range":{"start_line":186,"start_character":16,"end_line":186,"end_character":23},"in_reply_to":"ba0121b8_e3e004cf","updated":"2016-03-30 10:46:17.000000000","message":"double means double-precision float numbers in this case. I\u0027ll elaborate in the spec.","commit_id":"1d4084ab138da21ae2ec848374dee8a37df43483"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"b0ef663b27de6c1359bb375095c7cd19e89096d2","unresolved":false,"context_lines":[{"line_number":192,"context_line":""},{"line_number":193,"context_line":"  * New scoring package containing:"},{"line_number":194,"context_line":""},{"line_number":195,"context_line":"    * ScoringFactory class defining a factory for Scoring Engine"},{"line_number":196,"context_line":"      implementations."},{"line_number":197,"context_line":"    * engines subpackage containing implementations of the Scoring Engines."},{"line_number":198,"context_line":"      The Scoring Engines must all extend the ScoringEngine base class."}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba0121b8_23103ca8","line":195,"range":{"start_line":195,"start_character":6,"end_line":195,"end_character":20},"updated":"2016-03-29 17:14:10.000000000","message":"Sounds very Java-ish. What about \u0027AbstractScoringEngine\u0027 as an abstract base class that all ScoringEngine classes must inherit from?","commit_id":"1d4084ab138da21ae2ec848374dee8a37df43483"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"ab394456a15f5d6e7547fe331f99c84bf1b954c7","unresolved":false,"context_lines":[{"line_number":192,"context_line":""},{"line_number":193,"context_line":"  * New scoring package containing:"},{"line_number":194,"context_line":""},{"line_number":195,"context_line":"    * ScoringFactory class defining a factory for Scoring Engine"},{"line_number":196,"context_line":"      implementations."},{"line_number":197,"context_line":"    * engines subpackage containing implementations of the Scoring Engines."},{"line_number":198,"context_line":"      The Scoring Engines must all extend the ScoringEngine base class."}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba0121b8_6b6e6996","line":195,"range":{"start_line":195,"start_character":6,"end_line":195,"end_character":20},"in_reply_to":"ba0121b8_23103ca8","updated":"2016-03-30 10:46:17.000000000","message":"So the abstract class is the one above, this is exactly what it says - a helper factory class, which should help to get a reference to the actual scoring engine (having its ID).","commit_id":"1d4084ab138da21ae2ec848374dee8a37df43483"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"2b464ca9a1a1bcfc8d094ec7bdbe00ed4c408f0d","unresolved":false,"context_lines":[{"line_number":192,"context_line":""},{"line_number":193,"context_line":"  * New scoring package containing:"},{"line_number":194,"context_line":""},{"line_number":195,"context_line":"    * ScoringFactory class defining a factory for Scoring Engine"},{"line_number":196,"context_line":"      implementations."},{"line_number":197,"context_line":"    * engines subpackage containing implementations of the Scoring Engines."},{"line_number":198,"context_line":"      The Scoring Engines must all extend the ScoringEngine base class."}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba0121b8_d7c3b655","line":195,"range":{"start_line":195,"start_character":6,"end_line":195,"end_character":20},"in_reply_to":"ba0121b8_6b6e6996","updated":"2016-03-30 16:18:29.000000000","message":"If it\u0027s a method, why not make it a classmethod of the AbstractScoringEngine class? No need for a separate class from what I see.","commit_id":"1d4084ab138da21ae2ec848374dee8a37df43483"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"4c7056ed966be0fed1b7090ab1f46f5c89f7782a","unresolved":false,"context_lines":[{"line_number":192,"context_line":""},{"line_number":193,"context_line":"  * New scoring package containing:"},{"line_number":194,"context_line":""},{"line_number":195,"context_line":"    * ScoringFactory class defining a factory for Scoring Engine"},{"line_number":196,"context_line":"      implementations."},{"line_number":197,"context_line":"    * engines subpackage containing implementations of the Scoring Engines."},{"line_number":198,"context_line":"      The Scoring Engines must all extend the ScoringEngine base class."}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba0121b8_3ae912e7","line":195,"range":{"start_line":195,"start_character":6,"end_line":195,"end_character":20},"in_reply_to":"ba0121b8_d7c3b655","updated":"2016-03-31 11:50:43.000000000","message":"We\u0027ll see how this will end up, but it might be a few convenience methods at the end. Also, this method doesn\u0027t really belong to AbstractScoringEngine class (separation of concerns), I\u0027ll keep that as is unless you have a strong opinion about that, then let\u0027s discuss...","commit_id":"1d4084ab138da21ae2ec848374dee8a37df43483"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"b0ef663b27de6c1359bb375095c7cd19e89096d2","unresolved":false,"context_lines":[{"line_number":248,"context_line":"understand the fact, that it might change the other use cases. The"},{"line_number":249,"context_line":"recommendation is to update Scoring Engines only for small bug fixing and give"},{"line_number":250,"context_line":"a new ID to the Scoring Engines using different ML algorithm or trained using"},{"line_number":251,"context_line":"different learning data."},{"line_number":252,"context_line":""},{"line_number":253,"context_line":"Alternatives"},{"line_number":254,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba0121b8_0396d8ef","line":251,"updated":"2016-03-29 17:14:10.000000000","message":"Are the IDs generated internally by Watcher, or given by the author of the Engine? IOW, will the ID for the FooScoringEngine be the same across all Watcher deployments, or will it simply be unique within any given deployement?","commit_id":"1d4084ab138da21ae2ec848374dee8a37df43483"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"ab394456a15f5d6e7547fe331f99c84bf1b954c7","unresolved":false,"context_lines":[{"line_number":248,"context_line":"understand the fact, that it might change the other use cases. The"},{"line_number":249,"context_line":"recommendation is to update Scoring Engines only for small bug fixing and give"},{"line_number":250,"context_line":"a new ID to the Scoring Engines using different ML algorithm or trained using"},{"line_number":251,"context_line":"different learning data."},{"line_number":252,"context_line":""},{"line_number":253,"context_line":"Alternatives"},{"line_number":254,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba0121b8_51eeb2ec","line":251,"in_reply_to":"ba0121b8_0396d8ef","updated":"2016-03-30 10:46:17.000000000","message":"IDs will be created by the authors of the Scoring Engines. This way they can control the versioning of the Scoring Engines. They will have to be unique, the naming convention should help with that. So the IDs will be the same across all Watcher deployments.","commit_id":"1d4084ab138da21ae2ec848374dee8a37df43483"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"2b464ca9a1a1bcfc8d094ec7bdbe00ed4c408f0d","unresolved":false,"context_lines":[{"line_number":248,"context_line":"understand the fact, that it might change the other use cases. The"},{"line_number":249,"context_line":"recommendation is to update Scoring Engines only for small bug fixing and give"},{"line_number":250,"context_line":"a new ID to the Scoring Engines using different ML algorithm or trained using"},{"line_number":251,"context_line":"different learning data."},{"line_number":252,"context_line":""},{"line_number":253,"context_line":"Alternatives"},{"line_number":254,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba0121b8_5785e604","line":251,"in_reply_to":"ba0121b8_51eeb2ec","updated":"2016-03-30 16:18:29.000000000","message":"You should include something about how to avoid ID collisions if Watcher isn\u0027t controlling ID generation.","commit_id":"1d4084ab138da21ae2ec848374dee8a37df43483"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"4c7056ed966be0fed1b7090ab1f46f5c89f7782a","unresolved":false,"context_lines":[{"line_number":248,"context_line":"understand the fact, that it might change the other use cases. The"},{"line_number":249,"context_line":"recommendation is to update Scoring Engines only for small bug fixing and give"},{"line_number":250,"context_line":"a new ID to the Scoring Engines using different ML algorithm or trained using"},{"line_number":251,"context_line":"different learning data."},{"line_number":252,"context_line":""},{"line_number":253,"context_line":"Alternatives"},{"line_number":254,"context_line":"------------"}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba0121b8_9a95be34","line":251,"in_reply_to":"ba0121b8_5785e604","updated":"2016-03-31 11:50:43.000000000","message":"I plan to provide good examples and make an additional note in the documentation about that.","commit_id":"1d4084ab138da21ae2ec848374dee8a37df43483"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"b0ef663b27de6c1359bb375095c7cd19e89096d2","unresolved":false,"context_lines":[{"line_number":357,"context_line":""},{"line_number":358,"context_line":"Unit tests will be needed for the code in the new Scoring Module. Implementing"},{"line_number":359,"context_line":"this module will be transparent for the existing Watcher code base, so no"},{"line_number":360,"context_line":"existing tests or functionality will be affected."},{"line_number":361,"context_line":""},{"line_number":362,"context_line":"Documentation Impact"},{"line_number":363,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba0121b8_635a3452","line":360,"updated":"2016-03-29 17:14:10.000000000","message":"Since you are designing this as a plugin-style system, there should be specific interface tests so that plugin authors can verify that their engines are responding correctly.","commit_id":"1d4084ab138da21ae2ec848374dee8a37df43483"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"4c7056ed966be0fed1b7090ab1f46f5c89f7782a","unresolved":false,"context_lines":[{"line_number":357,"context_line":""},{"line_number":358,"context_line":"Unit tests will be needed for the code in the new Scoring Module. Implementing"},{"line_number":359,"context_line":"this module will be transparent for the existing Watcher code base, so no"},{"line_number":360,"context_line":"existing tests or functionality will be affected."},{"line_number":361,"context_line":""},{"line_number":362,"context_line":"Documentation Impact"},{"line_number":363,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba0121b8_dae9f6b8","line":360,"in_reply_to":"ba0121b8_170b6e8f","updated":"2016-03-31 11:50:43.000000000","message":"It\u0027s already proposed :) ScoringEngine is such abstract class (watcher/common package). All Scoring Engines should inherit from it.","commit_id":"1d4084ab138da21ae2ec848374dee8a37df43483"},{"author":{"_account_id":20515,"name":"Tomasz Kaczynski","email":"tomasz.kaczynski@intel.com","username":"tkaczynski"},"change_message_id":"ab394456a15f5d6e7547fe331f99c84bf1b954c7","unresolved":false,"context_lines":[{"line_number":357,"context_line":""},{"line_number":358,"context_line":"Unit tests will be needed for the code in the new Scoring Module. Implementing"},{"line_number":359,"context_line":"this module will be transparent for the existing Watcher code base, so no"},{"line_number":360,"context_line":"existing tests or functionality will be affected."},{"line_number":361,"context_line":""},{"line_number":362,"context_line":"Documentation Impact"},{"line_number":363,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba0121b8_b1c8ee23","line":360,"in_reply_to":"ba0121b8_635a3452","updated":"2016-03-30 10:46:17.000000000","message":"I agree, it would be a nice thing to have. But how do actually achieve that? Should the tests run when Watcher starts and loads the plugins (this is the time when integration happens)? I\u0027m not sure if this is a good idea, as calling a Scoring Engine might have some side effects, only known to the Scoring Engine author and Strategy developer.\n\nBut as a rule of thumb - of course we need tests on all possible levels, I will do my best to implement them.","commit_id":"1d4084ab138da21ae2ec848374dee8a37df43483"},{"author":{"_account_id":1063,"name":"Ed Leafe","email":"ed@leafe.com","username":"ed-leafe"},"change_message_id":"2b464ca9a1a1bcfc8d094ec7bdbe00ed4c408f0d","unresolved":false,"context_lines":[{"line_number":357,"context_line":""},{"line_number":358,"context_line":"Unit tests will be needed for the code in the new Scoring Module. Implementing"},{"line_number":359,"context_line":"this module will be transparent for the existing Watcher code base, so no"},{"line_number":360,"context_line":"existing tests or functionality will be affected."},{"line_number":361,"context_line":""},{"line_number":362,"context_line":"Documentation Impact"},{"line_number":363,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":6,"id":"ba0121b8_170b6e8f","line":360,"in_reply_to":"ba0121b8_b1c8ee23","updated":"2016-03-30 16:18:29.000000000","message":"Hmmm... you could provide an abstract class that plugins inherit from, and clearly mark which methods must not have their signatures changed. Just one idea.","commit_id":"1d4084ab138da21ae2ec848374dee8a37df43483"}]}