)]}'
{"docs/monasca-log-api-kafka.md":[{"author":{"_account_id":2419,"name":"Roland Hochmuth","email":"roland.hochmuth@hpe.com","username":"roland-hochmuth-s"},"change_message_id":"9c554b9e22a286070970a9de729ed9e3b55c28ed","unresolved":false,"context_lines":[{"line_number":81,"context_line":"   of each message prior to sending it"},{"line_number":82,"context_line":"2) log is enriched with property **truncated** set to **true** (```log[\u0027truncated\u0027] \u003d True```)"},{"line_number":83,"context_line":"3) log\u0027s message is truncated by ```diff + TRUNCATED_PROPERTY_SIZE```."},{"line_number":84,"context_line":"  **TRUNCATED_PROPERTY_SIZE** is the size of newly added property."},{"line_number":85,"context_line":""},{"line_number":86,"context_line":"## Configuration"},{"line_number":87,"context_line":""}],"source_content_type":"text/x-markdown","patch_set":13,"id":"7aa08908_e1676ce8","line":84,"range":{"start_line":84,"start_character":2,"end_line":84,"end_character":4},"updated":"2016-06-09 20:04:52.000000000","message":"I guess my concern with this change is that we are now handling something automatically by truncating the message, rather than letting the client receiving an error and taking appropriate actions, such as discarding or truncating themselves if they want.\n\nI\u0027m wondering why a 413 Request Entity Too Large, http error status code wouldn\u0027t be a better return type, so that the client could make the decision on whether to truncate the message, discard, or take some other action.\n\nIs it because the LogStash and Beaver agents would have difficult y handling the response.","commit_id":"0ed1900f4bddfb41bd03b203abb585eadb34c7ef"},{"author":{"_account_id":16168,"name":"Tomasz Trębski","email":"kornicameister@gmail.com","username":"trebskit"},"change_message_id":"710d7ac65949f53597734ba79ca580a04ea7e1d5","unresolved":false,"context_lines":[{"line_number":81,"context_line":"   of each message prior to sending it"},{"line_number":82,"context_line":"2) log is enriched with property **truncated** set to **true** (```log[\u0027truncated\u0027] \u003d True```)"},{"line_number":83,"context_line":"3) log\u0027s message is truncated by ```diff + TRUNCATED_PROPERTY_SIZE```."},{"line_number":84,"context_line":"  **TRUNCATED_PROPERTY_SIZE** is the size of newly added property."},{"line_number":85,"context_line":""},{"line_number":86,"context_line":"## Configuration"},{"line_number":87,"context_line":""}],"source_content_type":"text/x-markdown","patch_set":13,"id":"7aa08908_9bbfd7a9","line":84,"range":{"start_line":84,"start_character":2,"end_line":84,"end_character":4},"in_reply_to":"7aa08908_e1676ce8","updated":"2016-06-10 11:41:03.000000000","message":"See my comment in main view for this change.","commit_id":"0ed1900f4bddfb41bd03b203abb585eadb34c7ef"},{"author":{"_account_id":16168,"name":"Tomasz Trębski","email":"kornicameister@gmail.com","username":"trebskit"},"change_message_id":"6c2133aadc1385e6192697cc80cb898c5a1271c7","unresolved":false,"context_lines":[{"line_number":76,"context_line":"comply to Kafka limitation about [maximum message size](https://kafka.apache.org/08/configuration.html)."},{"line_number":77,"context_line":"If JSON message is too big following actions are taken"},{"line_number":78,"context_line":"1) difference between maximum allowed size and JSON message size (both in bytes)."},{"line_number":79,"context_line":"   ```diff \u003d (size(json_envelope) + size(envelope_key) + KAFKA_METADATA_SIZE) - maximum_allowed_size + _TRUNCATION_SAFE_OFFSET```."},{"line_number":80,"context_line":"   **KAFKA_METADATA_SIZE** is amount of bytes Kafka adds during transformation"},{"line_number":81,"context_line":"   of each message prior to sending it"},{"line_number":82,"context_line":"2) log is enriched with property **truncated** set to **true** (```log[\u0027truncated\u0027] \u003d True```)"}],"source_content_type":"text/x-markdown","patch_set":22,"id":"7aa08908_551c2c6a","line":79,"range":{"start_line":79,"start_character":41,"end_line":79,"end_character":53},"updated":"2016-06-16 04:04:00.000000000","message":"This field needs explanation.","commit_id":"a4d12773d857279b99ff9029636a9df13e6cd70b"},{"author":{"_account_id":16168,"name":"Tomasz Trębski","email":"kornicameister@gmail.com","username":"trebskit"},"change_message_id":"fc91e801ca8b06b1f14c2a706f4a927297eb6e25","unresolved":false,"context_lines":[{"line_number":76,"context_line":"comply to Kafka limitation about [maximum message size](https://kafka.apache.org/08/configuration.html)."},{"line_number":77,"context_line":"If JSON message is too big following actions are taken"},{"line_number":78,"context_line":"1) difference between maximum allowed size and JSON message size (both in bytes)."},{"line_number":79,"context_line":"   ```diff \u003d (size(json_envelope) + size(envelope_key) + KAFKA_METADATA_SIZE) - maximum_allowed_size + _TRUNCATION_SAFE_OFFSET```."},{"line_number":80,"context_line":"   **KAFKA_METADATA_SIZE** is amount of bytes Kafka adds during transformation"},{"line_number":81,"context_line":"   of each message prior to sending it"},{"line_number":82,"context_line":"2) log is enriched with property **truncated** set to **true** (```log[\u0027truncated\u0027] \u003d True```)"}],"source_content_type":"text/x-markdown","patch_set":22,"id":"7aa08908_bb561ecc","line":79,"range":{"start_line":79,"start_character":41,"end_line":79,"end_character":53},"in_reply_to":"7aa08908_551c2c6a","updated":"2016-06-16 07:41:20.000000000","message":"Done","commit_id":"a4d12773d857279b99ff9029636a9df13e6cd70b"}],"docs/monasca-log-api-spec.md":[{"author":{"_account_id":16222,"name":"witek","email":"witold.bedyk@suse.com","username":"witek"},"change_message_id":"83f891060a1b4f3846cc5f7da02420f0269a3a12","unresolved":false,"context_lines":[{"line_number":5,"context_line":"Document Version: v2.2.2"},{"line_number":6,"context_line":""},{"line_number":7,"context_line":"# Publishing logs to Kafka"},{"line_number":8,"context_line":"Each log that is received either via **v2.0** and **v3.0** is wrapped"},{"line_number":9,"context_line":"by structure called *envelope*"},{"line_number":10,"context_line":"```"},{"line_number":11,"context_line":"{"},{"line_number":12,"context_line":"  \"meta\": \u003cobject\u003e,"},{"line_number":13,"context_line":"  \"log\": \u003cobject\u003e,"},{"line_number":14,"context_line":"  \"creation_time\": \u003clong\u003e"},{"line_number":15,"context_line":"}"},{"line_number":16,"context_line":"```"},{"line_number":17,"context_line":""},{"line_number":18,"context_line":"*envelope* is the actual message that is sent to Kafka and it should be expected"},{"line_number":19,"context_line":"in the same form for any client reading from Kafka."},{"line_number":20,"context_line":""},{"line_number":21,"context_line":"## Truncating too large message"},{"line_number":22,"context_line":""}],"source_content_type":"text/x-markdown","patch_set":11,"id":"7aa08908_9c00f567","line":19,"range":{"start_line":8,"start_character":0,"end_line":19,"end_character":51},"updated":"2016-06-08 13:38:34.000000000","message":"That information is already in docs/monasca-log-api-kafka.md","commit_id":"06588be8973d253bcf27d3d719f894d38577cf18"},{"author":{"_account_id":16168,"name":"Tomasz Trębski","email":"kornicameister@gmail.com","username":"trebskit"},"change_message_id":"84cf1f53e62fb3a88bf42ea05167f02c31a63f41","unresolved":false,"context_lines":[{"line_number":5,"context_line":"Document Version: v2.2.2"},{"line_number":6,"context_line":""},{"line_number":7,"context_line":"# Publishing logs to Kafka"},{"line_number":8,"context_line":"Each log that is received either via **v2.0** and **v3.0** is wrapped"},{"line_number":9,"context_line":"by structure called *envelope*"},{"line_number":10,"context_line":"```"},{"line_number":11,"context_line":"{"},{"line_number":12,"context_line":"  \"meta\": \u003cobject\u003e,"},{"line_number":13,"context_line":"  \"log\": \u003cobject\u003e,"},{"line_number":14,"context_line":"  \"creation_time\": \u003clong\u003e"},{"line_number":15,"context_line":"}"},{"line_number":16,"context_line":"```"},{"line_number":17,"context_line":""},{"line_number":18,"context_line":"*envelope* is the actual message that is sent to Kafka and it should be expected"},{"line_number":19,"context_line":"in the same form for any client reading from Kafka."},{"line_number":20,"context_line":""},{"line_number":21,"context_line":"## Truncating too large message"},{"line_number":22,"context_line":""}],"source_content_type":"text/x-markdown","patch_set":11,"id":"7aa08908_05c90fac","line":19,"range":{"start_line":8,"start_character":0,"end_line":19,"end_character":51},"in_reply_to":"7aa08908_9c00f567","updated":"2016-06-09 07:56:27.000000000","message":"Actually...I haven\u0027t checked that file :)","commit_id":"06588be8973d253bcf27d3d719f894d38577cf18"},{"author":{"_account_id":16222,"name":"witek","email":"witold.bedyk@suse.com","username":"witek"},"change_message_id":"83f891060a1b4f3846cc5f7da02420f0269a3a12","unresolved":false,"context_lines":[{"line_number":18,"context_line":"*envelope* is the actual message that is sent to Kafka and it should be expected"},{"line_number":19,"context_line":"in the same form for any client reading from Kafka."},{"line_number":20,"context_line":""},{"line_number":21,"context_line":"## Truncating too large message"},{"line_number":22,"context_line":""},{"line_number":23,"context_line":"    Following section mostly applies to v3.0"},{"line_number":24,"context_line":"    "}],"source_content_type":"text/x-markdown","patch_set":11,"id":"7aa08908_9c8595d0","line":21,"range":{"start_line":21,"start_character":0,"end_line":21,"end_character":31},"updated":"2016-06-08 13:38:34.000000000","message":"I don\u0027t like having this section at the beginning of the document.","commit_id":"06588be8973d253bcf27d3d719f894d38577cf18"}],"monasca_log_api/reference/common/log_publisher.py":[{"author":{"_account_id":18286,"name":"Shinya Kawabata","email":"s-kawabata@wx.jp.nec.com","username":"kawabata"},"change_message_id":"a7d86e5f2b18013b2c7583fd0e0fe99babb3c018","unresolved":false,"context_lines":[{"line_number":142,"context_line":"    def _truncate(self, envelope):"},{"line_number":143,"context_line":"        \"\"\"Truncates the message if needed."},{"line_number":144,"context_line":""},{"line_number":145,"context_line":"        Each messsage send to kafka is verified."},{"line_number":146,"context_line":"        Method checks if message serialized to json"},{"line_number":147,"context_line":"        exceeds maximum allowed size that can be posted to kafka"},{"line_number":148,"context_line":"        queue. If so, method truncates message property of the log"}],"source_content_type":"text/x-python","patch_set":3,"id":"bab6814e_62c9cdac","line":145,"range":{"start_line":145,"start_character":13,"end_line":145,"end_character":21},"updated":"2016-05-24 09:35:31.000000000","message":"s/messsage/message/","commit_id":"5022fdbcc2c3c516750821b44cea3dedf69bea46"},{"author":{"_account_id":16168,"name":"Tomasz Trębski","email":"kornicameister@gmail.com","username":"trebskit"},"change_message_id":"e56b5de1bb7b4d24ebee4489ce7c1f8ef60a8c10","unresolved":false,"context_lines":[{"line_number":142,"context_line":"    def _truncate(self, envelope):"},{"line_number":143,"context_line":"        \"\"\"Truncates the message if needed."},{"line_number":144,"context_line":""},{"line_number":145,"context_line":"        Each messsage send to kafka is verified."},{"line_number":146,"context_line":"        Method checks if message serialized to json"},{"line_number":147,"context_line":"        exceeds maximum allowed size that can be posted to kafka"},{"line_number":148,"context_line":"        queue. If so, method truncates message property of the log"}],"source_content_type":"text/x-python","patch_set":3,"id":"bab6814e_f268e489","line":145,"range":{"start_line":145,"start_character":13,"end_line":145,"end_character":21},"in_reply_to":"bab6814e_62c9cdac","updated":"2016-05-24 11:19:46.000000000","message":"Done","commit_id":"5022fdbcc2c3c516750821b44cea3dedf69bea46"},{"author":{"_account_id":16168,"name":"Tomasz Trębski","email":"kornicameister@gmail.com","username":"trebskit"},"change_message_id":"ec76d46c41fa9c6c3619d59a4bd2e913663eb89a","unresolved":false,"context_lines":[{"line_number":171,"context_line":"            log_msg \u003d envelope[\u0027log\u0027][\u0027message\u0027]"},{"line_number":172,"context_line":"            truncated_log_msg \u003d log_msg[:-diff]"},{"line_number":173,"context_line":""},{"line_number":174,"context_line":"            envelope[\u0027log\u0027][\u0027message\u0027] \u003d truncated_log_msg"},{"line_number":175,"context_line":""},{"line_number":176,"context_line":"            # TODO(trebskit) should log be enriched with truncated property"},{"line_number":177,"context_line":"            # telling what just happened? That would require truncating"}],"source_content_type":"text/x-python","patch_set":5,"id":"bab6814e_9baff37c","line":174,"range":{"start_line":174,"start_character":41,"end_line":174,"end_character":58},"updated":"2016-05-24 12:11:12.000000000","message":"Enrich log property with truncated property.","commit_id":"3ebb2ecd69ff4c321af79d90fa4a5f2ea9edcb17"},{"author":{"_account_id":14273,"name":"Joe Keen","email":"joe.keen@hpe.com","username":"jkeen"},"change_message_id":"794a5cb15a0da2faaf76b295c773600857170c31","unresolved":false,"context_lines":[{"line_number":240,"context_line":""},{"line_number":241,"context_line":"                # keep on track how many msgs have been sent"},{"line_number":242,"context_line":"                sent_counter +\u003d len(bucket)"},{"line_number":243,"context_line":"        except kafka_common.KafkaUnavailableError as kue:"},{"line_number":244,"context_line":"            LOG.error(\u0027Kafka not available at %s\u0027,"},{"line_number":245,"context_line":"                      CONF.log_publisher.kafka_url)"},{"line_number":246,"context_line":"            LOG.exception(kue)"}],"source_content_type":"text/x-python","patch_set":5,"id":"bab6814e_b2800a48","line":243,"range":{"start_line":243,"start_character":15,"end_line":243,"end_character":27},"updated":"2016-05-24 15:34:42.000000000","message":"Should we push this trap into the Kafka publisher in monasca common?  Making users of our publisher understand kafka specific exceptions is something I wanted to avoid if possible.","commit_id":"3ebb2ecd69ff4c321af79d90fa4a5f2ea9edcb17"},{"author":{"_account_id":16168,"name":"Tomasz Trębski","email":"kornicameister@gmail.com","username":"trebskit"},"change_message_id":"b8c43c34ce8d47f72e6de5cb3b09563796e27aaa","unresolved":false,"context_lines":[{"line_number":240,"context_line":""},{"line_number":241,"context_line":"                # keep on track how many msgs have been sent"},{"line_number":242,"context_line":"                sent_counter +\u003d len(bucket)"},{"line_number":243,"context_line":"        except kafka_common.KafkaUnavailableError as kue:"},{"line_number":244,"context_line":"            LOG.error(\u0027Kafka not available at %s\u0027,"},{"line_number":245,"context_line":"                      CONF.log_publisher.kafka_url)"},{"line_number":246,"context_line":"            LOG.exception(kue)"}],"source_content_type":"text/x-python","patch_set":5,"id":"bab6814e_e0b620f8","line":243,"range":{"start_line":243,"start_character":15,"end_line":243,"end_character":27},"in_reply_to":"bab6814e_b2800a48","updated":"2016-05-25 06:06:21.000000000","message":"Yes, we can do that but I am not sure how.\n\nBasically here I need to know this particular error because that will be the only error that means \u0027stop sending\u0027. And the change is here because Roland merged Arthur\u0027s change before Arthur could mark that with -1 @ Workflow and do what I\u0027ve done here.\n\nAs for the common, all I can think of right now would be catching an exception for kafka and wrapping it by...let\u0027s say \n\n  KafkaException\n\nthat would be part of monasca-common. In other words users of monasca-common would be presented with single and named exception coming from I/O operations with Kafka. What about that idea ?","commit_id":"3ebb2ecd69ff4c321af79d90fa4a5f2ea9edcb17"},{"author":{"_account_id":14273,"name":"Joe Keen","email":"joe.keen@hpe.com","username":"jkeen"},"change_message_id":"f11021c0bde899380a6b2d7c3b142d9b6697b1ae","unresolved":false,"context_lines":[{"line_number":240,"context_line":""},{"line_number":241,"context_line":"                # keep on track how many msgs have been sent"},{"line_number":242,"context_line":"                sent_counter +\u003d len(bucket)"},{"line_number":243,"context_line":"        except kafka_common.KafkaUnavailableError as kue:"},{"line_number":244,"context_line":"            LOG.error(\u0027Kafka not available at %s\u0027,"},{"line_number":245,"context_line":"                      CONF.log_publisher.kafka_url)"},{"line_number":246,"context_line":"            LOG.exception(kue)"}],"source_content_type":"text/x-python","patch_set":5,"id":"bab6814e_b102813c","line":243,"range":{"start_line":243,"start_character":15,"end_line":243,"end_character":27},"in_reply_to":"bab6814e_e0b620f8","updated":"2016-05-25 17:49:46.000000000","message":"I think that\u0027s a good idea.  Right now the publisher is trapping any exceptions, logging, and raising.  More specific traps with better error logging would be better.\n\nIn this case if the publisher trapped the KafkaUnavailableError and logged properly it doesn\u0027t look like there would be any change necessary to your existing exception handling.  Does that sound right?","commit_id":"3ebb2ecd69ff4c321af79d90fa4a5f2ea9edcb17"},{"author":{"_account_id":16168,"name":"Tomasz Trębski","email":"kornicameister@gmail.com","username":"trebskit"},"change_message_id":"667b8dfed90ece606babcb0759b4bb5138d874f8","unresolved":false,"context_lines":[{"line_number":181,"context_line":"            envelope[\u0027log\u0027][\u0027message\u0027] \u003d truncated_log_msg"},{"line_number":182,"context_line":""},{"line_number":183,"context_line":"            # will just transform message once again without truncation"},{"line_number":184,"context_line":"            return self._truncate(envelope)"},{"line_number":185,"context_line":""},{"line_number":186,"context_line":"        return msg_str"},{"line_number":187,"context_line":""}],"source_content_type":"text/x-python","patch_set":9,"id":"9abb7d3a_4a3ad237","line":184,"range":{"start_line":184,"start_character":24,"end_line":184,"end_character":33},"updated":"2016-05-27 22:15:26.000000000","message":"TODO: Since truncating was done - sufficient would be to just call\n\n rest_utils.as_json(envelope)","commit_id":"d9cd8049d9300b43264c9b532f8aef1e25887a59"},{"author":{"_account_id":16222,"name":"witek","email":"witold.bedyk@suse.com","username":"witek"},"change_message_id":"83f891060a1b4f3846cc5f7da02420f0269a3a12","unresolved":false,"context_lines":[{"line_number":161,"context_line":"        envelope_size \u003d ((len(bytearray(msg_str)) + _KAFKA_META_DATA_SIZE)"},{"line_number":162,"context_line":"                         if msg_str is not None else -1)"},{"line_number":163,"context_line":""},{"line_number":164,"context_line":"        size_diff \u003d envelope_size - max_size + 1"},{"line_number":165,"context_line":""},{"line_number":166,"context_line":"        LOG.debug(\u0027_truncate(max_message_size\u003d%d, message_size\u003d%d, diff\u003d%d)\u0027,"},{"line_number":167,"context_line":"                  max_size, envelope_size, size_diff)"}],"source_content_type":"text/x-python","patch_set":11,"id":"7aa08908_ca6c6b0a","line":164,"range":{"start_line":164,"start_character":8,"end_line":164,"end_character":48},"updated":"2016-06-08 13:38:34.000000000","message":"Please move the \u0027+ 1\u0027 to the truncate_by variable. That way the variable value is consistent with the variable name.","commit_id":"06588be8973d253bcf27d3d719f894d38577cf18"},{"author":{"_account_id":16222,"name":"witek","email":"witold.bedyk@suse.com","username":"witek"},"change_message_id":"8ceef12d84dc71a8f94c2da05d48364173d705a2","unresolved":false,"context_lines":[{"line_number":161,"context_line":"        envelope_size \u003d ((len(bytearray(msg_str)) + _KAFKA_META_DATA_SIZE)"},{"line_number":162,"context_line":"                         if msg_str is not None else -1)"},{"line_number":163,"context_line":""},{"line_number":164,"context_line":"        size_diff \u003d envelope_size - max_size + 1"},{"line_number":165,"context_line":""},{"line_number":166,"context_line":"        LOG.debug(\u0027_truncate(max_message_size\u003d%d, message_size\u003d%d, diff\u003d%d)\u0027,"},{"line_number":167,"context_line":"                  max_size, envelope_size, size_diff)"}],"source_content_type":"text/x-python","patch_set":11,"id":"7aa08908_d4de973c","line":164,"range":{"start_line":164,"start_character":8,"end_line":164,"end_character":48},"in_reply_to":"7aa08908_52447f48","updated":"2016-06-09 10:24:29.000000000","message":"Actually I meant:\n\n size_diff \u003d envelope_size - max_size","commit_id":"06588be8973d253bcf27d3d719f894d38577cf18"},{"author":{"_account_id":16168,"name":"Tomasz Trębski","email":"kornicameister@gmail.com","username":"trebskit"},"change_message_id":"84cf1f53e62fb3a88bf42ea05167f02c31a63f41","unresolved":false,"context_lines":[{"line_number":161,"context_line":"        envelope_size \u003d ((len(bytearray(msg_str)) + _KAFKA_META_DATA_SIZE)"},{"line_number":162,"context_line":"                         if msg_str is not None else -1)"},{"line_number":163,"context_line":""},{"line_number":164,"context_line":"        size_diff \u003d envelope_size - max_size + 1"},{"line_number":165,"context_line":""},{"line_number":166,"context_line":"        LOG.debug(\u0027_truncate(max_message_size\u003d%d, message_size\u003d%d, diff\u003d%d)\u0027,"},{"line_number":167,"context_line":"                  max_size, envelope_size, size_diff)"}],"source_content_type":"text/x-python","patch_set":11,"id":"7aa08908_52447f48","line":164,"range":{"start_line":164,"start_character":8,"end_line":164,"end_character":48},"in_reply_to":"7aa08908_ca6c6b0a","updated":"2016-06-09 07:56:27.000000000","message":"Done","commit_id":"06588be8973d253bcf27d3d719f894d38577cf18"},{"author":{"_account_id":7874,"name":"Venkat Sundaram","email":"tsv@hp.com","username":"tsv"},"change_message_id":"db92dbce7c84070adfda01a874f35070acce8e95","unresolved":false,"context_lines":[{"line_number":174,"context_line":"                     max_size,"},{"line_number":175,"context_line":"                     truncate_by)"},{"line_number":176,"context_line":""},{"line_number":177,"context_line":"            log_msg \u003d envelope[\u0027log\u0027][\u0027message\u0027]"},{"line_number":178,"context_line":"            truncated_log_msg \u003d log_msg[:-truncate_by]"},{"line_number":179,"context_line":""},{"line_number":180,"context_line":"            envelope[\u0027log\u0027][\u0027truncated\u0027] \u003d True"}],"source_content_type":"text/x-python","patch_set":11,"id":"7aa08908_b30fcc35","line":177,"range":{"start_line":177,"start_character":39,"end_line":177,"end_character":46},"updated":"2016-06-08 16:05:26.000000000","message":"Though in most cases the \"message\" field should be the biggest field, what will happen if say, one ore more dimensions caused the size to exceed ? In other words, could there be a chance where the size is still over even after the truncation ?\nIn that case, should the message be dropped instead ?","commit_id":"06588be8973d253bcf27d3d719f894d38577cf18"},{"author":{"_account_id":16168,"name":"Tomasz Trębski","email":"kornicameister@gmail.com","username":"trebskit"},"change_message_id":"84cf1f53e62fb3a88bf42ea05167f02c31a63f41","unresolved":false,"context_lines":[{"line_number":174,"context_line":"                     max_size,"},{"line_number":175,"context_line":"                     truncate_by)"},{"line_number":176,"context_line":""},{"line_number":177,"context_line":"            log_msg \u003d envelope[\u0027log\u0027][\u0027message\u0027]"},{"line_number":178,"context_line":"            truncated_log_msg \u003d log_msg[:-truncate_by]"},{"line_number":179,"context_line":""},{"line_number":180,"context_line":"            envelope[\u0027log\u0027][\u0027truncated\u0027] \u003d True"}],"source_content_type":"text/x-python","patch_set":11,"id":"7aa08908_92b8a72b","line":177,"range":{"start_line":177,"start_character":39,"end_line":177,"end_character":46},"in_reply_to":"7aa08908_b30fcc35","updated":"2016-06-09 07:56:27.000000000","message":"Validating dimensions (global or local) covers that case. Both name and value are validated that way and both need to comply to 255 chars limit. \n\nAs we speak invalid dimensions will result in rejecting single log or entire bulk. However rejecting entire bulk is not that good that\u0027s why I am working on removing that flaw and ensuring that most of the bulk elements will be passed on. Truncation here is first step toward that.","commit_id":"06588be8973d253bcf27d3d719f894d38577cf18"},{"author":{"_account_id":16222,"name":"witek","email":"witold.bedyk@suse.com","username":"witek"},"change_message_id":"d9b44658a77138e25a6b6cb527edbe36849d4608","unresolved":false,"context_lines":[{"line_number":142,"context_line":""},{"line_number":143,"context_line":"        return True"},{"line_number":144,"context_line":""},{"line_number":145,"context_line":"    def _truncate(self, envelope, key_byte_size):"},{"line_number":146,"context_line":"        \"\"\"Truncates the message if needed."},{"line_number":147,"context_line":""},{"line_number":148,"context_line":"        Each message send to kafka is verified."}],"source_content_type":"text/x-python","patch_set":18,"id":"7aa08908_c59ac013","line":145,"range":{"start_line":145,"start_character":34,"end_line":145,"end_character":47},"updated":"2016-06-14 07:42:36.000000000","message":"We won\u0027t need that as we want to get rid of sending the partition key to the KafkaProducer.","commit_id":"246577a158727d86daa9e7ed59eeeb18153f99ed"},{"author":{"_account_id":16168,"name":"Tomasz Trębski","email":"kornicameister@gmail.com","username":"trebskit"},"change_message_id":"05d4d8820e5958fb391f2f2e424ed7b1ea78c2db","unresolved":false,"context_lines":[{"line_number":142,"context_line":""},{"line_number":143,"context_line":"        return True"},{"line_number":144,"context_line":""},{"line_number":145,"context_line":"    def _truncate(self, envelope, key_byte_size):"},{"line_number":146,"context_line":"        \"\"\"Truncates the message if needed."},{"line_number":147,"context_line":""},{"line_number":148,"context_line":"        Each message send to kafka is verified."}],"source_content_type":"text/x-python","patch_set":18,"id":"7aa08908_31291fe7","line":145,"range":{"start_line":145,"start_character":34,"end_line":145,"end_character":47},"in_reply_to":"7aa08908_c59ac013","updated":"2016-06-14 10:59:22.000000000","message":"Once change for removing keys is up I will remove that, right now I haven\u0027t seen change from Koji that would implement that.","commit_id":"246577a158727d86daa9e7ed59eeeb18153f99ed"},{"author":{"_account_id":16222,"name":"witek","email":"witold.bedyk@suse.com","username":"witek"},"change_message_id":"d9b44658a77138e25a6b6cb527edbe36849d4608","unresolved":false,"context_lines":[{"line_number":164,"context_line":"                          _KAFKA_META_DATA_SIZE)"},{"line_number":165,"context_line":"                         if msg_str is not None else -1)"},{"line_number":166,"context_line":""},{"line_number":167,"context_line":"        size_diff \u003d (envelope_size - max_size) + _TRUNCATION_SAFE_OFFSET"},{"line_number":168,"context_line":""},{"line_number":169,"context_line":"        LOG.debug(\u0027_truncate(max_message_size\u003d%d, message_size\u003d%d, diff\u003d%d)\u0027,"},{"line_number":170,"context_line":"                  max_size, envelope_size, size_diff)"}],"source_content_type":"text/x-python","patch_set":18,"id":"7aa08908_f9414baa","line":167,"range":{"start_line":167,"start_character":8,"end_line":167,"end_character":17},"updated":"2016-06-14 07:42:36.000000000","message":"Minor, the difference should really be the difference:\n\n size_diff \u003d envelope_size - max_size\n\nThen update lines 172-173 accordingly.","commit_id":"246577a158727d86daa9e7ed59eeeb18153f99ed"}],"monasca_log_api/tests/test_log_publisher.py":[{"author":{"_account_id":16168,"name":"Tomasz Trębski","email":"kornicameister@gmail.com","username":"trebskit"},"change_message_id":"f09cc31c0e7e2a6d2ccd21f21eceaada8cbab5ab","unresolved":false,"context_lines":[{"line_number":402,"context_line":"        max_message_size \u003d 1000"},{"line_number":403,"context_line":"        log_message_size \u003d (max_message_size -"},{"line_number":404,"context_line":"                            TestTruncation.EXTRA_CHARS_SIZE -"},{"line_number":405,"context_line":"                            log_publisher._KAFKA_META_DATA_SIZE)"},{"line_number":406,"context_line":"        expected_log_message_size \u003d log_message_size"},{"line_number":407,"context_line":""},{"line_number":408,"context_line":"        self._conf.config("}],"source_content_type":"text/x-python","patch_set":8,"id":"9abb7d3a_8ac79a44","line":405,"updated":"2016-05-27 22:11:04.000000000","message":"I think I have two tests doing the same thing.","commit_id":"ec770c62501f91f82d5a9615f32076e87dde42a4"},{"author":{"_account_id":16222,"name":"witek","email":"witold.bedyk@suse.com","username":"witek"},"change_message_id":"83f891060a1b4f3846cc5f7da02420f0269a3a12","unresolved":false,"context_lines":[{"line_number":30,"context_line":""},{"line_number":31,"context_line":""},{"line_number":32,"context_line":"def _generate_unique_message(size):"},{"line_number":33,"context_line":"    letters \u003d string.ascii_lowercase"},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"    def rand(amount, space\u003dTrue):"},{"line_number":36,"context_line":"        space \u003d \u0027 \u0027 if space else \u0027\u0027"}],"source_content_type":"text/x-python","patch_set":11,"id":"7aa08908_5de5e4a5","line":33,"range":{"start_line":33,"start_character":4,"end_line":33,"end_character":36},"updated":"2016-06-08 13:38:34.000000000","message":"Could you also add a test using multibyte characters message please.","commit_id":"06588be8973d253bcf27d3d719f894d38577cf18"},{"author":{"_account_id":16168,"name":"Tomasz Trębski","email":"kornicameister@gmail.com","username":"trebskit"},"change_message_id":"84cf1f53e62fb3a88bf42ea05167f02c31a63f41","unresolved":false,"context_lines":[{"line_number":30,"context_line":""},{"line_number":31,"context_line":""},{"line_number":32,"context_line":"def _generate_unique_message(size):"},{"line_number":33,"context_line":"    letters \u003d string.ascii_lowercase"},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"    def rand(amount, space\u003dTrue):"},{"line_number":36,"context_line":"        space \u003d \u0027 \u0027 if space else \u0027\u0027"}],"source_content_type":"text/x-python","patch_set":11,"id":"7aa08908_c50707c8","line":33,"range":{"start_line":33,"start_character":4,"end_line":33,"end_character":36},"in_reply_to":"7aa08908_327a9324","updated":"2016-06-09 07:56:27.000000000","message":"I have stashed those changes for now.\nAnyway the problem would be that I would have to hardcode those changes which would not be good, I think that requires more time to think about that and at least having Koji change published in pypi with newest monasca-common release.","commit_id":"06588be8973d253bcf27d3d719f894d38577cf18"},{"author":{"_account_id":16168,"name":"Tomasz Trębski","email":"kornicameister@gmail.com","username":"trebskit"},"change_message_id":"2942570ba6cb02db5a2c879981a5bd50b067588b","unresolved":false,"context_lines":[{"line_number":30,"context_line":""},{"line_number":31,"context_line":""},{"line_number":32,"context_line":"def _generate_unique_message(size):"},{"line_number":33,"context_line":"    letters \u003d string.ascii_lowercase"},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"    def rand(amount, space\u003dTrue):"},{"line_number":36,"context_line":"        space \u003d \u0027 \u0027 if space else \u0027\u0027"}],"source_content_type":"text/x-python","patch_set":11,"id":"7aa08908_bd9ad911","line":33,"range":{"start_line":33,"start_character":4,"end_line":33,"end_character":36},"in_reply_to":"7aa08908_5de5e4a5","updated":"2016-06-09 07:15:24.000000000","message":"I could but monasca-common that would contain fix from Koji has been deployed yet so they will fail anyway. Another issue is that I am having troubles with finding deterministic way to generate random multi byte message there are always minor differences between size of the messages (5-15 characters, depends on the test run).\n\nIf that is really necessary I will dig deeper for the solution.","commit_id":"06588be8973d253bcf27d3d719f894d38577cf18"},{"author":{"_account_id":16222,"name":"witek","email":"witold.bedyk@suse.com","username":"witek"},"change_message_id":"7789473cb4154bc512b1309e39dafe192f137a35","unresolved":false,"context_lines":[{"line_number":30,"context_line":""},{"line_number":31,"context_line":""},{"line_number":32,"context_line":"def _generate_unique_message(size):"},{"line_number":33,"context_line":"    letters \u003d string.ascii_lowercase"},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"    def rand(amount, space\u003dTrue):"},{"line_number":36,"context_line":"        space \u003d \u0027 \u0027 if space else \u0027\u0027"}],"source_content_type":"text/x-python","patch_set":11,"id":"7aa08908_327a9324","line":33,"range":{"start_line":33,"start_character":4,"end_line":33,"end_character":36},"in_reply_to":"7aa08908_bd9ad911","updated":"2016-06-09 07:41:17.000000000","message":"Koji\u0027s change 325740 is about using multibyte characters in Kafka partition keys. Since we don\u0027t use the message to generate the key it is not relevant here.\n\nThe message would not have to be random. Perhaps the placing of my comment was not the best one. I just would like to see if truncation of multibyte character message works properly.","commit_id":"06588be8973d253bcf27d3d719f894d38577cf18"},{"author":{"_account_id":16222,"name":"witek","email":"witold.bedyk@suse.com","username":"witek"},"change_message_id":"83f891060a1b4f3846cc5f7da02420f0269a3a12","unresolved":false,"context_lines":[{"line_number":421,"context_line":"        instance \u003d log_publisher.LogPublisher()"},{"line_number":422,"context_line":""},{"line_number":423,"context_line":"        envelope_copy \u003d copy.deepcopy(envelope)"},{"line_number":424,"context_line":"        json_envelope \u003d instance._truncate(envelope_copy)"},{"line_number":425,"context_line":""},{"line_number":426,"context_line":"        parsed_envelope \u003d ujson.loads(json_envelope)"},{"line_number":427,"context_line":"        parsed_log_message \u003d parsed_envelope[\u0027log\u0027][\u0027message\u0027]"}],"source_content_type":"text/x-python","patch_set":11,"id":"7aa08908_7d66e051","line":424,"range":{"start_line":424,"start_character":24,"end_line":424,"end_character":57},"updated":"2016-06-08 13:38:34.000000000","message":"I would prefer to test the send_message method instead. The test criteria would be easier to build and read.","commit_id":"06588be8973d253bcf27d3d719f894d38577cf18"},{"author":{"_account_id":16168,"name":"Tomasz Trębski","email":"kornicameister@gmail.com","username":"trebskit"},"change_message_id":"84cf1f53e62fb3a88bf42ea05167f02c31a63f41","unresolved":false,"context_lines":[{"line_number":421,"context_line":"        instance \u003d log_publisher.LogPublisher()"},{"line_number":422,"context_line":""},{"line_number":423,"context_line":"        envelope_copy \u003d copy.deepcopy(envelope)"},{"line_number":424,"context_line":"        json_envelope \u003d instance._truncate(envelope_copy)"},{"line_number":425,"context_line":""},{"line_number":426,"context_line":"        parsed_envelope \u003d ujson.loads(json_envelope)"},{"line_number":427,"context_line":"        parsed_log_message \u003d parsed_envelope[\u0027log\u0027][\u0027message\u0027]"}],"source_content_type":"text/x-python","patch_set":11,"id":"7aa08908_cf33a812","line":424,"range":{"start_line":424,"start_character":24,"end_line":424,"end_character":57},"in_reply_to":"7aa08908_7d66e051","updated":"2016-06-09 07:56:27.000000000","message":"In what way they would be easier to define ? \n\nBoth methods accepts pretty much the same argument type (send_message additionally accepts list) and in send_message there are couple places that have nothing to do with truncating - for instance I am not interested in building a key where I would have to set meta property in envelope for this method to even work or just mock _build_key.\n\nThe essence of this test is to verify truncation mechanism. So all I need is \"some\" dictionary that contains log and log contains message (a.k.a. envelope)\n\nSo setup is actually shorter. All other settings like setting configuration in self._conf or generating message would be the same, also an argument passed into send_message wouldn\u0027t change - that has to be an envelope all along.","commit_id":"06588be8973d253bcf27d3d719f894d38577cf18"},{"author":{"_account_id":16222,"name":"witek","email":"witold.bedyk@suse.com","username":"witek"},"change_message_id":"8ceef12d84dc71a8f94c2da05d48364173d705a2","unresolved":false,"context_lines":[{"line_number":421,"context_line":"        instance \u003d log_publisher.LogPublisher()"},{"line_number":422,"context_line":""},{"line_number":423,"context_line":"        envelope_copy \u003d copy.deepcopy(envelope)"},{"line_number":424,"context_line":"        json_envelope \u003d instance._truncate(envelope_copy)"},{"line_number":425,"context_line":""},{"line_number":426,"context_line":"        parsed_envelope \u003d ujson.loads(json_envelope)"},{"line_number":427,"context_line":"        parsed_log_message \u003d parsed_envelope[\u0027log\u0027][\u0027message\u0027]"}],"source_content_type":"text/x-python","patch_set":11,"id":"7aa08908_74e403ab","line":424,"range":{"start_line":424,"start_character":24,"end_line":424,"end_character":57},"in_reply_to":"7aa08908_cf33a812","updated":"2016-06-09 10:24:29.000000000","message":"You would avoid calculating the sizes. But it\u0027s OK this way as well.","commit_id":"06588be8973d253bcf27d3d719f894d38577cf18"},{"author":{"_account_id":16222,"name":"witek","email":"witold.bedyk@suse.com","username":"witek"},"change_message_id":"af7bac3bc96c3e8055e72421e13d63a21c102d25","unresolved":false,"context_lines":[{"line_number":403,"context_line":"        log_message_size \u003d (max_message_size -"},{"line_number":404,"context_line":"                            TestTruncation.EXTRA_CHARS_SIZE -"},{"line_number":405,"context_line":"                            log_publisher._KAFKA_META_DATA_SIZE -"},{"line_number":406,"context_line":"                            50)"},{"line_number":407,"context_line":"        expected_log_message_size \u003d log_message_size"},{"line_number":408,"context_line":""},{"line_number":409,"context_line":"        self._conf.config("}],"source_content_type":"text/x-python","patch_set":13,"id":"7aa08908_d15d067b","line":406,"range":{"start_line":406,"start_character":28,"end_line":406,"end_character":30},"updated":"2016-06-09 10:24:40.000000000","message":"Why not a stronger test? 1 should also do.","commit_id":"0ed1900f4bddfb41bd03b203abb585eadb34c7ef"},{"author":{"_account_id":15359,"name":"Erickson Filipe Guedes dos Santos","email":"erickson@lsd.ufcg.edu.br","username":"ericksonfilipe"},"change_message_id":"578c4d9a8d0d1a7d5b815a7fbabdcf07292da332","unresolved":false,"context_lines":[{"line_number":403,"context_line":"        log_message_size \u003d (max_message_size -"},{"line_number":404,"context_line":"                            TestTruncation.EXTRA_CHARS_SIZE -"},{"line_number":405,"context_line":"                            log_publisher._KAFKA_META_DATA_SIZE -"},{"line_number":406,"context_line":"                            50)"},{"line_number":407,"context_line":"        expected_log_message_size \u003d log_message_size"},{"line_number":408,"context_line":""},{"line_number":409,"context_line":"        self._conf.config("}],"source_content_type":"text/x-python","patch_set":13,"id":"7aa08908_df86a415","line":406,"range":{"start_line":406,"start_character":28,"end_line":406,"end_character":30},"in_reply_to":"7aa08908_d15d067b","updated":"2016-06-09 14:00:39.000000000","message":"+1","commit_id":"0ed1900f4bddfb41bd03b203abb585eadb34c7ef"},{"author":{"_account_id":16168,"name":"Tomasz Trębski","email":"kornicameister@gmail.com","username":"trebskit"},"change_message_id":"710d7ac65949f53597734ba79ca580a04ea7e1d5","unresolved":false,"context_lines":[{"line_number":403,"context_line":"        log_message_size \u003d (max_message_size -"},{"line_number":404,"context_line":"                            TestTruncation.EXTRA_CHARS_SIZE -"},{"line_number":405,"context_line":"                            log_publisher._KAFKA_META_DATA_SIZE -"},{"line_number":406,"context_line":"                            50)"},{"line_number":407,"context_line":"        expected_log_message_size \u003d log_message_size"},{"line_number":408,"context_line":""},{"line_number":409,"context_line":"        self._conf.config("}],"source_content_type":"text/x-python","patch_set":13,"id":"7aa08908_0da1eaa9","line":406,"range":{"start_line":406,"start_character":28,"end_line":406,"end_character":30},"in_reply_to":"7aa08908_df86a415","updated":"2016-06-10 11:41:03.000000000","message":"Done","commit_id":"0ed1900f4bddfb41bd03b203abb585eadb34c7ef"},{"author":{"_account_id":16222,"name":"witek","email":"witold.bedyk@suse.com","username":"witek"},"change_message_id":"af7bac3bc96c3e8055e72421e13d63a21c102d25","unresolved":false,"context_lines":[{"line_number":472,"context_line":"        log_message_size \u003d (max_message_size -"},{"line_number":473,"context_line":"                            log_publisher._KAFKA_META_DATA_SIZE -"},{"line_number":474,"context_line":"                            TestTruncation.EXTRA_CHARS_SIZE +"},{"line_number":475,"context_line":"                            10)"},{"line_number":476,"context_line":"        # adding 1 extra char because it is subtracted"},{"line_number":477,"context_line":"        # in diff calculation"},{"line_number":478,"context_line":"        truncate_by \u003d ((max_message_size -"}],"source_content_type":"text/x-python","patch_set":13,"id":"7aa08908_d1d4e6be","line":475,"range":{"start_line":475,"start_character":28,"end_line":475,"end_character":30},"updated":"2016-06-09 10:24:40.000000000","message":"see above","commit_id":"0ed1900f4bddfb41bd03b203abb585eadb34c7ef"},{"author":{"_account_id":16168,"name":"Tomasz Trębski","email":"kornicameister@gmail.com","username":"trebskit"},"change_message_id":"710d7ac65949f53597734ba79ca580a04ea7e1d5","unresolved":false,"context_lines":[{"line_number":472,"context_line":"        log_message_size \u003d (max_message_size -"},{"line_number":473,"context_line":"                            log_publisher._KAFKA_META_DATA_SIZE -"},{"line_number":474,"context_line":"                            TestTruncation.EXTRA_CHARS_SIZE +"},{"line_number":475,"context_line":"                            10)"},{"line_number":476,"context_line":"        # adding 1 extra char because it is subtracted"},{"line_number":477,"context_line":"        # in diff calculation"},{"line_number":478,"context_line":"        truncate_by \u003d ((max_message_size -"}],"source_content_type":"text/x-python","patch_set":13,"id":"7aa08908_4dabf289","line":475,"range":{"start_line":475,"start_character":28,"end_line":475,"end_character":30},"in_reply_to":"7aa08908_3f8528ad","updated":"2016-06-10 11:41:03.000000000","message":"Done","commit_id":"0ed1900f4bddfb41bd03b203abb585eadb34c7ef"},{"author":{"_account_id":15359,"name":"Erickson Filipe Guedes dos Santos","email":"erickson@lsd.ufcg.edu.br","username":"ericksonfilipe"},"change_message_id":"578c4d9a8d0d1a7d5b815a7fbabdcf07292da332","unresolved":false,"context_lines":[{"line_number":472,"context_line":"        log_message_size \u003d (max_message_size -"},{"line_number":473,"context_line":"                            log_publisher._KAFKA_META_DATA_SIZE -"},{"line_number":474,"context_line":"                            TestTruncation.EXTRA_CHARS_SIZE +"},{"line_number":475,"context_line":"                            10)"},{"line_number":476,"context_line":"        # adding 1 extra char because it is subtracted"},{"line_number":477,"context_line":"        # in diff calculation"},{"line_number":478,"context_line":"        truncate_by \u003d ((max_message_size -"}],"source_content_type":"text/x-python","patch_set":13,"id":"7aa08908_3f8528ad","line":475,"range":{"start_line":475,"start_character":28,"end_line":475,"end_character":30},"in_reply_to":"7aa08908_d1d4e6be","updated":"2016-06-09 14:00:39.000000000","message":"Agreed with Witold, test it with + 1 instead of + 10 would make this a stronger test.","commit_id":"0ed1900f4bddfb41bd03b203abb585eadb34c7ef"}]}