)]}'
{"/COMMIT_MSG":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"fd02bd4ed575ac8645de80a13502c1475eb8f8a4","unresolved":false,"context_lines":[{"line_number":18,"context_line":"    diskfile \u003d replication.fs"},{"line_number":19,"context_line":"    diskfile.devices \u003d /srv/1/node"},{"line_number":20,"context_line":"    diskfile.mount_check \u003d false"},{"line_number":21,"context_line":"    diskfile.disable_fallocate \u003d true"},{"line_number":22,"context_line":""},{"line_number":23,"context_line":"It even allows for different storage policy to use different"},{"line_number":24,"context_line":"configuration for the same diskfile (eg: keep_cache_size, reclaim_age,"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"1a1ced50_0e1e3d0c","line":21,"updated":"2017-03-17 22:44:23.000000000","message":"oh... interesting... some kind of inheritance/hierarchy would be nice\n\nI think we have some examples where different daemons/servers use different diskfile settings...\n\nbut they\u0027re probably re-used for all diskfiles on the node - it makes sense to me it might be needed to be different per policy.\n\nI think memcache.conf is one possible existing pattern to consider here.  I would like to continue to be able to define my diskfile settings in object-server.d/xxx.conf ... but with perhaps some provision to \"overlay\" a per-policy config-file/section\n\nbrainstomring:\n\n    [DEFAULT]\n    reclaim_age \u003d 604800\n    keep_cache ...\n\n    [diskfile-policy:1]\n    reclaim_age \u003d 86400  # i\u0027m really into dark data in this policy on this node\n\n\nLooking at it another way:\n\nI don\u0027t want to have diskfile settings in swift.conf\n\n1) I do checksum comparision on all nodes swift.conf - having them be different is bad and I\u0027d constantly be fighting blacklisting some vars because I do occasionally run different tweaked vars on different generations of hardware\n\n2) I don\u0027t like having an option in the config that\u0027s different from the process for very long.  If I changed a swift.conf I\u0027d want to bounce proxies - even tho the new values *shouldn\u0027t* have any effect (obviously if I change something as dramatic as \"the diskfile entrypoint for the policy\" - it\u0027s totally reasonable to bounce proxies - even if doesn\u0027t technically affect that service - but I\u0027m thinking more like twiddling some bytes/tuning sort of twiddly thing)","commit_id":"915db8631e7d8e1fa5f698aad055dfacfb76f27f"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"fd02bd4ed575ac8645de80a13502c1475eb8f8a4","unresolved":false,"context_lines":[{"line_number":24,"context_line":"configuration for the same diskfile (eg: keep_cache_size, reclaim_age,"},{"line_number":25,"context_line":"...)"},{"line_number":26,"context_line":""},{"line_number":27,"context_line":"The DiskFileRouter is removed. The DiskFileManager is now accessed through"},{"line_number":28,"context_line":"the policy instance:"},{"line_number":29,"context_line":"    df_router[policy].get_diskfile(...)"},{"line_number":30,"context_line":"become:"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"1a1ced50_0e939d5c","line":27,"updated":"2017-03-17 22:44:23.000000000","message":"good riddance!","commit_id":"915db8631e7d8e1fa5f698aad055dfacfb76f27f"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"fd02bd4ed575ac8645de80a13502c1475eb8f8a4","unresolved":false,"context_lines":[{"line_number":28,"context_line":"the policy instance:"},{"line_number":29,"context_line":"    df_router[policy].get_diskfile(...)"},{"line_number":30,"context_line":"become:"},{"line_number":31,"context_line":"    policy.diskfile_manager.get_diskfile(...)"},{"line_number":32,"context_line":""},{"line_number":33,"context_line":"FIXME: it breaks SAIO as configuration parameter \"devices\""},{"line_number":34,"context_line":"(eg: /srv/1/node) is now in swift.conf, but there can\u0027t be one swift.conf"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"1a1ced50_8ea6adbf","line":31,"updated":"2017-03-17 22:44:23.000000000","message":"I think that looks great - I always sort of hated df_router!","commit_id":"915db8631e7d8e1fa5f698aad055dfacfb76f27f"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"fd02bd4ed575ac8645de80a13502c1475eb8f8a4","unresolved":false,"context_lines":[{"line_number":32,"context_line":""},{"line_number":33,"context_line":"FIXME: it breaks SAIO as configuration parameter \"devices\""},{"line_number":34,"context_line":"(eg: /srv/1/node) is now in swift.conf, but there can\u0027t be one swift.conf"},{"line_number":35,"context_line":"per object-server instance"},{"line_number":36,"context_line":""},{"line_number":37,"context_line":"Change-Id: I070c21bc1eaf1c71ac0652cec9e813cadcc14851"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":1,"id":"1a1ced50_ee7c8122","line":35,"updated":"2017-03-17 22:44:23.000000000","message":"heh, because it\u0027s not a global option - I think the fix is to not put diskfile configuration in here","commit_id":"915db8631e7d8e1fa5f698aad055dfacfb76f27f"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"0697c1b3aab241cb991ff6a5b42be7e569495e36","unresolved":false,"context_lines":[{"line_number":14,"context_line":"    name \u003d gold"},{"line_number":15,"context_line":"    policy_type \u003d replication"},{"line_number":16,"context_line":"    default \u003d yes"},{"line_number":17,"context_line":"    diskfile \u003d egg:swift#replication.fs"},{"line_number":18,"context_line":""},{"line_number":19,"context_line":"The diskfile configuration item accepts the same format than middlewares"},{"line_number":20,"context_line":"declaration: [[scheme:]egg_name#]entry_point"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":10,"id":"5f7c97a3_5fab1e1d","line":17,"range":{"start_line":17,"start_character":4,"end_line":17,"end_character":12},"updated":"2018-06-07 16:29:05.000000000","message":"we might want to consider calling this something more abstract than \u0027diskfile\u0027. IDK, \u0027object_backend\u0027, \u0027driver\u0027??\n\nDon\u0027t change it without wider discussion.","commit_id":"fd290c33fa3328a62400bc8f2b736330440ca8be"},{"author":{"_account_id":25251,"name":"Alexandre Lécuyer","email":"alexandre.lecuyer@corp.ovh.com","username":"alecuyer"},"change_message_id":"05fd05f2ee693e574d011c2bf0acb8536398096c","unresolved":false,"context_lines":[{"line_number":14,"context_line":"    name \u003d gold"},{"line_number":15,"context_line":"    policy_type \u003d replication"},{"line_number":16,"context_line":"    default \u003d yes"},{"line_number":17,"context_line":"    diskfile \u003d egg:swift#replication.fs"},{"line_number":18,"context_line":""},{"line_number":19,"context_line":"The diskfile configuration item accepts the same format than middlewares"},{"line_number":20,"context_line":"declaration: [[scheme:]egg_name#]entry_point"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":10,"id":"5f7c97a3_28d4d84a","line":17,"range":{"start_line":17,"start_character":4,"end_line":17,"end_character":12},"in_reply_to":"5f7c97a3_5fab1e1d","updated":"2018-07-02 13:22:15.000000000","message":"I have left it as is for now but we can change it. \u0027diskfile\u0027 taken from the class names and diskfile.py, but maybe not the best name for the configuration file.","commit_id":"fd290c33fa3328a62400bc8f2b736330440ca8be"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"0697c1b3aab241cb991ff6a5b42be7e569495e36","unresolved":false,"context_lines":[{"line_number":20,"context_line":"declaration: [[scheme:]egg_name#]entry_point"},{"line_number":21,"context_line":"The egg_name is optional and default to \"swift\". The scheme is optional"},{"line_number":22,"context_line":"and default to the only valid value \"egg\". The upstream entry points are"},{"line_number":23,"context_line":"\"replication.fs\" and \"erasure_coding.fs\"."},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"The DiskFileRouter is removed. The DiskFileManager is now accessed through"},{"line_number":26,"context_line":"the policy instance:"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":10,"id":"5f7c97a3_bfeef288","line":23,"range":{"start_line":23,"start_character":13,"end_line":23,"end_character":15},"updated":"2018-06-07 16:29:05.000000000","message":"what is the significance of \u0027.fs\u0027 ?","commit_id":"fd290c33fa3328a62400bc8f2b736330440ca8be"},{"author":{"_account_id":25251,"name":"Alexandre Lécuyer","email":"alexandre.lecuyer@corp.ovh.com","username":"alecuyer"},"change_message_id":"05fd05f2ee693e574d011c2bf0acb8536398096c","unresolved":false,"context_lines":[{"line_number":20,"context_line":"declaration: [[scheme:]egg_name#]entry_point"},{"line_number":21,"context_line":"The egg_name is optional and default to \"swift\". The scheme is optional"},{"line_number":22,"context_line":"and default to the only valid value \"egg\". The upstream entry points are"},{"line_number":23,"context_line":"\"replication.fs\" and \"erasure_coding.fs\"."},{"line_number":24,"context_line":""},{"line_number":25,"context_line":"The DiskFileRouter is removed. The DiskFileManager is now accessed through"},{"line_number":26,"context_line":"the policy instance:"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":10,"id":"5f7c97a3_9f5b0454","line":23,"range":{"start_line":23,"start_character":13,"end_line":23,"end_character":15},"in_reply_to":"5f7c97a3_bfeef288","updated":"2018-07-02 13:22:15.000000000","message":"We put this for \"filesystem\", as objects are stored as files on a regular filesystem.\nShould we use \".filesystem\"?  Or another word that would indicate that the policy will use the default swift implementation?","commit_id":"fd290c33fa3328a62400bc8f2b736330440ca8be"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"54ddc22ce69fd378142cb7180a90766ead02b946","unresolved":false,"context_lines":[{"line_number":14,"context_line":"    name \u003d gold"},{"line_number":15,"context_line":"    policy_type \u003d replication"},{"line_number":16,"context_line":"    default \u003d yes"},{"line_number":17,"context_line":"    diskfile \u003d egg:swift#replication.fs"},{"line_number":18,"context_line":""},{"line_number":19,"context_line":"The diskfile configuration item accepts the same format than middlewares"},{"line_number":20,"context_line":"declaration: [[scheme:]egg_name#]entry_point"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":20,"id":"3f79a3b5_53e07bcc","line":17,"updated":"2018-09-19 18:33:28.000000000","message":"So what happens if someone does something silly like specifying an erasure-coding diskfile with policy_type\u003dreplication, or vice-versa?\n\nSeparately, is the expectation that diskfile will be configured consistently across the cluster, or would some nodes use use diskfile \"classic\" and others the LOSF diskfile? Also, does this affect any services besides object-server/replicator/reconstructor?\n\nI guess I\u0027m wondering if this setting would be better in swift.conf (where I typically deploy the exact same file to all nodes) or object-server.conf (where it makes sense that there may be per-node differences).","commit_id":"673fda76203be7c70afe6236d3fadc88deb75117"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"06d486987322bd15d435dfa8422c1ff22f8c6379","unresolved":false,"context_lines":[{"line_number":14,"context_line":"    name \u003d gold"},{"line_number":15,"context_line":"    policy_type \u003d replication"},{"line_number":16,"context_line":"    default \u003d yes"},{"line_number":17,"context_line":"    diskfile \u003d egg:swift#replication.fs"},{"line_number":18,"context_line":""},{"line_number":19,"context_line":"The diskfile configuration item accepts the same format than middlewares"},{"line_number":20,"context_line":"declaration: [[scheme:]egg_name#]entry_point"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":20,"id":"3f79a3b5_22f65292","line":17,"range":{"start_line":17,"start_character":4,"end_line":17,"end_character":12},"updated":"2018-09-26 05:55:17.000000000","message":"This should be diskfile_module\n\nFound it out the hard way when creating my own arbitrary different matt.fs. And pasted the value from here :)","commit_id":"673fda76203be7c70afe6236d3fadc88deb75117"},{"author":{"_account_id":25251,"name":"Alexandre Lécuyer","email":"alexandre.lecuyer@corp.ovh.com","username":"alecuyer"},"change_message_id":"4afd187e4d44dba0714436bb227b35ff2d37142e","unresolved":false,"context_lines":[{"line_number":14,"context_line":"    name \u003d gold"},{"line_number":15,"context_line":"    policy_type \u003d replication"},{"line_number":16,"context_line":"    default \u003d yes"},{"line_number":17,"context_line":"    diskfile \u003d egg:swift#replication.fs"},{"line_number":18,"context_line":""},{"line_number":19,"context_line":"The diskfile configuration item accepts the same format than middlewares"},{"line_number":20,"context_line":"declaration: [[scheme:]egg_name#]entry_point"}],"source_content_type":"text/x-gerrit-commit-message","patch_set":20,"id":"3f79a3b5_3251b531","line":17,"range":{"start_line":17,"start_character":4,"end_line":17,"end_character":12},"in_reply_to":"3f79a3b5_22f65292","updated":"2018-09-26 14:27:34.000000000","message":"Ok we will update the commit message once the \u0027replication.XX\u0027 choice is made","commit_id":"673fda76203be7c70afe6236d3fadc88deb75117"}],"doc/source/overview_policies.rst":[{"author":{"_account_id":14766,"name":"Kazuhiro MIYAHARA","email":"miyahara.kazuhiro@lab.ntt.co.jp","username":"miyahara"},"change_message_id":"27d50d17d1d83af01404bfc354021c8cdb429f46","unresolved":false,"context_lines":[{"line_number":325,"context_line":"    - The default value is ``replication``."},{"line_number":326,"context_line":"    - When defining an EC policy use the value ``erasure_coding``."},{"line_number":327,"context_line":"* ``diskfile \u003d \u003centry point\u003e`` (optional)"},{"line_number":328,"context_line":"    - The option ``diskfile`` is used to load an alternate diskfile"},{"line_number":329,"context_line":"      implementation. The entry_point must exist in entry_points.txt in"},{"line_number":330,"context_line":"      the swift.diskfile section."},{"line_number":331,"context_line":"    - The default value is ``replication.fs`` or ``erasure_coding.fs``"},{"line_number":332,"context_line":"      depending on the policy type."}],"source_content_type":"text/x-rst","patch_set":16,"id":"5f7c97a3_6640dc60","line":329,"range":{"start_line":328,"start_character":59,"end_line":329,"end_character":21},"updated":"2018-07-25 04:34:10.000000000","message":"nit:\nBecause we know diskfile class in swift implementation, we can understand what \"diskfile implementation\" means.\nBut this document can be referred by non-contributors.\nSo I feel \"backend object storage plug-in architeture\" is better as same as line 46 of this document.","commit_id":"30db6a95721e18c40027d199e5c6b1f1a5c84929"},{"author":{"_account_id":13852,"name":"Romain LE DISEZ","email":"romain.le-disez@corp.ovh.com","username":"rledisez"},"change_message_id":"e406534231295980f4eb6daba9e6a7f307a70d3a","unresolved":false,"context_lines":[{"line_number":325,"context_line":"    - The default value is ``replication``."},{"line_number":326,"context_line":"    - When defining an EC policy use the value ``erasure_coding``."},{"line_number":327,"context_line":"* ``diskfile \u003d \u003centry point\u003e`` (optional)"},{"line_number":328,"context_line":"    - The option ``diskfile`` is used to load an alternate diskfile"},{"line_number":329,"context_line":"      implementation. The entry_point must exist in entry_points.txt in"},{"line_number":330,"context_line":"      the swift.diskfile section."},{"line_number":331,"context_line":"    - The default value is ``replication.fs`` or ``erasure_coding.fs``"},{"line_number":332,"context_line":"      depending on the policy type."}],"source_content_type":"text/x-rst","patch_set":16,"id":"5f7c97a3_5ac6cc4c","line":329,"range":{"start_line":328,"start_character":59,"end_line":329,"end_character":21},"in_reply_to":"5f7c97a3_6640dc60","updated":"2018-07-25 10:07:25.000000000","message":"Done","commit_id":"30db6a95721e18c40027d199e5c6b1f1a5c84929"},{"author":{"_account_id":14766,"name":"Kazuhiro MIYAHARA","email":"miyahara.kazuhiro@lab.ntt.co.jp","username":"miyahara"},"change_message_id":"27d50d17d1d83af01404bfc354021c8cdb429f46","unresolved":false,"context_lines":[{"line_number":328,"context_line":"    - The option ``diskfile`` is used to load an alternate diskfile"},{"line_number":329,"context_line":"      implementation. The entry_point must exist in entry_points.txt in"},{"line_number":330,"context_line":"      the swift.diskfile section."},{"line_number":331,"context_line":"    - The default value is ``replication.fs`` or ``erasure_coding.fs``"},{"line_number":332,"context_line":"      depending on the policy type."},{"line_number":333,"context_line":""},{"line_number":334,"context_line":"The EC policy type has additional required options. See"}],"source_content_type":"text/x-rst","patch_set":16,"id":"5f7c97a3_51f9dc4c","line":331,"range":{"start_line":331,"start_character":27,"end_line":331,"end_character":70},"updated":"2018-07-25 04:34:10.000000000","message":"The default value is ``egg:swift#replication.fs`` or ``egg:swift#erasure_coding.fs``, isn\u0027t it?\nWe can omit scheme and egg name in configuratin files, but I think explanation should not omit them.","commit_id":"30db6a95721e18c40027d199e5c6b1f1a5c84929"},{"author":{"_account_id":13852,"name":"Romain LE DISEZ","email":"romain.le-disez@corp.ovh.com","username":"rledisez"},"change_message_id":"e406534231295980f4eb6daba9e6a7f307a70d3a","unresolved":false,"context_lines":[{"line_number":328,"context_line":"    - The option ``diskfile`` is used to load an alternate diskfile"},{"line_number":329,"context_line":"      implementation. The entry_point must exist in entry_points.txt in"},{"line_number":330,"context_line":"      the swift.diskfile section."},{"line_number":331,"context_line":"    - The default value is ``replication.fs`` or ``erasure_coding.fs``"},{"line_number":332,"context_line":"      depending on the policy type."},{"line_number":333,"context_line":""},{"line_number":334,"context_line":"The EC policy type has additional required options. See"}],"source_content_type":"text/x-rst","patch_set":16,"id":"5f7c97a3_1ac0542b","line":331,"range":{"start_line":331,"start_character":27,"end_line":331,"end_character":70},"in_reply_to":"5f7c97a3_51f9dc4c","updated":"2018-07-25 10:07:25.000000000","message":"Done","commit_id":"30db6a95721e18c40027d199e5c6b1f1a5c84929"},{"author":{"_account_id":7233,"name":"Matthew Oliver","email":"matt@oliver.net.au","username":"mattoliverau"},"change_message_id":"06d486987322bd15d435dfa8422c1ff22f8c6379","unresolved":false,"context_lines":[{"line_number":353,"context_line":"        [storage-policy:1]"},{"line_number":354,"context_line":"        name \u003d silver"},{"line_number":355,"context_line":"        policy_type \u003d replication"},{"line_number":356,"context_line":"        diskfile_module \u003d replication.fs"},{"line_number":357,"context_line":"        deprecated \u003d yes"},{"line_number":358,"context_line":""},{"line_number":359,"context_line":""}],"source_content_type":"text/x-rst","patch_set":20,"id":"3f79a3b5_820de690","line":356,"updated":"2018-09-26 05:55:17.000000000","message":"NIT: not 100% sure about the ending in `.fs`. I guess in the case of LoSFs it\u0027s pretty much a different filesystem. I would have thought maybe replication.df would be better.. but meh. I can live with it.","commit_id":"673fda76203be7c70afe6236d3fadc88deb75117"},{"author":{"_account_id":15343,"name":"Tim Burke","email":"tburke@nvidia.com","username":"tburke"},"change_message_id":"3bfe37c1860d594838601a1b001ee0de402f1524","unresolved":false,"context_lines":[{"line_number":353,"context_line":"        [storage-policy:1]"},{"line_number":354,"context_line":"        name \u003d silver"},{"line_number":355,"context_line":"        policy_type \u003d replication"},{"line_number":356,"context_line":"        diskfile_module \u003d replication.fs"},{"line_number":357,"context_line":"        deprecated \u003d yes"},{"line_number":358,"context_line":""},{"line_number":359,"context_line":""}],"source_content_type":"text/x-rst","patch_set":20,"id":"3f79a3b5_425b3c85","line":356,"in_reply_to":"3f79a3b5_127d99b1","updated":"2018-09-26 19:37:59.000000000","message":"It starts to make sense when you think about what the *new* options will end up being: presumably, something like replication.losf and erasure_coding.losf. (Will lots-of-small-files still require two modules?)\n\nI guess I maybe lean toward .xfs, just because we have so many assumptions that the filesystem used will be xfs...\n\nI think I\u0027d be opposed to .df; diskfile is the abstraction. Any alternate diskfiles would be justified in wanting to use .df as well, at which point why have the suffix at all?","commit_id":"673fda76203be7c70afe6236d3fadc88deb75117"},{"author":{"_account_id":25251,"name":"Alexandre Lécuyer","email":"alexandre.lecuyer@corp.ovh.com","username":"alecuyer"},"change_message_id":"4afd187e4d44dba0714436bb227b35ff2d37142e","unresolved":false,"context_lines":[{"line_number":353,"context_line":"        [storage-policy:1]"},{"line_number":354,"context_line":"        name \u003d silver"},{"line_number":355,"context_line":"        policy_type \u003d replication"},{"line_number":356,"context_line":"        diskfile_module \u003d replication.fs"},{"line_number":357,"context_line":"        deprecated \u003d yes"},{"line_number":358,"context_line":""},{"line_number":359,"context_line":""}],"source_content_type":"text/x-rst","patch_set":20,"id":"3f79a3b5_127d99b1","line":356,"in_reply_to":"3f79a3b5_820de690","updated":"2018-09-26 14:27:34.000000000","message":"I think you\u0027re the third person to mention that \u0027.fs\u0027 is not a great choice, so we probably want to change it.\nWhat would you all think is best? (\u0027df\u0027, \u0027xfs\u0027..).","commit_id":"673fda76203be7c70afe6236d3fadc88deb75117"}],"etc/swift.conf-sample":[{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"0697c1b3aab241cb991ff6a5b42be7e569495e36","unresolved":false,"context_lines":[{"line_number":30,"context_line":"name \u003d Policy-0"},{"line_number":31,"context_line":"default \u003d yes"},{"line_number":32,"context_line":"#policy_type \u003d replication"},{"line_number":33,"context_line":"#diskfile \u003d egg:swift#replication.fs"},{"line_number":34,"context_line":"aliases \u003d yellow, orange"},{"line_number":35,"context_line":""},{"line_number":36,"context_line":"# the following section would declare a policy called \u0027silver\u0027, the number of"}],"source_content_type":"application/octet-stream","patch_set":10,"id":"5f7c97a3_d54b8f5c","line":33,"range":{"start_line":33,"start_character":0,"end_line":33,"end_character":36},"updated":"2018-06-07 16:29:05.000000000","message":"some additional comment above would be useful to explain this option and how it defaults.\n\nAlso, overview_policies.rst \u0027Configuring Policies\u0027 section will need to be updated.","commit_id":"fd290c33fa3328a62400bc8f2b736330440ca8be"},{"author":{"_account_id":25251,"name":"Alexandre Lécuyer","email":"alexandre.lecuyer@corp.ovh.com","username":"alecuyer"},"change_message_id":"05fd05f2ee693e574d011c2bf0acb8536398096c","unresolved":false,"context_lines":[{"line_number":30,"context_line":"name \u003d Policy-0"},{"line_number":31,"context_line":"default \u003d yes"},{"line_number":32,"context_line":"#policy_type \u003d replication"},{"line_number":33,"context_line":"#diskfile \u003d egg:swift#replication.fs"},{"line_number":34,"context_line":"aliases \u003d yellow, orange"},{"line_number":35,"context_line":""},{"line_number":36,"context_line":"# the following section would declare a policy called \u0027silver\u0027, the number of"}],"source_content_type":"application/octet-stream","patch_set":10,"id":"5f7c97a3_0d248e0d","line":33,"range":{"start_line":33,"start_character":0,"end_line":33,"end_character":36},"in_reply_to":"5f7c97a3_d54b8f5c","updated":"2018-07-02 13:22:15.000000000","message":"I have added comments there and in overview_policies.rst","commit_id":"fd290c33fa3328a62400bc8f2b736330440ca8be"},{"author":{"_account_id":14766,"name":"Kazuhiro MIYAHARA","email":"miyahara.kazuhiro@lab.ntt.co.jp","username":"miyahara"},"change_message_id":"27d50d17d1d83af01404bfc354021c8cdb429f46","unresolved":false,"context_lines":[{"line_number":26,"context_line":"#"},{"line_number":27,"context_line":"# A \u0027policy_type\u0027 argument is also supported but is not mandatory.  Default"},{"line_number":28,"context_line":"# policy type \u0027replication\u0027 is used when \u0027policy_type\u0027 is unspecified."},{"line_number":29,"context_line":"# A \u0027diskfile\u0027 optional argument lets you specify an alternate diskfile"},{"line_number":30,"context_line":"# implementation. The default is \"replication.fs\", or \"erasure_coding.fs\","},{"line_number":31,"context_line":"# depending on the policy type."},{"line_number":32,"context_line":"[storage-policy:0]"},{"line_number":33,"context_line":"name \u003d Policy-0"}],"source_content_type":"application/octet-stream","patch_set":16,"id":"5f7c97a3_b1b2b0e7","line":30,"range":{"start_line":29,"start_character":63,"end_line":30,"end_character":17},"updated":"2018-07-25 04:34:10.000000000","message":"nit: Same as \"doc/source/overview_policies.rst\"","commit_id":"30db6a95721e18c40027d199e5c6b1f1a5c84929"},{"author":{"_account_id":13852,"name":"Romain LE DISEZ","email":"romain.le-disez@corp.ovh.com","username":"rledisez"},"change_message_id":"e406534231295980f4eb6daba9e6a7f307a70d3a","unresolved":false,"context_lines":[{"line_number":26,"context_line":"#"},{"line_number":27,"context_line":"# A \u0027policy_type\u0027 argument is also supported but is not mandatory.  Default"},{"line_number":28,"context_line":"# policy type \u0027replication\u0027 is used when \u0027policy_type\u0027 is unspecified."},{"line_number":29,"context_line":"# A \u0027diskfile\u0027 optional argument lets you specify an alternate diskfile"},{"line_number":30,"context_line":"# implementation. The default is \"replication.fs\", or \"erasure_coding.fs\","},{"line_number":31,"context_line":"# depending on the policy type."},{"line_number":32,"context_line":"[storage-policy:0]"},{"line_number":33,"context_line":"name \u003d Policy-0"}],"source_content_type":"application/octet-stream","patch_set":16,"id":"5f7c97a3_3abbd0be","line":30,"range":{"start_line":29,"start_character":63,"end_line":30,"end_character":17},"in_reply_to":"5f7c97a3_b1b2b0e7","updated":"2018-07-25 10:07:25.000000000","message":"Done","commit_id":"30db6a95721e18c40027d199e5c6b1f1a5c84929"},{"author":{"_account_id":4608,"name":"Kota Tsuyuzaki","email":"bloodeagle40234@gmail.com","username":"tsuyuzaki-kota"},"change_message_id":"fccd279bd6434de098642918a3ff5f695dabd24d","unresolved":false,"context_lines":[{"line_number":33,"context_line":"name \u003d Policy-0"},{"line_number":34,"context_line":"default \u003d yes"},{"line_number":35,"context_line":"#policy_type \u003d replication"},{"line_number":36,"context_line":"#diskfile \u003d replication.fs"},{"line_number":37,"context_line":"aliases \u003d yellow, orange"},{"line_number":38,"context_line":""},{"line_number":39,"context_line":"# the following section would declare a policy called \u0027silver\u0027, the number of"}],"source_content_type":"application/octet-stream","patch_set":16,"id":"5f7c97a3_18fc00b5","line":36,"updated":"2018-07-24 08:26:53.000000000","message":"The commit msessage says as \"diskfile \u003d egg:swift#replication.fs\" so IMO, we should have the explanation for the option format like as \n\n [[scheme:]package#]replication.fs","commit_id":"30db6a95721e18c40027d199e5c6b1f1a5c84929"},{"author":{"_account_id":13852,"name":"Romain LE DISEZ","email":"romain.le-disez@corp.ovh.com","username":"rledisez"},"change_message_id":"e406534231295980f4eb6daba9e6a7f307a70d3a","unresolved":false,"context_lines":[{"line_number":33,"context_line":"name \u003d Policy-0"},{"line_number":34,"context_line":"default \u003d yes"},{"line_number":35,"context_line":"#policy_type \u003d replication"},{"line_number":36,"context_line":"#diskfile \u003d replication.fs"},{"line_number":37,"context_line":"aliases \u003d yellow, orange"},{"line_number":38,"context_line":""},{"line_number":39,"context_line":"# the following section would declare a policy called \u0027silver\u0027, the number of"}],"source_content_type":"application/octet-stream","patch_set":16,"id":"5f7c97a3_dad73cf4","line":36,"in_reply_to":"5f7c97a3_18fc00b5","updated":"2018-07-25 10:07:25.000000000","message":"Done","commit_id":"30db6a95721e18c40027d199e5c6b1f1a5c84929"},{"author":{"_account_id":4608,"name":"Kota Tsuyuzaki","email":"bloodeagle40234@gmail.com","username":"tsuyuzaki-kota"},"change_message_id":"fccd279bd6434de098642918a3ff5f695dabd24d","unresolved":false,"context_lines":[{"line_number":93,"context_line":"# Duplicated EC fragments is proof-of-concept experimental support to enable"},{"line_number":94,"context_line":"# Global Erasure Coding policies with multiple regions acting as independent"},{"line_number":95,"context_line":"# failure domains.  Do not change the default except in development/testing."},{"line_number":96,"context_line":"#ec_duplication_factor \u003d 1"},{"line_number":97,"context_line":""},{"line_number":98,"context_line":"# The swift-constraints section sets the basic constraints on data"},{"line_number":99,"context_line":"# saved in the swift cluster. These constraints are automatically"}],"source_content_type":"application/octet-stream","patch_set":16,"id":"5f7c97a3_d821a8a3","line":96,"updated":"2018-07-24 08:26:53.000000000","message":"if \"replication.fs\" is the default for diskfile, should we add \"erasure_coding.fs\" around here?\n\nAh... it\u0027s wrong. it depends policy-type so that ec policy uses erasure_coding.fs by default.","commit_id":"30db6a95721e18c40027d199e5c6b1f1a5c84929"},{"author":{"_account_id":13852,"name":"Romain LE DISEZ","email":"romain.le-disez@corp.ovh.com","username":"rledisez"},"change_message_id":"e406534231295980f4eb6daba9e6a7f307a70d3a","unresolved":false,"context_lines":[{"line_number":93,"context_line":"# Duplicated EC fragments is proof-of-concept experimental support to enable"},{"line_number":94,"context_line":"# Global Erasure Coding policies with multiple regions acting as independent"},{"line_number":95,"context_line":"# failure domains.  Do not change the default except in development/testing."},{"line_number":96,"context_line":"#ec_duplication_factor \u003d 1"},{"line_number":97,"context_line":""},{"line_number":98,"context_line":"# The swift-constraints section sets the basic constraints on data"},{"line_number":99,"context_line":"# saved in the swift cluster. These constraints are automatically"}],"source_content_type":"application/octet-stream","patch_set":16,"id":"5f7c97a3_fada381a","line":96,"in_reply_to":"5f7c97a3_d821a8a3","updated":"2018-07-25 10:07:25.000000000","message":"Done","commit_id":"30db6a95721e18c40027d199e5c6b1f1a5c84929"}],"setup.cfg":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"fd02bd4ed575ac8645de80a13502c1475eb8f8a4","unresolved":false,"context_lines":[{"line_number":102,"context_line":""},{"line_number":103,"context_line":"swift.diskfile \u003d"},{"line_number":104,"context_line":"    replication.fs \u003d swift.obj.diskfile:DiskFileManager"},{"line_number":105,"context_line":"    erasure_coding.fs \u003d swift.obj.diskfile:ECDiskFileManager"},{"line_number":106,"context_line":""},{"line_number":107,"context_line":"[build_sphinx]"},{"line_number":108,"context_line":"all_files \u003d 1"}],"source_content_type":"text/x-ttcn-cfg","patch_set":1,"id":"1a1ced50_6effb101","line":105,"updated":"2017-03-17 22:44:23.000000000","message":"what does \u0027fs\u0027 mean here?","commit_id":"915db8631e7d8e1fa5f698aad055dfacfb76f27f"}],"swift/cli/relinker.py":[{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"0697c1b3aab241cb991ff6a5b42be7e569495e36","unresolved":false,"context_lines":[{"line_number":96,"context_line":"            # has been increased, but cleanup did not yet run)"},{"line_number":97,"context_line":"            loc \u003d diskfile.AuditLocation("},{"line_number":98,"context_line":"                os.path.dirname(expected_fname), device, partition, policy)"},{"line_number":99,"context_line":"            policy.load_diskfile_manager(conf, logger)"},{"line_number":100,"context_line":"            diskfile_mgr \u003d policy.diskfile_manager"},{"line_number":101,"context_line":"            df \u003d diskfile_mgr.get_diskfile_from_audit_location(loc)"},{"line_number":102,"context_line":"            try:"}],"source_content_type":"text/x-python","patch_set":10,"id":"5f7c97a3_6a4e3ac2","line":99,"range":{"start_line":99,"start_character":47,"end_line":99,"end_character":53},"updated":"2018-06-07 16:29:05.000000000","message":"before the DFM got its own logger, now it is passed the relinker\u0027s logger. This causes the test failures in test_relinker.py","commit_id":"fd290c33fa3328a62400bc8f2b736330440ca8be"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"0697c1b3aab241cb991ff6a5b42be7e569495e36","unresolved":false,"context_lines":[{"line_number":96,"context_line":"            # has been increased, but cleanup did not yet run)"},{"line_number":97,"context_line":"            loc \u003d diskfile.AuditLocation("},{"line_number":98,"context_line":"                os.path.dirname(expected_fname), device, partition, policy)"},{"line_number":99,"context_line":"            policy.load_diskfile_manager(conf, logger)"},{"line_number":100,"context_line":"            diskfile_mgr \u003d policy.diskfile_manager"},{"line_number":101,"context_line":"            df \u003d diskfile_mgr.get_diskfile_from_audit_location(loc)"},{"line_number":102,"context_line":"            try:"},{"line_number":103,"context_line":"                with df.open():"}],"source_content_type":"text/x-python","patch_set":10,"id":"5f7c97a3_5a3a4ccd","line":100,"range":{"start_line":99,"start_character":12,"end_line":100,"end_character":50},"updated":"2018-06-07 16:29:05.000000000","message":"In light of my comments in storage_policy.py, here I\u0027d suggest perhaps just:\n\n  diskfile_mgr \u003d policy.get_diskfile_manager(conf, logger)\n\nIn this use case it\u0027s not obvious why policy needs to cache the diskfile_manager instance.","commit_id":"fd290c33fa3328a62400bc8f2b736330440ca8be"}],"swift/common/base_storage_server.py":[{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"0697c1b3aab241cb991ff6a5b42be7e569495e36","unresolved":false,"context_lines":[{"line_number":25,"context_line":"    \"\"\""},{"line_number":26,"context_line":""},{"line_number":27,"context_line":"    def __init__(self, conf, **kwargs):"},{"line_number":28,"context_line":"        self.conf \u003d conf"},{"line_number":29,"context_line":"        self._allowed_methods \u003d None"},{"line_number":30,"context_line":"        replication_server \u003d conf.get(\u0027replication_server\u0027, None)"},{"line_number":31,"context_line":"        if replication_server is not None:"}],"source_content_type":"text/x-python","patch_set":10,"id":"5f7c97a3_678c3a35","line":28,"range":{"start_line":28,"start_character":8,"end_line":28,"end_character":24},"updated":"2018-06-07 16:29:05.000000000","message":"only used in tests?","commit_id":"fd290c33fa3328a62400bc8f2b736330440ca8be"}],"swift/common/storage_policy.py":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"fd02bd4ed575ac8645de80a13502c1475eb8f8a4","unresolved":false,"context_lines":[{"line_number":163,"context_line":""},{"line_number":164,"context_line":"    def __init__(self, idx, name\u003d\u0027\u0027, is_default\u003dFalse, is_deprecated\u003dFalse,"},{"line_number":165,"context_line":"                 object_ring\u003dNone, aliases\u003d\u0027\u0027,"},{"line_number":166,"context_line":"                 diskfile\u003d\u0027replication.fs\u0027, diskfile_config\u003d{}):"},{"line_number":167,"context_line":"        # do not allow BaseStoragePolicy class to be instantiated directly"},{"line_number":168,"context_line":"        if type(self) \u003d\u003d BaseStoragePolicy:"},{"line_number":169,"context_line":"            raise TypeError(\"Can\u0027t instantiate BaseStoragePolicy directly\")"}],"source_content_type":"text/x-python","patch_set":1,"id":"1a1ced50_339442e1","line":166,"updated":"2017-03-17 22:44:23.000000000","message":"I think diskfile_entrypoint is the only \"required\" new kwarg here - the defaults are good.  I don\u0027t like having this thing hold a reference to the diskfile_config.  I would like in testing if I could pass in a FakeDiskFileManager via diskfile_manager kwarg (default to None)","commit_id":"915db8631e7d8e1fa5f698aad055dfacfb76f27f"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"fd02bd4ed575ac8645de80a13502c1475eb8f8a4","unresolved":false,"context_lines":[{"line_number":195,"context_line":""},{"line_number":196,"context_line":"        self._diskfile_name \u003d diskfile"},{"line_number":197,"context_line":"        self._diskfile_config \u003d diskfile_config"},{"line_number":198,"context_line":"        self._diskfile_manager \u003d None"},{"line_number":199,"context_line":""},{"line_number":200,"context_line":"    @property"},{"line_number":201,"context_line":"    def name(self):"}],"source_content_type":"text/x-python","patch_set":1,"id":"1a1ced50_93c176e7","line":198,"updated":"2017-03-17 22:44:23.000000000","message":"I think we should follow the pattern of the object_ring\n\nA swift process certainly should be able to access the policies class w/o loading the object_ring or creating an instance of the dfm\n\nIf you want the ring you:\n\n    load_ring(*params_needed_to_load_ring)\n\nIf you want the dfm:\n\n    load_diskfile_manager(*params_needed_to_load_dfm)","commit_id":"915db8631e7d8e1fa5f698aad055dfacfb76f27f"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"fd02bd4ed575ac8645de80a13502c1475eb8f8a4","unresolved":false,"context_lines":[{"line_number":213,"context_line":"    @property"},{"line_number":214,"context_line":"    def diskfile_manager(self):"},{"line_number":215,"context_line":"        # Dynamic property to avoid import loop between common.storage_policy"},{"line_number":216,"context_line":"        # and diskfile"},{"line_number":217,"context_line":"        if not self._diskfile_manager:"},{"line_number":218,"context_line":"            # FIXME: get a real logger?"},{"line_number":219,"context_line":"            dfm_logger \u003d logging.getLogger(\"swift.obj.diskfile_manager\")"}],"source_content_type":"text/x-python","patch_set":1,"id":"1a1ced50_ae7c4973","line":216,"updated":"2017-03-17 22:44:23.000000000","message":"I don\u0027t love this as a dynamic lazy initialization here - I would prefer an explicit initialization.  If you access a policy.diskfile_manager w/o first initializing it should be an error.  This would be consistent with the object_ring initialization - which has so far worked well.","commit_id":"915db8631e7d8e1fa5f698aad055dfacfb76f27f"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"fd02bd4ed575ac8645de80a13502c1475eb8f8a4","unresolved":false,"context_lines":[{"line_number":216,"context_line":"        # and diskfile"},{"line_number":217,"context_line":"        if not self._diskfile_manager:"},{"line_number":218,"context_line":"            # FIXME: get a real logger?"},{"line_number":219,"context_line":"            dfm_logger \u003d logging.getLogger(\"swift.obj.diskfile_manager\")"},{"line_number":220,"context_line":"            if not dfm_logger.handlers:"},{"line_number":221,"context_line":"                # If nothing else, log to stderr"},{"line_number":222,"context_line":"                dfm_logger.addHandler(logging.StreamHandler(sys.__stderr__))"}],"source_content_type":"text/x-python","patch_set":1,"id":"1a1ced50_ce81854c","line":219,"updated":"2017-03-17 22:44:23.000000000","message":"I think you hunt down where processes are currently calling policy.load_ring you will have all the args (and logger) right there to call policy.load_diskfile_manager","commit_id":"915db8631e7d8e1fa5f698aad055dfacfb76f27f"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"fd02bd4ed575ac8645de80a13502c1475eb8f8a4","unresolved":false,"context_lines":[{"line_number":221,"context_line":"                # If nothing else, log to stderr"},{"line_number":222,"context_line":"                dfm_logger.addHandler(logging.StreamHandler(sys.__stderr__))"},{"line_number":223,"context_line":"            dfm_cls \u003d pkg_resources.load_entry_point(\u0027swift\u0027, \u0027swift.diskfile\u0027,"},{"line_number":224,"context_line":"                                                     self._diskfile_name)"},{"line_number":225,"context_line":"            self._diskfile_manager \u003d dfm_cls(self._diskfile_config, dfm_logger)"},{"line_number":226,"context_line":"        return self._diskfile_manager"},{"line_number":227,"context_line":""}],"source_content_type":"text/x-python","patch_set":1,"id":"1a1ced50_8e878d60","line":224,"updated":"2017-03-17 22:44:23.000000000","message":"Why not let the configured value also define a diskfile entrypoint from another module?\n\n    swift#replication.fs\n\nor\n\n    ovh#replication.fs\n\nI looked at doing something similar a while ago for some other reasons; but it didn\u0027t end up being entirely needed:\n\nhttps://docs.google.com/presentation/d/1xzp2RYPAYGRsdj4zanptune-UO2kM6Lu-Oar--slYdg/edit?usp\u003dsharing\n\n^ it\u0027s better when I can tell my story while presenting!","commit_id":"915db8631e7d8e1fa5f698aad055dfacfb76f27f"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"fd02bd4ed575ac8645de80a13502c1475eb8f8a4","unresolved":false,"context_lines":[{"line_number":387,"context_line":"        \"\"\""},{"line_number":388,"context_line":"        if self.object_ring:"},{"line_number":389,"context_line":"            return"},{"line_number":390,"context_line":"        self.object_ring \u003d Ring(swift_dir, ring_name\u003dself.ring_name)"},{"line_number":391,"context_line":""},{"line_number":392,"context_line":"    @property"},{"line_number":393,"context_line":"    def quorum(self):"}],"source_content_type":"text/x-python","patch_set":1,"id":"1a1ced50_33828295","line":390,"updated":"2017-03-17 22:44:23.000000000","message":"here is load_ring","commit_id":"915db8631e7d8e1fa5f698aad055dfacfb76f27f"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"004d5f4d42e74ad52b959a21aa289c6e79740340","unresolved":false,"context_lines":[{"line_number":375,"context_line":"        if self.diskfile_manager:"},{"line_number":376,"context_line":"            return"},{"line_number":377,"context_line":"        for ep in pkg_resources.iter_entry_points(\u0027swift.diskfile\u0027,"},{"line_number":378,"context_line":"                                                  self.diskfile_name):"},{"line_number":379,"context_line":"            dfm_cls \u003d ep.load()"},{"line_number":380,"context_line":"            if self.policy_type !\u003d dfm_cls.policy_type:"},{"line_number":381,"context_line":"                logger.warning(\u0027Invalid policy_type on diskfile %s for policy \u0027"}],"source_content_type":"text/x-python","patch_set":2,"id":"1a1ced50_1122ab00","line":378,"updated":"2017-03-20 18:24:28.000000000","message":"So I was acctually suggesting to continue to use pkg_resources.load_entry_point - but just not requiring \u0027swift\u0027 to get a hard coded magic string in the code - and instead just parse it out of the config option:\n\ni.e.\n\n    use_diskfile \u003d egg:swift#replicated.xfs.dir\n\nAnd then just parse it same as paste.deploy does:\n\nhttps://bitbucket.org/ianb/pastedeploy/src/4b27133a2a7db58b213ae55b580039c11d2055c0/paste/deploy/loadwsgi.py?at\u003ddefault\u0026fileviewer\u003dfile-view-default#loadwsgi.py-275\n\nIt\u0027d have some internal consistency with how we load wsgi apps today that I think would be helpful if we ever end up taking over maintenance of all that mess too.\n\nThe exact format of the config option is up for debate - I\u0027m sure we could code it up where the \u0027egg\u0027 is required/assumed and the \u0027swift\u0027 part is the default unless otherwise specified.\n\ne.g.\n\n    use_diskfile \u003d replicated.whatever\n\nis the same as\n\n    use_diskfile \u003d swift#replicated.whatever\n\nis the same as\n\n    use_diskfile \u003d egg:swift#replicated.whatever\n\nbut is very different than\n\n    use_diskfile \u003d egg:third.party#replicatd.whatever","commit_id":"ab0f7b7c645e920b4adc45f41a178a63c47a4dd0"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"004d5f4d42e74ad52b959a21aa289c6e79740340","unresolved":false,"context_lines":[{"line_number":827,"context_line":"            raise PolicyError(\"No policy with index %s\" % policy_idx)"},{"line_number":828,"context_line":"        if not policy.object_ring:"},{"line_number":829,"context_line":"            policy.load_ring(swift_dir)"},{"line_number":830,"context_line":"        return policy.object_ring"},{"line_number":831,"context_line":""},{"line_number":832,"context_line":"    def get_diskfile_manager(self, policy_idx, conf, logger):"},{"line_number":833,"context_line":"        \"\"\""}],"source_content_type":"text/x-python","patch_set":2,"id":"1a1ced50_71783f90","line":830,"updated":"2017-03-20 18:24:28.000000000","message":"I had forgotten this helper existed.  Perhaps we should strive for less schizophrenia.  The sloppiness matters more with rings, but I think it\u0027s reasonable to apply similar discipline DFM\u0027s (some DFM\u0027s might have a connection pool or resource limits)\n\nWe want them to be loaded explicitly exactly once on process startup on only the processes that need them - and we want to be able to find in the code when that happens.\n\nFrom that point on we want them to be accessible as an instance on the policy global.","commit_id":"ab0f7b7c645e920b4adc45f41a178a63c47a4dd0"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"004d5f4d42e74ad52b959a21aa289c6e79740340","unresolved":false,"context_lines":[{"line_number":838,"context_line":"        :param policy_idx: policy index as defined in swift.conf"},{"line_number":839,"context_line":"        :param swift_dir: swift_dir used by the caller"},{"line_number":840,"context_line":"        :returns: appropriate ring object"},{"line_number":841,"context_line":"        \"\"\""},{"line_number":842,"context_line":"        policy \u003d self.get_by_index(policy_idx)"},{"line_number":843,"context_line":"        if not policy:"},{"line_number":844,"context_line":"            raise PolicyError(\"No policy with index %s\" % policy_idx)"}],"source_content_type":"text/x-python","patch_set":2,"id":"1a1ced50_8ea03a45","line":841,"updated":"2017-03-20 18:24:28.000000000","message":"this helper seems similar to the reconstructor/replicator\u0027s load_diskfile_manager helper","commit_id":"ab0f7b7c645e920b4adc45f41a178a63c47a4dd0"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"0697c1b3aab241cb991ff6a5b42be7e569495e36","unresolved":false,"context_lines":[{"line_number":289,"context_line":"                info.pop(\u0027default\u0027)"},{"line_number":290,"context_line":"            if not self.is_deprecated:"},{"line_number":291,"context_line":"                info.pop(\u0027deprecated\u0027)"},{"line_number":292,"context_line":"            info.pop(\u0027policy_type\u0027)"},{"line_number":293,"context_line":"        return info"},{"line_number":294,"context_line":""},{"line_number":295,"context_line":"    def _validate_policy_name(self, name):"}],"source_content_type":"text/x-python","patch_set":9,"id":"5f7c97a3_a39fc663","line":292,"updated":"2018-06-07 16:29:05.000000000","message":"also pop \u0027diskfile\u0027 \n\nThe intent is to not publish anything in the swift /info response that reveals too much about how the op has implemented their policies. So, for example, the published policies info does not currently include anything that reveals if a policy is replication or EC.","commit_id":"c5b6202f36c364f4d700aa19f8c312a41ca53783"},{"author":{"_account_id":25251,"name":"Alexandre Lécuyer","email":"alexandre.lecuyer@corp.ovh.com","username":"alecuyer"},"change_message_id":"05fd05f2ee693e574d011c2bf0acb8536398096c","unresolved":false,"context_lines":[{"line_number":289,"context_line":"                info.pop(\u0027default\u0027)"},{"line_number":290,"context_line":"            if not self.is_deprecated:"},{"line_number":291,"context_line":"                info.pop(\u0027deprecated\u0027)"},{"line_number":292,"context_line":"            info.pop(\u0027policy_type\u0027)"},{"line_number":293,"context_line":"        return info"},{"line_number":294,"context_line":""},{"line_number":295,"context_line":"    def _validate_policy_name(self, name):"}],"source_content_type":"text/x-python","patch_set":9,"id":"5f7c97a3_0d3b2e39","line":292,"in_reply_to":"5f7c97a3_a39fc663","updated":"2018-07-02 13:22:15.000000000","message":"Done","commit_id":"c5b6202f36c364f4d700aa19f8c312a41ca53783"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"0697c1b3aab241cb991ff6a5b42be7e569495e36","unresolved":false,"context_lines":[{"line_number":824,"context_line":"            policy.load_ring(swift_dir)"},{"line_number":825,"context_line":"        return policy.object_ring"},{"line_number":826,"context_line":""},{"line_number":827,"context_line":"    def get_diskfile_manager(self, policy_idx, conf, logger):"},{"line_number":828,"context_line":"        \"\"\""},{"line_number":829,"context_line":"        Get the ring object to use to handle a request based on its policy."},{"line_number":830,"context_line":""},{"line_number":831,"context_line":"        An index of None will be treated as 0."},{"line_number":832,"context_line":""},{"line_number":833,"context_line":"        :param policy_idx: policy index as defined in swift.conf"},{"line_number":834,"context_line":"        :param swift_dir: swift_dir used by the caller"},{"line_number":835,"context_line":"        :returns: appropriate ring object"},{"line_number":836,"context_line":"        \"\"\""},{"line_number":837,"context_line":"        policy \u003d self.get_by_index(policy_idx)"},{"line_number":838,"context_line":"        if not policy:"},{"line_number":839,"context_line":"            raise PolicyError(\"No policy with index %s\" % policy_idx)"},{"line_number":840,"context_line":"        if not policy.diskfile_manager:"},{"line_number":841,"context_line":"            policy.load_diskfile_manager(conf, logger)"},{"line_number":842,"context_line":"        return policy.diskfile_manager"},{"line_number":843,"context_line":""},{"line_number":844,"context_line":"    def get_policy_info(self):"},{"line_number":845,"context_line":"        \"\"\""}],"source_content_type":"text/x-python","patch_set":9,"id":"5f7c97a3_463a80c4","line":842,"range":{"start_line":827,"start_character":0,"end_line":842,"end_character":38},"updated":"2018-06-07 16:29:05.000000000","message":"not used","commit_id":"c5b6202f36c364f4d700aa19f8c312a41ca53783"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"0697c1b3aab241cb991ff6a5b42be7e569495e36","unresolved":false,"context_lines":[{"line_number":377,"context_line":"        \"\"\""},{"line_number":378,"context_line":"        Load the diskfile manager for this policy immediately."},{"line_number":379,"context_line":"        \"\"\""},{"line_number":380,"context_line":"        if self.diskfile_manager:"},{"line_number":381,"context_line":"            return"},{"line_number":382,"context_line":""},{"line_number":383,"context_line":"        dfm_cls \u003d load_pkg_resource(\u0027swift.diskfile\u0027, self.diskfile_uri)"},{"line_number":384,"context_line":"        if self.policy_type !\u003d dfm_cls.policy_type:"}],"source_content_type":"text/x-python","patch_set":10,"id":"5f7c97a3_1fd4c6e8","line":381,"range":{"start_line":380,"start_character":8,"end_line":381,"end_character":18},"updated":"2018-06-07 16:29:05.000000000","message":"I\u0027m not sure about caching the first diskfile manager (DFM)instance in the policy. I\u0027ve read back over Clay\u0027s comments on earlier patchsets, and hope I\u0027m not giving conflicting input, but unlike the ring, diskfile managers are configurable.\n\nBefore, each DiskfileRouter loaded a cache of per-policy DFMs configured using the conf and logger provided to the DiskFileRouter. In theory there could be mulitple DiskFileRouters instantiated in a process each having a map of DFMs with different configs. Now, the first caller in a process to call load_diskfile_manager will dictate the conf and logger.\n\nIn practice this may not matter - I\u0027m not sure we have multiple DiskfileRouters instantiated in same process. But the interface  in IMHO not intuitive and could in the future prove too inflexible or confusing. i.e. a call to load_diskfile_manager(myconf, mylogger) may *or may NOT* return a diskfile manager using myconf and mylogger, depending on how the method was previously called.\n\nAn alternative would be a simplified DiskfileRouter that loads DFMs on demand and maintains a map of policy-\u003eDFM. Something like https://gist.github.com/alistairncoles/352b1f9f047eb72129bec2e7c7cededd\n\nOr, maybe we don\u0027t even need to cache DFM instances?","commit_id":"fd290c33fa3328a62400bc8f2b736330440ca8be"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"0697c1b3aab241cb991ff6a5b42be7e569495e36","unresolved":false,"context_lines":[{"line_number":381,"context_line":"            return"},{"line_number":382,"context_line":""},{"line_number":383,"context_line":"        dfm_cls \u003d load_pkg_resource(\u0027swift.diskfile\u0027, self.diskfile_uri)"},{"line_number":384,"context_line":"        if self.policy_type !\u003d dfm_cls.policy_type:"},{"line_number":385,"context_line":"            raise LookupError(\u0027Invalid policy_type on diskfile %s for policy \u0027"},{"line_number":386,"context_line":"                              \u0027%s\u0027 % (self.diskfile_uri, self.name))"},{"line_number":387,"context_line":"        self.diskfile_manager \u003d dfm_cls(conf, logger)"}],"source_content_type":"text/x-python","patch_set":10,"id":"5f7c97a3_fa647823","line":384,"range":{"start_line":384,"start_character":8,"end_line":384,"end_character":51},"updated":"2018-06-07 16:29:05.000000000","message":"this check is a good idea to avoid config errors, but the implementation does not seem very flexible in terms of future diskfile implementations. For example, it restricts each dfm_cls to only support one policy type, but assumes that it can support all variants of that policy type.\n\nHow about something like:\n\n  if not dfm_cls.check_policy(policy):\n      raise ....\n\n\noff-topic:\n\n...which got me thinking, why do we init  DFM instance per-policy, but not encapuslate the policy in the DFM? Instead, policy is separately passed to the DFM instance methods. And there is no check that it the *same* policy that was used here!\n\nIt would be probably be too disruptive, but I wonder if policy should become an attribute of a DFM instance?\n\nNot for this patch, as it is unnecessary disruption.","commit_id":"fd290c33fa3328a62400bc8f2b736330440ca8be"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"0697c1b3aab241cb991ff6a5b42be7e569495e36","unresolved":false,"context_lines":[{"line_number":382,"context_line":""},{"line_number":383,"context_line":"        dfm_cls \u003d load_pkg_resource(\u0027swift.diskfile\u0027, self.diskfile_uri)"},{"line_number":384,"context_line":"        if self.policy_type !\u003d dfm_cls.policy_type:"},{"line_number":385,"context_line":"            raise LookupError(\u0027Invalid policy_type on diskfile %s for policy \u0027"},{"line_number":386,"context_line":"                              \u0027%s\u0027 % (self.diskfile_uri, self.name))"},{"line_number":387,"context_line":"        self.diskfile_manager \u003d dfm_cls(conf, logger)"},{"line_number":388,"context_line":""}],"source_content_type":"text/x-python","patch_set":10,"id":"5f7c97a3_c7a18e97","line":385,"range":{"start_line":385,"start_character":18,"end_line":385,"end_character":29},"updated":"2018-06-07 16:29:05.000000000","message":"LookupError doesn\u0027t seem appropriate. ValueError? TypeError?","commit_id":"fd290c33fa3328a62400bc8f2b736330440ca8be"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"0697c1b3aab241cb991ff6a5b42be7e569495e36","unresolved":false,"context_lines":[{"line_number":824,"context_line":"            policy.load_ring(swift_dir)"},{"line_number":825,"context_line":"        return policy.object_ring"},{"line_number":826,"context_line":""},{"line_number":827,"context_line":"    def get_diskfile_manager(self, policy_idx, conf, logger):"},{"line_number":828,"context_line":"        \"\"\""},{"line_number":829,"context_line":"        Get the ring object to use to handle a request based on its policy."},{"line_number":830,"context_line":""},{"line_number":831,"context_line":"        An index of None will be treated as 0."},{"line_number":832,"context_line":""},{"line_number":833,"context_line":"        :param policy_idx: policy index as defined in swift.conf"},{"line_number":834,"context_line":"        :param swift_dir: swift_dir used by the caller"},{"line_number":835,"context_line":"        :returns: appropriate ring object"},{"line_number":836,"context_line":"        \"\"\""},{"line_number":837,"context_line":"        policy \u003d self.get_by_index(policy_idx)"},{"line_number":838,"context_line":"        if not policy:"},{"line_number":839,"context_line":"            raise PolicyError(\"No policy with index %s\" % policy_idx)"},{"line_number":840,"context_line":"        if not policy.diskfile_manager:"},{"line_number":841,"context_line":"            policy.load_diskfile_manager(conf, logger)"},{"line_number":842,"context_line":"        return policy.diskfile_manager"},{"line_number":843,"context_line":""},{"line_number":844,"context_line":"    def get_policy_info(self):"},{"line_number":845,"context_line":"        \"\"\""}],"source_content_type":"text/x-python","patch_set":10,"id":"5f7c97a3_ff4d4ac4","line":842,"range":{"start_line":827,"start_character":4,"end_line":842,"end_character":38},"updated":"2018-06-07 16:29:05.000000000","message":"not used","commit_id":"fd290c33fa3328a62400bc8f2b736330440ca8be"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"14146f12fbe490b052ad6e81e5e42fdac72d4554","unresolved":false,"context_lines":[{"line_number":188,"context_line":"        self.ring_name \u003d _get_policy_string(\u0027object\u0027, self.idx)"},{"line_number":189,"context_line":"        self.object_ring \u003d object_ring"},{"line_number":190,"context_line":""},{"line_number":191,"context_line":"        self.diskfile_uri \u003d diskfile_uri"},{"line_number":192,"context_line":""},{"line_number":193,"context_line":"    @property"},{"line_number":194,"context_line":"    def name(self):"}],"source_content_type":"text/x-python","patch_set":14,"id":"5f7c97a3_ce0e70e3","line":191,"updated":"2018-07-03 15:04:15.000000000","message":"I tried this:\n\n  dfm_cls \u003d load_pkg_resource(\u0027swift.diskfile\u0027,  self.diskfile_uri)\n\nso that we\u0027d see errors as policies were loaded. But there is a circular import :/","commit_id":"29065746b5957026010654ac99cb6c422ff95ae9"},{"author":{"_account_id":4608,"name":"Kota Tsuyuzaki","email":"bloodeagle40234@gmail.com","username":"tsuyuzaki-kota"},"change_message_id":"fccd279bd6434de098642918a3ff5f695dabd24d","unresolved":false,"context_lines":[{"line_number":255,"context_line":"            \u0027policy_type\u0027: \u0027policy_type\u0027,"},{"line_number":256,"context_line":"            \u0027default\u0027: \u0027is_default\u0027,"},{"line_number":257,"context_line":"            \u0027deprecated\u0027: \u0027is_deprecated\u0027,"},{"line_number":258,"context_line":"            \u0027diskfile\u0027: \u0027diskfile_uri\u0027"},{"line_number":259,"context_line":"        }"},{"line_number":260,"context_line":""},{"line_number":261,"context_line":"    @classmethod"}],"source_content_type":"text/x-python","patch_set":16,"id":"5f7c97a3_386d3c2f","line":258,"range":{"start_line":258,"start_character":12,"end_line":258,"end_character":38},"updated":"2018-07-24 08:26:53.000000000","message":"Could we make the name being consistent between the option name and inner variable name?\n\nI understand, probably, the name diskfile_uri is preferred to avoid conflict meaning the actual diskfile instance and \"diskfile\" in config is for user friendly option name, tho.\n\nSomething like \"diskfile_path\", \"diskfile_module\" ???","commit_id":"30db6a95721e18c40027d199e5c6b1f1a5c84929"},{"author":{"_account_id":13852,"name":"Romain LE DISEZ","email":"romain.le-disez@corp.ovh.com","username":"rledisez"},"change_message_id":"e406534231295980f4eb6daba9e6a7f307a70d3a","unresolved":false,"context_lines":[{"line_number":255,"context_line":"            \u0027policy_type\u0027: \u0027policy_type\u0027,"},{"line_number":256,"context_line":"            \u0027default\u0027: \u0027is_default\u0027,"},{"line_number":257,"context_line":"            \u0027deprecated\u0027: \u0027is_deprecated\u0027,"},{"line_number":258,"context_line":"            \u0027diskfile\u0027: \u0027diskfile_uri\u0027"},{"line_number":259,"context_line":"        }"},{"line_number":260,"context_line":""},{"line_number":261,"context_line":"    @classmethod"}],"source_content_type":"text/x-python","patch_set":16,"id":"5f7c97a3_7ac62844","line":258,"range":{"start_line":258,"start_character":12,"end_line":258,"end_character":38},"in_reply_to":"5f7c97a3_386d3c2f","updated":"2018-07-25 10:07:25.000000000","message":"Done","commit_id":"30db6a95721e18c40027d199e5c6b1f1a5c84929"},{"author":{"_account_id":4608,"name":"Kota Tsuyuzaki","email":"bloodeagle40234@gmail.com","username":"tsuyuzaki-kota"},"change_message_id":"c5757b11371bfc18eb0972b71bc1d40165264b63","unresolved":false,"context_lines":[{"line_number":395,"context_line":"        try:"},{"line_number":396,"context_line":"            dfm_cls \u003d load_pkg_resource(\u0027swift.diskfile\u0027, self.diskfile_module)"},{"line_number":397,"context_line":"        except ImportError as err:"},{"line_number":398,"context_line":"            raise PolicyError(\u0027Unable to load diskfile %s for policy %s: %s\u0027 %"},{"line_number":399,"context_line":"                              (self.diskfile_module, self.name, err))"},{"line_number":400,"context_line":"        try:"},{"line_number":401,"context_line":"            dfm_cls.check_policy(self)"}],"source_content_type":"text/x-python","patch_set":18,"id":"5f7c97a3_f8d509b9","line":398,"range":{"start_line":398,"start_character":46,"end_line":398,"end_character":54},"updated":"2018-07-30 10:57:53.000000000","message":"nits) diskfile module","commit_id":"4b850da4c9f600aaead6ccfc987c3481c21244fe"},{"author":{"_account_id":4608,"name":"Kota Tsuyuzaki","email":"bloodeagle40234@gmail.com","username":"tsuyuzaki-kota"},"change_message_id":"c5757b11371bfc18eb0972b71bc1d40165264b63","unresolved":false,"context_lines":[{"line_number":400,"context_line":"        try:"},{"line_number":401,"context_line":"            dfm_cls.check_policy(self)"},{"line_number":402,"context_line":"        except ValueError as err:"},{"line_number":403,"context_line":"            raise PolicyError(\u0027Invalid diskfile %s for policy %s: %s\u0027 %"},{"line_number":404,"context_line":"                              (self.diskfile_module, self.name, err))"},{"line_number":405,"context_line":""},{"line_number":406,"context_line":"        return dfm_cls(*args, **kwargs)"}],"source_content_type":"text/x-python","patch_set":18,"id":"5f7c97a3_d8d84dee","line":403,"range":{"start_line":403,"start_character":39,"end_line":403,"end_character":50},"updated":"2018-07-30 10:57:53.000000000","message":"nits) ditto","commit_id":"4b850da4c9f600aaead6ccfc987c3481c21244fe"},{"author":{"_account_id":4608,"name":"Kota Tsuyuzaki","email":"bloodeagle40234@gmail.com","username":"tsuyuzaki-kota"},"change_message_id":"c5757b11371bfc18eb0972b71bc1d40165264b63","unresolved":false,"context_lines":[{"line_number":401,"context_line":"            dfm_cls.check_policy(self)"},{"line_number":402,"context_line":"        except ValueError as err:"},{"line_number":403,"context_line":"            raise PolicyError(\u0027Invalid diskfile %s for policy %s: %s\u0027 %"},{"line_number":404,"context_line":"                              (self.diskfile_module, self.name, err))"},{"line_number":405,"context_line":""},{"line_number":406,"context_line":"        return dfm_cls(*args, **kwargs)"},{"line_number":407,"context_line":""}],"source_content_type":"text/x-python","patch_set":18,"id":"5f7c97a3_d236e9e8","line":404,"range":{"start_line":404,"start_character":62,"end_line":404,"end_character":67},"updated":"2018-07-30 10:57:53.000000000","message":"What I\u0027d recommend for fixing Kazuhiro comment is just removing error message from here because it\u0027s not from policy perspective (just a diskfile manager perspective)","commit_id":"4b850da4c9f600aaead6ccfc987c3481c21244fe"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"6fdecae0060ba5a2a6dfedf511cac700be6ab433","unresolved":false,"context_lines":[{"line_number":403,"context_line":"        except ValueError as err:"},{"line_number":404,"context_line":"            raise PolicyError("},{"line_number":405,"context_line":"                \u0027Invalid diskfile_module %s for policy %s:%s (%s)\u0027 %"},{"line_number":406,"context_line":"                (self.diskfile_module, int(self), self.name, self.policy_type))"},{"line_number":407,"context_line":""},{"line_number":408,"context_line":"        return dfm_cls(*args, **kwargs)"},{"line_number":409,"context_line":""}],"source_content_type":"text/x-python","patch_set":20,"id":"3f79a3b5_3d4d8871","line":406,"updated":"2018-09-19 21:58:00.000000000","message":"oh, watch out operator error.  nice.","commit_id":"673fda76203be7c70afe6236d3fadc88deb75117"}],"swift/common/utils.py":[{"author":{"_account_id":4608,"name":"Kota Tsuyuzaki","email":"bloodeagle40234@gmail.com","username":"tsuyuzaki-kota"},"change_message_id":"fccd279bd6434de098642918a3ff5f695dabd24d","unresolved":false,"context_lines":[{"line_number":5122,"context_line":"        dist \u003d uri"},{"line_number":5123,"context_line":""},{"line_number":5124,"context_line":"    if scheme !\u003d \u0027egg\u0027:"},{"line_number":5125,"context_line":"        raise TypeError(\u0027Unhandled URI scheme: %r\u0027 % scheme)"},{"line_number":5126,"context_line":"    return pkg_resources.load_entry_point(dist, group, name)"},{"line_number":5127,"context_line":""},{"line_number":5128,"context_line":""}],"source_content_type":"text/x-python","patch_set":16,"id":"5f7c97a3_78741438","line":5125,"updated":"2018-07-24 08:26:53.000000000","message":"Hmm... I\u0027m not sure if this check is enough or not.\n\nWhen i made a mistake on the url scheme like \"notegg.swift#replication.fs\" (using dot instead of colon), it fails as package not found other than scheme error.\n\nAh... it\u0027s because, if no colon found in the scheme, this function assumes \"it uses shorter expression\" of the package. Hmm...\n\n\nI\u0027m feeling that how we want to use this staff in the future. The key points are:\n\n - will we need to support any schemes other than egg?\n - will we need third party diskfile located in out of swift package?\n\nThinking of losf, if it requires a module *located in Swift package* using *egg scheme* I think we don\u0027t have to support full path uri like \"egg:swift#another_diskfile.fs\", just \"another_diskfile.fs\" is enough for users.","commit_id":"30db6a95721e18c40027d199e5c6b1f1a5c84929"},{"author":{"_account_id":4608,"name":"Kota Tsuyuzaki","email":"bloodeagle40234@gmail.com","username":"tsuyuzaki-kota"},"change_message_id":"c5757b11371bfc18eb0972b71bc1d40165264b63","unresolved":false,"context_lines":[{"line_number":5122,"context_line":"        dist \u003d uri"},{"line_number":5123,"context_line":""},{"line_number":5124,"context_line":"    if scheme !\u003d \u0027egg\u0027:"},{"line_number":5125,"context_line":"        raise TypeError(\u0027Unhandled URI scheme: %r\u0027 % scheme)"},{"line_number":5126,"context_line":"    return pkg_resources.load_entry_point(dist, group, name)"},{"line_number":5127,"context_line":""},{"line_number":5128,"context_line":""}],"source_content_type":"text/x-python","patch_set":16,"id":"5f7c97a3_922b96e8","line":5125,"in_reply_to":"5f7c97a3_3adaf0e5","updated":"2018-07-30 10:57:53.000000000","message":"Those all makes me sense. Thanks.\n\nEven if we don\u0027t want to support any scheme other than egg, entry point system with python packaging might be user friendly api for any other diskfile venders. Then if we follow the packaging style that can specify the custom package in out of swift scope, it may be useful. I think swift-on-file can follow the scheme so that i\u0027d like to here tdasilva\u0027s opinion for the format.","commit_id":"30db6a95721e18c40027d199e5c6b1f1a5c84929"},{"author":{"_account_id":13852,"name":"Romain LE DISEZ","email":"romain.le-disez@corp.ovh.com","username":"rledisez"},"change_message_id":"e406534231295980f4eb6daba9e6a7f307a70d3a","unresolved":false,"context_lines":[{"line_number":5122,"context_line":"        dist \u003d uri"},{"line_number":5123,"context_line":""},{"line_number":5124,"context_line":"    if scheme !\u003d \u0027egg\u0027:"},{"line_number":5125,"context_line":"        raise TypeError(\u0027Unhandled URI scheme: %r\u0027 % scheme)"},{"line_number":5126,"context_line":"    return pkg_resources.load_entry_point(dist, group, name)"},{"line_number":5127,"context_line":""},{"line_number":5128,"context_line":""}],"source_content_type":"text/x-python","patch_set":16,"id":"5f7c97a3_3adaf0e5","line":5125,"in_reply_to":"5f7c97a3_78741438","updated":"2018-07-25 10:07:25.000000000","message":"\u003e - will we need to support any schemes other than egg?\n\nProbably not. I thought it was a good idea being consistent between middlewares and diskfiles. It can be removed easily, I have no strong opinion on that.\n\n \u003e - will we need third party diskfile located in out of swift\n \u003e package?\n\nI think not all implementations should be included in the core of Swift. A disk manufacturer could implement a diskfile for its very specific disk API (eg: egg:manufacturer#replication.http_disk)\n\n \n \u003e Thinking of losf, if it requires a module *located in Swift\n \u003e package* using *egg scheme* I think we don\u0027t have to support full\n \u003e path uri like \"egg:swift#another_diskfile.fs\", just\n \u003e \"another_diskfile.fs\" is enough for users.\n\n\nThis part of the patch is not specific to LOSF. What you describe was the first proposal (patchset 1). It changed on a suggestion of Clay. I think it\u0027s a good move.","commit_id":"30db6a95721e18c40027d199e5c6b1f1a5c84929"},{"author":{"_account_id":14766,"name":"Kazuhiro MIYAHARA","email":"miyahara.kazuhiro@lab.ntt.co.jp","username":"miyahara"},"change_message_id":"27d50d17d1d83af01404bfc354021c8cdb429f46","unresolved":false,"context_lines":[{"line_number":5107,"context_line":"    return os.sep.join(path_components)"},{"line_number":5108,"context_line":""},{"line_number":5109,"context_line":""},{"line_number":5110,"context_line":"def load_pkg_resource(group, uri):"},{"line_number":5111,"context_line":"    if \u0027#\u0027 in uri:"},{"line_number":5112,"context_line":"        uri, name \u003d uri.split(\u0027#\u0027, 1)"},{"line_number":5113,"context_line":"    else:"},{"line_number":5114,"context_line":"        name \u003d uri"},{"line_number":5115,"context_line":"        uri \u003d \u0027egg:swift\u0027"},{"line_number":5116,"context_line":""},{"line_number":5117,"context_line":"    if \u0027:\u0027 in uri:"},{"line_number":5118,"context_line":"        scheme, dist \u003d uri.split(\u0027:\u0027, 1)"},{"line_number":5119,"context_line":"        scheme \u003d scheme.lower()"},{"line_number":5120,"context_line":"    else:"},{"line_number":5121,"context_line":"        scheme \u003d \u0027egg\u0027"},{"line_number":5122,"context_line":"        dist \u003d uri"},{"line_number":5123,"context_line":""},{"line_number":5124,"context_line":"    if scheme !\u003d \u0027egg\u0027:"},{"line_number":5125,"context_line":"        raise TypeError(\u0027Unhandled URI scheme: %r\u0027 % scheme)"},{"line_number":5126,"context_line":"    return pkg_resources.load_entry_point(dist, group, name)"},{"line_number":5127,"context_line":""},{"line_number":5128,"context_line":""},{"line_number":5129,"context_line":"class PipeMutex(object):"}],"source_content_type":"text/x-python","patch_set":16,"id":"5f7c97a3_6c0e7b3e","line":5126,"range":{"start_line":5110,"start_character":0,"end_line":5126,"end_character":60},"updated":"2018-07-25 04:34:10.000000000","message":"My concern is that this function looks like re-implementation of pastedeploy library[1].\nSo if \"egg:swift#another_diskfile.fs\" style is required, I think pastedeploy should be used.\n\n[1]: http://pastedeploy.readthedocs.io/en/latest/","commit_id":"30db6a95721e18c40027d199e5c6b1f1a5c84929"},{"author":{"_account_id":13852,"name":"Romain LE DISEZ","email":"romain.le-disez@corp.ovh.com","username":"rledisez"},"change_message_id":"e406534231295980f4eb6daba9e6a7f307a70d3a","unresolved":false,"context_lines":[{"line_number":5107,"context_line":"    return os.sep.join(path_components)"},{"line_number":5108,"context_line":""},{"line_number":5109,"context_line":""},{"line_number":5110,"context_line":"def load_pkg_resource(group, uri):"},{"line_number":5111,"context_line":"    if \u0027#\u0027 in uri:"},{"line_number":5112,"context_line":"        uri, name \u003d uri.split(\u0027#\u0027, 1)"},{"line_number":5113,"context_line":"    else:"},{"line_number":5114,"context_line":"        name \u003d uri"},{"line_number":5115,"context_line":"        uri \u003d \u0027egg:swift\u0027"},{"line_number":5116,"context_line":""},{"line_number":5117,"context_line":"    if \u0027:\u0027 in uri:"},{"line_number":5118,"context_line":"        scheme, dist \u003d uri.split(\u0027:\u0027, 1)"},{"line_number":5119,"context_line":"        scheme \u003d scheme.lower()"},{"line_number":5120,"context_line":"    else:"},{"line_number":5121,"context_line":"        scheme \u003d \u0027egg\u0027"},{"line_number":5122,"context_line":"        dist \u003d uri"},{"line_number":5123,"context_line":""},{"line_number":5124,"context_line":"    if scheme !\u003d \u0027egg\u0027:"},{"line_number":5125,"context_line":"        raise TypeError(\u0027Unhandled URI scheme: %r\u0027 % scheme)"},{"line_number":5126,"context_line":"    return pkg_resources.load_entry_point(dist, group, name)"},{"line_number":5127,"context_line":""},{"line_number":5128,"context_line":""},{"line_number":5129,"context_line":"class PipeMutex(object):"}],"source_content_type":"text/x-python","patch_set":16,"id":"5f7c97a3_3a88b0c1","line":5126,"range":{"start_line":5110,"start_character":0,"end_line":5126,"end_character":60},"in_reply_to":"5f7c97a3_6c0e7b3e","updated":"2018-07-25 10:07:25.000000000","message":"IIRW, I took the code from paste;deploy. I had to copy it because it\u0027s not usable outside of the module. It is meant to load apps or filters, not random classes.","commit_id":"30db6a95721e18c40027d199e5c6b1f1a5c84929"}],"swift/obj/diskfile.py":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"fd02bd4ed575ac8645de80a13502c1475eb8f8a4","unresolved":false,"context_lines":[{"line_number":541,"context_line":"    def __getitem__(self, policy):"},{"line_number":542,"context_line":"        return self.policy_to_manager[policy]"},{"line_number":543,"context_line":""},{"line_number":544,"context_line":""},{"line_number":545,"context_line":"class BaseDiskFileManager(object):"},{"line_number":546,"context_line":"    \"\"\""},{"line_number":547,"context_line":"    Management class for devices, providing common place for shared parameters"}],"source_content_type":"text/x-python","patch_set":1,"id":"1a1ced50_eed1e1fc","side":"PARENT","line":544,"updated":"2017-03-17 22:44:23.000000000","message":"Yay!","commit_id":"5c0814daa6135982e6a4d29f3a12743f698fe4b5"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"14146f12fbe490b052ad6e81e5e42fdac72d4554","unresolved":false,"context_lines":[{"line_number":607,"context_line":"        self.args \u003d args"},{"line_number":608,"context_line":"        self.kwargs \u003d kwargs"},{"line_number":609,"context_line":""},{"line_number":610,"context_line":"    def _load_diskfile_manager(self, policy):"},{"line_number":611,"context_line":"        dfm_cls \u003d load_pkg_resource(\u0027swift.diskfile\u0027, policy.diskfile_uri)"},{"line_number":612,"context_line":"        try:"},{"line_number":613,"context_line":"            dfm_cls.check_policy(policy)"}],"source_content_type":"text/x-python","patch_set":14,"id":"5f7c97a3_ce2190a0","line":610,"range":{"start_line":610,"start_character":8,"end_line":610,"end_character":30},"updated":"2018-07-03 15:04:15.000000000","message":"I am still undecided whether this is better here or in the BaseStoragePolicy class where it would provide a convenient way to instantiate a disk file manager without needing a DiskFileRouter.","commit_id":"29065746b5957026010654ac99cb6c422ff95ae9"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"14146f12fbe490b052ad6e81e5e42fdac72d4554","unresolved":false,"context_lines":[{"line_number":608,"context_line":"        self.kwargs \u003d kwargs"},{"line_number":609,"context_line":""},{"line_number":610,"context_line":"    def _load_diskfile_manager(self, policy):"},{"line_number":611,"context_line":"        dfm_cls \u003d load_pkg_resource(\u0027swift.diskfile\u0027, policy.diskfile_uri)"},{"line_number":612,"context_line":"        try:"},{"line_number":613,"context_line":"            dfm_cls.check_policy(policy)"},{"line_number":614,"context_line":"        except ValueError as err:"}],"source_content_type":"text/x-python","patch_set":14,"id":"5f7c97a3_ae053412","line":611,"range":{"start_line":611,"start_character":18,"end_line":611,"end_character":35},"updated":"2018-07-03 15:04:15.000000000","message":"nit: catch any ImportError and add some context","commit_id":"29065746b5957026010654ac99cb6c422ff95ae9"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"14146f12fbe490b052ad6e81e5e42fdac72d4554","unresolved":false,"context_lines":[{"line_number":621,"context_line":"        policy_index \u003d int(policy)"},{"line_number":622,"context_line":"        if policy_index not in self.policy_to_manager:"},{"line_number":623,"context_line":"            self.policy_to_manager[policy_index] \u003d \\"},{"line_number":624,"context_line":"                self._load_diskfile_manager(policy)"},{"line_number":625,"context_line":"        return self.policy_to_manager[policy_index]"},{"line_number":626,"context_line":""},{"line_number":627,"context_line":""}],"source_content_type":"text/x-python","patch_set":14,"id":"5f7c97a3_5bff8401","line":624,"range":{"start_line":624,"start_character":16,"end_line":624,"end_character":51},"updated":"2018-07-03 15:04:15.000000000","message":"exceptions are only raised when the policy is used - we should instantiate all policy diskfile managers during the init so that the object server fails during startup if there is a bad policy config.","commit_id":"29065746b5957026010654ac99cb6c422ff95ae9"},{"author":{"_account_id":14766,"name":"Kazuhiro MIYAHARA","email":"miyahara.kazuhiro@lab.ntt.co.jp","username":"miyahara"},"change_message_id":"27d50d17d1d83af01404bfc354021c8cdb429f46","unresolved":false,"context_lines":[{"line_number":2819,"context_line":"    @classmethod"},{"line_number":2820,"context_line":"    def check_policy(cls, policy):"},{"line_number":2821,"context_line":"        if policy.policy_type !\u003d REPL_POLICY:"},{"line_number":2822,"context_line":"            raise ValueError(\u0027Invalid policy_type: %s\u0027 % policy.policy_type)"},{"line_number":2823,"context_line":""},{"line_number":2824,"context_line":"    def _process_ondisk_files(self, exts, results, **kwargs):"},{"line_number":2825,"context_line":"        \"\"\""}],"source_content_type":"text/x-python","patch_set":16,"id":"5f7c97a3_5769cc8a","line":2822,"range":{"start_line":2822,"start_character":29,"end_line":2822,"end_character":75},"updated":"2018-07-25 04:34:10.000000000","message":"With this implementation, user will get error message \"Invalid policy_type: erasure_coding\" in following mistaken swift.conf case.\n\n  [storage-policy:1]\n  name \u003d silver\n  policy_type \u003d erasure_coding\n  diskfile \u003d replication.fs\n\nIn this case, erasure_coding policy_type itself is valid.\nCombination of policy_type and diskfile is invalid.\n\nSo I think error message should tell about policy_type doesn\u0027t match diskfile\u0027s requirement.","commit_id":"30db6a95721e18c40027d199e5c6b1f1a5c84929"},{"author":{"_account_id":13852,"name":"Romain LE DISEZ","email":"romain.le-disez@corp.ovh.com","username":"rledisez"},"change_message_id":"e406534231295980f4eb6daba9e6a7f307a70d3a","unresolved":false,"context_lines":[{"line_number":2819,"context_line":"    @classmethod"},{"line_number":2820,"context_line":"    def check_policy(cls, policy):"},{"line_number":2821,"context_line":"        if policy.policy_type !\u003d REPL_POLICY:"},{"line_number":2822,"context_line":"            raise ValueError(\u0027Invalid policy_type: %s\u0027 % policy.policy_type)"},{"line_number":2823,"context_line":""},{"line_number":2824,"context_line":"    def _process_ondisk_files(self, exts, results, **kwargs):"},{"line_number":2825,"context_line":"        \"\"\""}],"source_content_type":"text/x-python","patch_set":16,"id":"5f7c97a3_9ab22496","line":2822,"range":{"start_line":2822,"start_character":29,"end_line":2822,"end_character":75},"in_reply_to":"5f7c97a3_5769cc8a","updated":"2018-07-25 10:07:25.000000000","message":"Done","commit_id":"30db6a95721e18c40027d199e5c6b1f1a5c84929"},{"author":{"_account_id":14766,"name":"Kazuhiro MIYAHARA","email":"miyahara.kazuhiro@lab.ntt.co.jp","username":"miyahara"},"change_message_id":"cf95c55327ec58be78481c62be6d50085a9889f8","unresolved":false,"context_lines":[{"line_number":2819,"context_line":"    @classmethod"},{"line_number":2820,"context_line":"    def check_policy(cls, policy):"},{"line_number":2821,"context_line":"        if policy.policy_type !\u003d REPL_POLICY:"},{"line_number":2822,"context_line":"            raise ValueError(\u0027Invalid policy_type: %s\u0027 % policy.policy_type)"},{"line_number":2823,"context_line":""},{"line_number":2824,"context_line":"    def _process_ondisk_files(self, exts, results, **kwargs):"},{"line_number":2825,"context_line":"        \"\"\""}],"source_content_type":"text/x-python","patch_set":16,"id":"5f7c97a3_18144e9e","line":2822,"range":{"start_line":2822,"start_character":29,"end_line":2822,"end_character":75},"in_reply_to":"5f7c97a3_5e58a587","updated":"2018-08-01 15:03:58.000000000","message":"That makes sense. I agree kota\u0027s opinion.","commit_id":"30db6a95721e18c40027d199e5c6b1f1a5c84929"},{"author":{"_account_id":4608,"name":"Kota Tsuyuzaki","email":"bloodeagle40234@gmail.com","username":"tsuyuzaki-kota"},"change_message_id":"c5757b11371bfc18eb0972b71bc1d40165264b63","unresolved":false,"context_lines":[{"line_number":2819,"context_line":"    @classmethod"},{"line_number":2820,"context_line":"    def check_policy(cls, policy):"},{"line_number":2821,"context_line":"        if policy.policy_type !\u003d REPL_POLICY:"},{"line_number":2822,"context_line":"            raise ValueError(\u0027Invalid policy_type: %s\u0027 % policy.policy_type)"},{"line_number":2823,"context_line":""},{"line_number":2824,"context_line":"    def _process_ondisk_files(self, exts, results, **kwargs):"},{"line_number":2825,"context_line":"        \"\"\""}],"source_content_type":"text/x-python","patch_set":16,"id":"5f7c97a3_5e58a587","line":2822,"range":{"start_line":2822,"start_character":29,"end_line":2822,"end_character":75},"in_reply_to":"5f7c97a3_9ab22496","updated":"2018-07-30 10:57:53.000000000","message":"I think the original code was correct. This is the perspective from diskfile manager so that  disk file just checked if the policy was correct or not. Then, *StoragePolicy* should be responsible to handle the invalid policy type about the mis configuration of diskfile module","commit_id":"30db6a95721e18c40027d199e5c6b1f1a5c84929"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"6fdecae0060ba5a2a6dfedf511cac700be6ab433","unresolved":false,"context_lines":[{"line_number":709,"context_line":"    @classmethod"},{"line_number":710,"context_line":"    def check_policy(cls, policy):"},{"line_number":711,"context_line":"        if policy.policy_type !\u003d cls.policy:"},{"line_number":712,"context_line":"            raise ValueError(\u0027Invalid policy_type: %s\u0027 % policy.policy_type)"},{"line_number":713,"context_line":""},{"line_number":714,"context_line":"    def make_on_disk_filename(self, timestamp, ext\u003dNone,"},{"line_number":715,"context_line":"                              ctype_timestamp\u003dNone, *a, **kw):"}],"source_content_type":"text/x-python","patch_set":20,"id":"3f79a3b5_7d31c0ec","line":712,"updated":"2018-09-19 21:58:00.000000000","message":"boom!  there\u0027s your safty net - excellent.","commit_id":"673fda76203be7c70afe6236d3fadc88deb75117"}],"swift/obj/reconstructor.py":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"fd02bd4ed575ac8645de80a13502c1475eb8f8a4","unresolved":false,"context_lines":[{"line_number":492,"context_line":"            self.last_reconstruction_count \u003d self.reconstruction_count"},{"line_number":493,"context_line":""},{"line_number":494,"context_line":"    def _get_hashes(self, policy, path, recalculate\u003dNone, do_listdir\u003dFalse):"},{"line_number":495,"context_line":"        df_mgr \u003d policy.diskfile_manager"},{"line_number":496,"context_line":"        hashed, suffix_hashes \u003d tpool_reraise("},{"line_number":497,"context_line":"            df_mgr._get_hashes, path, recalculate\u003drecalculate,"},{"line_number":498,"context_line":"            do_listdir\u003ddo_listdir)"}],"source_content_type":"text/x-python","patch_set":1,"id":"1a1ced50_0ecf1d57","line":495,"updated":"2017-03-17 22:44:23.000000000","message":"I think that looks great!","commit_id":"915db8631e7d8e1fa5f698aad055dfacfb76f27f"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"fd02bd4ed575ac8645de80a13502c1475eb8f8a4","unresolved":false,"context_lines":[{"line_number":879,"context_line":"        policy2devices \u003d {}"},{"line_number":880,"context_line":""},{"line_number":881,"context_line":"        for policy in ec_policies:"},{"line_number":882,"context_line":"            self.load_object_ring(policy)"},{"line_number":883,"context_line":"            local_devices \u003d list(six.moves.filter("},{"line_number":884,"context_line":"                lambda dev: dev and is_local_device("},{"line_number":885,"context_line":"                    ips, self.port,"}],"source_content_type":"text/x-python","patch_set":1,"id":"1a1ced50_ce034572","line":882,"updated":"2017-03-17 22:44:23.000000000","message":"I\u0027m surprised this this doesn\u0027t happen on process init... maybe there\u0027s a good reason?","commit_id":"915db8631e7d8e1fa5f698aad055dfacfb76f27f"},{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"004d5f4d42e74ad52b959a21aa289c6e79740340","unresolved":false,"context_lines":[{"line_number":907,"context_line":"        all_parts \u003d []"},{"line_number":908,"context_line":""},{"line_number":909,"context_line":"        for policy, local_devices in policy2devices.items():"},{"line_number":910,"context_line":"            df_mgr \u003d self.load_diskfile_manager(policy)"},{"line_number":911,"context_line":"            for local_dev in local_devices:"},{"line_number":912,"context_line":"                dev_path \u003d df_mgr.get_dev_path(local_dev[\u0027device\u0027])"},{"line_number":913,"context_line":"                if not dev_path:"}],"source_content_type":"text/x-python","patch_set":2,"id":"1a1ced50_2e73c64e","line":910,"updated":"2017-03-20 18:24:28.000000000","message":"nit: slightly prefer explicitly calling policy.load_dfm here and then then accessing the attribute directly everywhere it\u0027s needed.  Seems less sloppy and highlights were we normally expect the policy to get loaded.\n\nIf some tests are calling methods on this object directly w/o instantiating the dfm here - we should update those tests to flesh out their setup.","commit_id":"ab0f7b7c645e920b4adc45f41a178a63c47a4dd0"}],"swift/obj/replicator.py":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"fd02bd4ed575ac8645de80a13502c1475eb8f8a4","unresolved":false,"context_lines":[{"line_number":678,"context_line":"                        str(policy.idx) not in override_policies):"},{"line_number":679,"context_line":"                    continue"},{"line_number":680,"context_line":"                # ensure rings are loaded for policy"},{"line_number":681,"context_line":"                self.load_object_ring(policy)"},{"line_number":682,"context_line":"                jobs +\u003d self.build_replication_jobs("},{"line_number":683,"context_line":"                    policy, ips, override_devices\u003doverride_devices,"},{"line_number":684,"context_line":"                    override_partitions\u003doverride_partitions)"}],"source_content_type":"text/x-python","patch_set":1,"id":"1a1ced50_ce3a6536","line":681,"updated":"2017-03-17 22:44:23.000000000","message":"oh, I guess the idea is we only need to load the ring for policies of the type we wish to handle... same probably goes for loading the dfm!  Granted the memory consumption of an instance of a dfm is much less than a ring - but still for hygiene it seems reasonable to me.\n\nI might be able to be convinced it\u0027d be ok to let this be automatically - seems like there\u0027s some risk avoiding by never letting it happen \"by accident\" w/o someone writing code to say \"load this thing\"","commit_id":"915db8631e7d8e1fa5f698aad055dfacfb76f27f"}],"swift/obj/server.py":[{"author":{"_account_id":1179,"name":"Clay Gerrard","email":"clay.gerrard@gmail.com","username":"clay-gerrard"},"change_message_id":"004d5f4d42e74ad52b959a21aa289c6e79740340","unresolved":false,"context_lines":[{"line_number":208,"context_line":"        DiskFile class would simply over-ride this method to provide that"},{"line_number":209,"context_line":"        behavior."},{"line_number":210,"context_line":"        \"\"\""},{"line_number":211,"context_line":"        policy.load_diskfile_manager(self.conf, self.logger)"},{"line_number":212,"context_line":"        return policy.diskfile_manager.get_diskfile("},{"line_number":213,"context_line":"            device, partition, account, container, obj, policy, **kwargs)"},{"line_number":214,"context_line":""}],"source_content_type":"text/x-python","patch_set":2,"id":"1a1ced50_2e74a6fb","line":211,"updated":"2017-03-20 18:24:28.000000000","message":"Hrm... hard to know which policies this object server process will requests for... maybe better to load the dfm on every defined policy on process start up (that\u0027s basically what the router did)","commit_id":"ab0f7b7c645e920b4adc45f41a178a63c47a4dd0"}],"test/unit/cli/test_relinker.py":[{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"0697c1b3aab241cb991ff6a5b42be7e569495e36","unresolved":false,"context_lines":[{"line_number":154,"context_line":"        # Actually all fragments are non-durable and raise and DiskFileNotExist"},{"line_number":155,"context_line":"        # in EC in this test. However, if the counterpart exists in the new"},{"line_number":156,"context_line":"        # location, this is ok - it will be fixed by the reconstructor later on"},{"line_number":157,"context_line":"        storage_policy._POLICIES[0].policy_type \u003d \u0027erasure_coding\u0027"},{"line_number":158,"context_line":"        storage_policy._POLICIES[0].diskfile_uri \u003d \\"},{"line_number":159,"context_line":"            \u0027egg:swift#erasure_coding.fs\u0027"},{"line_number":160,"context_line":"        storage_policy._POLICIES[0].diskfile_manager \u003d None"},{"line_number":161,"context_line":""},{"line_number":162,"context_line":"        self.assertEqual("},{"line_number":163,"context_line":"            0, relinker.cleanup(self.testdir, self.devices, True, self.logger))"}],"source_content_type":"text/x-python","patch_set":10,"id":"5f7c97a3_4a1c9e76","line":160,"range":{"start_line":157,"start_character":0,"end_line":160,"end_character":59},"updated":"2018-06-07 16:29:05.000000000","message":"it would be better to create a new EC policy instance","commit_id":"fd290c33fa3328a62400bc8f2b736330440ca8be"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"14146f12fbe490b052ad6e81e5e42fdac72d4554","unresolved":false,"context_lines":[{"line_number":155,"context_line":"        # Actually all fragments are non-durable and raise and DiskFileNotExist"},{"line_number":156,"context_line":"        # in EC in this test. However, if the counterpart exists in the new"},{"line_number":157,"context_line":"        # location, this is ok - it will be fixed by the reconstructor later on"},{"line_number":158,"context_line":"        storage_policy._POLICIES[0].policy_type \u003d \u0027erasure_coding\u0027"},{"line_number":159,"context_line":""},{"line_number":160,"context_line":"        # Disable check_policy for this test"},{"line_number":161,"context_line":"        with mock.patch(\u0027swift.obj.diskfile.DiskFileManager.check_policy\u0027):"}],"source_content_type":"text/x-python","patch_set":14,"id":"5f7c97a3_dbb71485","line":158,"range":{"start_line":158,"start_character":8,"end_line":158,"end_character":66},"updated":"2018-07-03 15:04:15.000000000","message":"this actually results in an invalid policy instance","commit_id":"29065746b5957026010654ac99cb6c422ff95ae9"},{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"14146f12fbe490b052ad6e81e5e42fdac72d4554","unresolved":false,"context_lines":[{"line_number":157,"context_line":"        # location, this is ok - it will be fixed by the reconstructor later on"},{"line_number":158,"context_line":"        storage_policy._POLICIES[0].policy_type \u003d \u0027erasure_coding\u0027"},{"line_number":159,"context_line":""},{"line_number":160,"context_line":"        # Disable check_policy for this test"},{"line_number":161,"context_line":"        with mock.patch(\u0027swift.obj.diskfile.DiskFileManager.check_policy\u0027):"},{"line_number":162,"context_line":"            self.assertEqual("},{"line_number":163,"context_line":"                0, relinker.cleanup(self.testdir, self.devices, True,"},{"line_number":164,"context_line":"                                    self.logger))"}],"source_content_type":"text/x-python","patch_set":14,"id":"5f7c97a3_bbb49883","line":161,"range":{"start_line":160,"start_character":8,"end_line":161,"end_character":75},"updated":"2018-07-03 15:04:15.000000000","message":"It would be better to correctly setup the policy.\n\nIt seems that what this test is doing is switching the policy to erasure coding so that the diskfiles appear to be non-durable. Changing the policy_type is sufficient to make the files non-durable, but now breaks check_policy. We should just replace the policy instance with an erasure coding policy.","commit_id":"29065746b5957026010654ac99cb6c422ff95ae9"},{"author":{"_account_id":4608,"name":"Kota Tsuyuzaki","email":"bloodeagle40234@gmail.com","username":"tsuyuzaki-kota"},"change_message_id":"c5757b11371bfc18eb0972b71bc1d40165264b63","unresolved":false,"context_lines":[{"line_number":159,"context_line":"            0, name\u003d\u0027platin\u0027, is_default\u003dTrue, ec_type\u003dDEFAULT_TEST_EC_TYPE,"},{"line_number":160,"context_line":"            ec_ndata\u003d4, ec_nparity\u003d2)]"},{"line_number":161,"context_line":"        storage_policy._POLICIES \u003d StoragePolicyCollection(test_policies)"},{"line_number":162,"context_line":""},{"line_number":163,"context_line":"        self.assertEqual("},{"line_number":164,"context_line":"            0, relinker.cleanup(self.testdir, self.devices, True,"},{"line_number":165,"context_line":"                                self.logger))"}],"source_content_type":"text/x-python","patch_set":18,"id":"5f7c97a3_c3e7fe0f","line":162,"updated":"2018-07-30 10:57:53.000000000","message":"I prefer to use patch_policies instead of mangle the _POLICIES singleton to avoid any influence with other test cases.\n\nThe diff is as follows. I\u0027ll push another follow up with the diff tho.\n\n@@ -26,7 +26,8 @@ from swift.common.storage_policy import (\n \n from swift.obj.diskfile import write_metadata\n \n-from test.unit import FakeLogger, skip_if_no_xattrs, DEFAULT_TEST_EC_TYPE\n+from test.unit import FakeLogger, skip_if_no_xattrs, DEFAULT_TEST_EC_TYPE, \\\n+    patch_policies\n \n\n@@ -148,6 +149,10 @@ class TestRelinker(unittest.TestCase):\n                          [\u0027Error cleaning up %s: %s\u0027 % (self.objname,\n                           repr(exceptions.DiskFileNotExist()))])\n \n+    @patch_policies(\n+        [ECStoragePolicy(\n+         0, name\u003d\u0027platin\u0027, is_default\u003dTrue, ec_type\u003dDEFAULT_TEST_EC_TYPE,\n+         ec_ndata\u003d4, ec_nparity\u003d2)])\n     def test_cleanup_non_durable_fragment(self):\n         self._common_test_cleanup()\n \n@@ -155,11 +160,6 @@ class TestRelinker(unittest.TestCase):\n         # and raise a DiskFileNotExist in EC in this test. However, if the\n         # counterpart exists in the new location, this is ok - it will be fixed\n         # by the reconstructor later on\n-        test_policies \u003d [ECStoragePolicy(\n-            0, name\u003d\u0027platin\u0027, is_default\u003dTrue, ec_type\u003dDEFAULT_TEST_EC_TYPE,\n-            ec_ndata\u003d4, ec_nparity\u003d2)]\n-        storage_policy._POLICIES \u003d StoragePolicyCollection(test_policies)\n-\n         self.assertEqual(\n             0, relinker.cleanup(self.testdir, self.devices, True,\n                                 self.logger))","commit_id":"4b850da4c9f600aaead6ccfc987c3481c21244fe"},{"author":{"_account_id":4608,"name":"Kota Tsuyuzaki","email":"bloodeagle40234@gmail.com","username":"tsuyuzaki-kota"},"change_message_id":"9b43cfec9263de3976af1e729ca61501d52244f0","unresolved":false,"context_lines":[{"line_number":156,"context_line":"    def test_cleanup_non_durable_fragment(self):"},{"line_number":157,"context_line":"        self._common_test_cleanup()"},{"line_number":158,"context_line":""},{"line_number":159,"context_line":"        # Switch the policy type so that actually all fragments are non-durable"},{"line_number":160,"context_line":"        # and raise a DiskFileNotExist in EC in this test. However, if the"},{"line_number":161,"context_line":"        # counterpart exists in the new location, this is ok - it will be fixed"},{"line_number":162,"context_line":"        # by the reconstructor later on"}],"source_content_type":"text/x-python","patch_set":19,"id":"3f79a3b5_b641eb3b","line":159,"range":{"start_line":159,"start_character":10,"end_line":159,"end_character":36},"updated":"2018-08-23 05:40:21.000000000","message":"Perhaps, we should reword around here to address recent change.","commit_id":"71124752e4b3977e826f6bc6b636f7f2a1be91ed"}],"test/unit/common/test_storage_policy.py":[{"author":{"_account_id":4608,"name":"Kota Tsuyuzaki","email":"bloodeagle40234@gmail.com","username":"tsuyuzaki-kota"},"change_message_id":"c5757b11371bfc18eb0972b71bc1d40165264b63","unresolved":false,"context_lines":[{"line_number":1464,"context_line":""},{"line_number":1465,"context_line":"        dfm \u003d policy.get_diskfile_manager({\u0027devices\u0027: \u0027sdb1\u0027}, FakeLogger())"},{"line_number":1466,"context_line":"        self.assertEqual(\u0027sdb1\u0027, dfm.devices)"},{"line_number":1467,"context_line":"        dfm \u003d policy.get_diskfile_manager({\u0027devices\u0027: \u0027sdb2\u0027}, FakeLogger())"},{"line_number":1468,"context_line":"        self.assertEqual(\u0027sdb2\u0027, dfm.devices)"},{"line_number":1469,"context_line":"        dfm2 \u003d policy.get_diskfile_manager({\u0027devices\u0027: \u0027sdb2\u0027}, FakeLogger())"},{"line_number":1470,"context_line":"        self.assertEqual(\u0027sdb2\u0027, dfm2.devices)"},{"line_number":1471,"context_line":"        self.assertIsNot(dfm, dfm2)"},{"line_number":1472,"context_line":""},{"line_number":1473,"context_line":"    def test_get_diskfile_manager_custom_diskfile(self):"},{"line_number":1474,"context_line":"        calls \u003d []"}],"source_content_type":"text/x-python","patch_set":18,"id":"5f7c97a3_c3185ecf","line":1471,"range":{"start_line":1467,"start_character":0,"end_line":1471,"end_character":35},"updated":"2018-07-30 10:57:53.000000000","message":"actually, I\u0027m not sure this is nice for us because basically our swift is designed to define storage policy as a list (i.e. we could have more than 1 same storage policy instance) then, each policy will call get_diskfile_manager once at the DiskFileRouter initiation so we don\u0027t expect to call get_diskfile_manager several times for a policy isn\u0027t it?\n\nAnd if it can occur, perhaps caching the diskfile manager might be efficient at the perspective of resource consumption.\n\nCurrently it\u0027s ok we leave this as it is (not broken and if my understanding above was correct, we won\u0027t waste any resource because the diskfile manager is generate once per policy) tho.\n\nAny ideas?","commit_id":"4b850da4c9f600aaead6ccfc987c3481c21244fe"}],"test/unit/common/test_utils.py":[{"author":{"_account_id":4608,"name":"Kota Tsuyuzaki","email":"bloodeagle40234@gmail.com","username":"tsuyuzaki-kota"},"change_message_id":"c5757b11371bfc18eb0972b71bc1d40165264b63","unresolved":false,"context_lines":[{"line_number":4374,"context_line":"    @mock.patch(\u0027pkg_resources.load_entry_point\u0027)"},{"line_number":4375,"context_line":"    def test_load_pkg_resource(self, mock_driver):"},{"line_number":4376,"context_line":"        tests \u003d {"},{"line_number":4377,"context_line":"            (\u0027swift.diskfile_module\u0027, \u0027egg:swift#replication.fs\u0027):"},{"line_number":4378,"context_line":"                (\u0027swift\u0027, \u0027swift.diskfile_module\u0027, \u0027replication.fs\u0027),"},{"line_number":4379,"context_line":"            (\u0027swift.diskfile_module\u0027, \u0027egg:swift#erasure_coding.fs\u0027):"},{"line_number":4380,"context_line":"                (\u0027swift\u0027, \u0027swift.diskfile_module\u0027, \u0027erasure_coding.fs\u0027),"}],"source_content_type":"text/x-python","patch_set":18,"id":"5f7c97a3_3e4e096d","line":4377,"range":{"start_line":4377,"start_character":20,"end_line":4377,"end_character":36},"updated":"2018-07-30 10:57:53.000000000","message":"this should be an entry point group name so that swift.diskfile correctly.\n\nI don\u0027t know we could have the hardcoded entry point name but if other  packages follow the syntax (e.g. swift-on-file would have swift.diskfile entry point) it might not be a problem actually.","commit_id":"4b850da4c9f600aaead6ccfc987c3481c21244fe"}],"test/unit/obj/test_diskfile.py":[{"author":{"_account_id":4608,"name":"Kota Tsuyuzaki","email":"bloodeagle40234@gmail.com","username":"tsuyuzaki-kota"},"change_message_id":"c5757b11371bfc18eb0972b71bc1d40165264b63","unresolved":false,"context_lines":[{"line_number":684,"context_line":"        self.assertIn(\u0027Invalid diskfile replication.fs\u0027, str(cm.exception))"},{"line_number":685,"context_line":""},{"line_number":686,"context_line":"        bad_policy \u003d StoragePolicy(0, name\u003d\u0027zero\u0027, is_default\u003dTrue,"},{"line_number":687,"context_line":"                                   diskfile_module\u003d\u0027thin_air.fs\u0027)"},{"line_number":688,"context_line":""},{"line_number":689,"context_line":"        with patch_policies([bad_policy]):"},{"line_number":690,"context_line":"            with self.assertRaises(PolicyError) as cm:"}],"source_content_type":"text/x-python","patch_set":18,"id":"5f7c97a3_1e1a2d17","line":687,"updated":"2018-07-30 10:57:53.000000000","message":"Perhaps, listing all bad policy might be nice to notice operators?\n\ni.e. gathering error at Router initiation, then, if bad policy(ies) found, show all bad policy in the exception message.\n\nIt\u0027s ok to do in another patch tho.","commit_id":"4b850da4c9f600aaead6ccfc987c3481c21244fe"},{"author":{"_account_id":4608,"name":"Kota Tsuyuzaki","email":"bloodeagle40234@gmail.com","username":"tsuyuzaki-kota"},"change_message_id":"c5757b11371bfc18eb0972b71bc1d40165264b63","unresolved":false,"context_lines":[{"line_number":1836,"context_line":"            diskfile.DiskFileManager.check_policy(POLICIES[1])"},{"line_number":1837,"context_line":"        self.assertEqual(\u0027Invalid diskfile_module \u0027"},{"line_number":1838,"context_line":"                         \u0027\"egg:swift#erasure_coding.fs\" for policy_type \u0027"},{"line_number":1839,"context_line":"                         \u0027\"erasure_coding\"\u0027, str(cm.exception))"},{"line_number":1840,"context_line":""},{"line_number":1841,"context_line":""},{"line_number":1842,"context_line":"@patch_policies(with_ec_default\u003dTrue)"}],"source_content_type":"text/x-python","patch_set":18,"id":"5f7c97a3_d86b6d51","line":1839,"updated":"2018-07-30 10:57:53.000000000","message":"Curious. Why erasurecoding.fs is invalid for erasure_coding policy?\n\nOMG, it is because the message refers the default diskfile module at the policy so it\u0027s different from DiskFIleManager\u0027s entry point in this test case.","commit_id":"4b850da4c9f600aaead6ccfc987c3481c21244fe"},{"author":{"_account_id":4608,"name":"Kota Tsuyuzaki","email":"bloodeagle40234@gmail.com","username":"tsuyuzaki-kota"},"change_message_id":"c5757b11371bfc18eb0972b71bc1d40165264b63","unresolved":false,"context_lines":[{"line_number":3115,"context_line":"        with self.assertRaises(ValueError) as cm:"},{"line_number":3116,"context_line":"            diskfile.ECDiskFileManager.check_policy(POLICIES[0])"},{"line_number":3117,"context_line":"        self.assertEqual(\u0027Invalid diskfile_module \"egg:swift#replication.fs\" \u0027"},{"line_number":3118,"context_line":"                         \u0027for policy_type \"replication\"\u0027, str(cm.exception))"},{"line_number":3119,"context_line":""},{"line_number":3120,"context_line":""},{"line_number":3121,"context_line":"class DiskFileMixin(BaseDiskFileTestMixin):"}],"source_content_type":"text/x-python","patch_set":18,"id":"5f7c97a3_b8955140","line":3118,"updated":"2018-07-30 10:57:53.000000000","message":"Ditto","commit_id":"4b850da4c9f600aaead6ccfc987c3481c21244fe"}],"test/unit/obj/test_ssync.py":[{"author":{"_account_id":7847,"name":"Alistair Coles","email":"alistairncoles@gmail.com","username":"acoles"},"change_message_id":"0697c1b3aab241cb991ff6a5b42be7e569495e36","unresolved":false,"context_lines":[{"line_number":341,"context_line":"        tx_objs \u003d {}"},{"line_number":342,"context_line":"        rx_objs \u003d {}"},{"line_number":343,"context_line":"        tx_tombstones \u003d {}"},{"line_number":344,"context_line":"        policy.diskfile_manager \u003d None"},{"line_number":345,"context_line":"        policy.load_diskfile_manager(self.daemon.conf, self.daemon.logger)"},{"line_number":346,"context_line":"        tx_df_mgr \u003d policy.diskfile_manager"},{"line_number":347,"context_line":"        policy.diskfile_manager \u003d None"},{"line_number":348,"context_line":"        policy.load_diskfile_manager(self.rx_conf,"},{"line_number":349,"context_line":"                                     self.rx_logger)"},{"line_number":350,"context_line":"        rx_df_mgr \u003d policy.diskfile_manager"},{"line_number":351,"context_line":"        # o1 has primary and handoff fragment archives"},{"line_number":352,"context_line":"        t1 \u003d next(self.ts_iter)"},{"line_number":353,"context_line":"        tx_objs[\u0027o1\u0027] \u003d self._create_ondisk_files("},{"line_number":354,"context_line":"            tx_df_mgr, \u0027o1\u0027, policy, t1, (rx_node_index, tx_node_index))"}],"source_content_type":"text/x-python","patch_set":10,"id":"5f7c97a3_8d4e5036","line":351,"range":{"start_line":344,"start_character":0,"end_line":351,"end_character":1},"updated":"2018-06-07 16:29:05.000000000","message":"here I think is an example the loading of a single diskfile manager instance into a policy instance biting back - the tx and rx have different confs and before would have different DFM instances each with different confs, but now there is only one DFM instance per policy. So the test has to keep clearing the DFM in the policy to force it to reload with a instance :/ and the test still fails.","commit_id":"fd290c33fa3328a62400bc8f2b736330440ca8be"}]}