)]}'
{"/PATCHSET_LEVEL":[{"author":{"_account_id":25402,"name":"Francesco Pantano","email":"fpantano@redhat.com","username":"fmount"},"change_message_id":"d7893656c0bcccce21cf97726ed87b4055147d71","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":2,"id":"c7b2cd6f_c45f175a","updated":"2024-05-02 20:20:17.000000000","message":"Just to clarify something related to a context where the NFS share is mounted via the `Volume`/`VolumeMount` interface provided by kubernetes.\nI think it\u0027s clear enough that Glance won\u0027t start in case the share isn\u0027t ready (it might be related to missing rbac, connectivity issues and so forth).\nI tried to dig more into this, and after doing some testing I can confirm that if the NFS share is not available **after** Glance is started (e.g., it suddenly stop working), an upload operation will fail with the following error:\n \n```\nsh-5.1$ openstack image create --container-format bare --disk-format raw --file /tmp/cirros-0.5.2-x86_64-disk.img cirros\nConflictException: 409: Client Error for url: https://glance-default-public-openstack.apps-crc.testing/v2/images/0ce1f894-5af7-44fa-987d-f4c47c77d0cf/file, Conflict\n```\n\nLooking at the glanceAPI we can see the stack trace:\n\n```\n2024-05-02 18:33:41.868 20 INFO eventlet.wsgi.server [-] 10.217.0.2,::1 - - [02/May/2024 18:33:41] \"GET /healthcheck HTTP/1.1\" 200 142 0.001349\n2024-05-02 18:33:41.870 20 INFO eventlet.wsgi.server [-] 10.217.0.2,::1 - - [02/May/2024 18:33:41] \"GET /healthcheck HTTP/1.1\" 200 142 0.001243\n2024-05-02 18:33:49.829 20 ERROR glance.api.v2.image_data [None req-e17fc2f5-9c77-47b1-aa28-62a5a80cdafb 564f3a4d9c9e44d39cd9e44a14ca1f56 83532087594c4328bc7e93f0cd29726e - - default default] Image status transition from saving to saving is not allowed: glance.common.exception.InvalidImageStatusTransition: Image status transition from saving to saving is not allowed\n2024-05-02 18:33:49.829 20 ERROR glance.api.v2.image_data Traceback (most recent call last):\n2024-05-02 18:33:49.829 20 ERROR glance.api.v2.image_data   File \"/usr/lib/python3.9/site-packages/glance/api/v2/image_data.py\", line 139, in upload\n2024-05-02 18:33:49.829 20 ERROR glance.api.v2.image_data     image.status \u003d \u0027saving\u0027\n2024-05-02 18:33:49.829 20 ERROR glance.api.v2.image_data   File \"/usr/lib/python3.9/site-packages/glance/domain/proxy.py\", line 23, in set_attr\n2024-05-02 18:33:49.829 20 ERROR glance.api.v2.image_data     return setattr(getattr(self, target), attr, value)\n2024-05-02 18:33:49.829 20 ERROR glance.api.v2.image_data   File \"/usr/lib/python3.9/site-packages/glance/domain/proxy.py\", line 23, in set_attr\n2024-05-02 18:33:49.829 20 ERROR glance.api.v2.image_data     return setattr(getattr(self, target), attr, value)\n2024-05-02 18:33:49.829 20 ERROR glance.api.v2.image_data   File \"/usr/lib/python3.9/site-packages/glance/domain/proxy.py\", line 23, in set_attr\n2024-05-02 18:33:49.829 20 ERROR glance.api.v2.image_data     return setattr(getattr(self, target), attr, value)\n2024-05-02 18:33:49.829 20 ERROR glance.api.v2.image_data   [Previous line repeated 1 more time]\n2024-05-02 18:33:49.829 20 ERROR glance.api.v2.image_data   File \"/usr/lib/python3.9/site-packages/glance/domain/__init__.py\", line 157, in status\n2024-05-02 18:33:49.829 20 ERROR glance.api.v2.image_data     raise e\n2024-05-02 18:33:49.829 20 ERROR glance.api.v2.image_data glance.common.exception.InvalidImageStatusTransition: Image status transition from saving to saving is not allowed\n2024-05-02 18:33:49.829 20 ERROR glance.api.v2.image_data\n\n```\n\nEven though the `Pod` is still up, we can see both `liveness` and `readiness` probes failing:\n\n```\n  Normal   Started         12m                    kubelet            Started container glance-api\n  Warning  Unhealthy       5m24s (x2 over 9m24s)  kubelet            Liveness probe failed: Get \"https://10.217.0.247:9292/healthcheck\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)\n  Warning  Unhealthy       5m24s (x3 over 9m24s)  kubelet            Liveness probe failed: Get \"https://10.217.0.247:9292/healthcheck\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)\n  Warning  Unhealthy       5m24s                  kubelet            Readiness probe failed: Get \"https://10.217.0.247:9292/healthcheck\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)\n  Warning  Unhealthy       4m54s (x2 over 9m24s)  kubelet            Readiness probe failed: Get \"https://10.217.0.247:9292/healthcheck\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)\n  Warning  Unhealthy       4m54s                  kubelet            Readiness probe failed: Get \"https://10.217.0.247:9292/healthcheck\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)\n```\n\nand as a result the glance `Pods` are marked as `Unhealthy`.\nLater in time, according to the failure threshold set for the `Pod` [1], the `kubelet` marks the Pod as `Failed`, and we can see a failure, and given that the policy is supposed to recreate it:\n\n```\nglance-default-single-0                                         0/3     CreateContainerError   4 (3m39s ago)   28m\n```\n\n```\n$ oc describe pod glance-default-single-0 | tail\nNormal   Started    29m                    kubelet   Started container glance-api\nWarning  Unhealthy  10m (x3 over 26m)      kubelet   Readiness probe failed: Get \"https://10.217.0.247:9292/healthcheck\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)\nWarning  Unhealthy  10m                    kubelet   Liveness probe failed: Get \"https://10.217.0.247:9292/healthcheck\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)\nWarning  Unhealthy  10m                    kubelet   Readiness probe failed: Get \"https://10.217.0.247:9292/healthcheck\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nWarning  Unhealthy  9m30s (x4 over 26m)    kubelet   Liveness probe failed: Get \"https://10.217.0.247:9292/healthcheck\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)\nWarning  Unhealthy  9m30s (x5 over 26m)    kubelet   Liveness probe failed: Get \"https://10.217.0.247:9292/healthcheck\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)\nWarning  Unhealthy  9m30s (x2 over 22m)    kubelet   Readiness probe failed: Get \"https://10.217.0.247:9292/healthcheck\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)\nWarning  Unhealthy  9m30s (x3 over 22m)    kubelet   Readiness probe failed: Get \"https://10.217.0.247:9292/healthcheck\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)\nWarning  Unhealthy  9m30s                  kubelet   Liveness probe failed: Get \"https://10.217.0.247:9292/healthcheck\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\nWarning  Failed     4m47s (x2 over 6m48s)  kubelet   Error: context deadline exceeded\n ```\n\nWe can definitely state that if the `NFS` share is not available, the application\n(Glance) won\u0027t be able to upload any image in the fs local to the container, and according to the failure threshold, the Pod will be marked as failed and it fails to be recreated.\nI think we should clarify in the spec that this is going to cover use cases different from kubernetes (or at least optimize it with a better handling of the stack trace pasted above), and make sure the current behavior is preserved with the new implementation.\n\n[1] https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes","commit_id":"158ac627364485b553d76a19adaf355f21bc9ec7"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"25a83f4672869346a3f8d07142aa919709fe39b5","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":2,"id":"5cbc1e5d_1fc1be02","updated":"2024-05-03 06:18:29.000000000","message":"Thank you Eric and Francesco for inputs, made changes to the spec as per suggestion.","commit_id":"158ac627364485b553d76a19adaf355f21bc9ec7"},{"author":{"_account_id":27615,"name":"Rajat Dhasmana","email":"rajatdhasmana@gmail.com","username":"whoami-rajat"},"change_message_id":"0886ef5b61aec338ea3d70a6ba36189868265f34","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":2,"id":"a36466aa_0fcbdcec","updated":"2024-04-29 08:36:08.000000000","message":"one question inline.","commit_id":"158ac627364485b553d76a19adaf355f21bc9ec7"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"25a83f4672869346a3f8d07142aa919709fe39b5","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":2,"id":"0ae2995b_455b3a78","in_reply_to":"999dfec2_7dba05c3","updated":"2024-05-03 06:18:29.000000000","message":"Done","commit_id":"158ac627364485b553d76a19adaf355f21bc9ec7"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"5590dd491216921d59460ffca91c54375976d4bd","unresolved":true,"context_lines":[],"source_content_type":"","patch_set":2,"id":"999dfec2_7dba05c3","in_reply_to":"c7b2cd6f_c45f175a","updated":"2024-05-03 05:02:37.000000000","message":"Ack, I will mention this in a problem statement.","commit_id":"158ac627364485b553d76a19adaf355f21bc9ec7"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"754a37bd8c1cd64ba42ccea2aa967ac288611691","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"c0023028_b677001c","updated":"2024-05-15 05:18:20.000000000","message":"Hi Dan,\n\nThank you for inputs.","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"0e22231a5a691e670dfdd4b8be256d1e523d718f","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"274568d2_d04a9a88","updated":"2024-05-09 14:26:37.000000000","message":"It seems to me like there might be some better way to go about this than the NFS-specific (and perhaps not actually guaranteed) fsid\u003d0 check. That could be a marker file on disk perhaps, or something else we could check. That said, it seems to me like the proper way to do this is to have the healthcheck monitor and report the problem and *not* have glance return \"user at fault\" codes if the storage goes away.","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"2eeac230d198c15bbe3053b4aa9423111131aeb8","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"c752c136_4b54b2e5","updated":"2024-05-10 05:55:50.000000000","message":"Thank you Dan for review, please find inline replies for your comments.","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"7859bb6a85e44cd3aaaf3edff397a2b75bf4cb3f","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"b6a24d8f_b9ed0dc3","updated":"2024-05-17 06:18:21.000000000","message":"Thank you for inputs, made changes as per suggestion.","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":4523,"name":"Eric Harney","email":"eharney@redhat.com","username":"eharney"},"change_message_id":"9d33b36b087aca7d728acc6247a529abe4559480","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"9d07742b_36015d39","in_reply_to":"274568d2_d04a9a88","updated":"2024-05-10 14:04:48.000000000","message":"If we just detect whether fsid changes between its starting and current value, this can detect a change in the underlying FS, and this isn\u0027t NFS-specific.  Using a marker file is another way to accomplish this, too.","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"c1f3fdf4492e6bfefeefc290592aec0c77c28dcc","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":3,"id":"9952fe44_ff0d4c40","in_reply_to":"9d07742b_36015d39","updated":"2024-05-10 14:19:13.000000000","message":"Yes, but it can\u0027t tell the difference between two different NFS mounts and it assumes that it\u0027s right at startup. You could go unhealthy any time it changes, but that means if there was some delay between startup and mounting, you\u0027d constantly think it was becoming unhealthy when in reality it was the opposite.\n\nBut yeah, my primary point here is that I think we need to expose this via healthcheck and not by making image operations report that the user did something wrong. I\u0027d prefer something more stable and predictable like a marker, but the mechanism by which we report it is the important thing. If using fsid ends up too flaky, we could move to a marker file without changing the behavior.\n\nI will say, however, that nova now prefers a scenario where a uuid identity file is pre-provisioned in its own state directory by deployment tools for similar reasons. This is a similar arrangement so there is prior art here.","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"8d85ebadfbc9f933636e4fa8921c2fbe5c3c7b22","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":4,"id":"445e4977_8963fdb0","updated":"2024-05-17 13:45:35.000000000","message":"One comment about testing and a style nit, but otherwise looks good to me!","commit_id":"bfe7265b93bb2dda70bbf0b4fa3b95a0e2aeb15b"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"0f42646347c5e963da50285168e54f5db92e2b7e","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":4,"id":"44cd857c_3d221f42","updated":"2024-05-17 13:53:07.000000000","message":"Thank you for inputs Dan, will make changes and push a new revision shortly.","commit_id":"bfe7265b93bb2dda70bbf0b4fa3b95a0e2aeb15b"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"60a511ab3a97ed85cc6196e3cbba56179539c0a5","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":5,"id":"e3cc67df_4fde98ac","updated":"2024-05-20 16:10:28.000000000","message":"Oslo Middleware Implementation Patch: https://review.opendev.org/c/openstack/oslo.middleware/+/920055","commit_id":"af6b71401bcce7c505728b59e682131b806317ce"},{"author":{"_account_id":8122,"name":"Cyril Roelandt","email":"cyril@redhat.com","username":"cyril.roelandt.enovance"},"change_message_id":"a5c69f73ebe64cc5edab4c8346b1b301f2cbfe05","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":5,"id":"ccbfebe8_ca15e009","updated":"2024-05-20 15:37:10.000000000","message":"This looks really nice, but I have a few questions","commit_id":"af6b71401bcce7c505728b59e682131b806317ce"},{"author":{"_account_id":19138,"name":"Pranali Deore","email":"pdeore@redhat.com","username":"PranaliD"},"change_message_id":"a795b584d1bb44184bc28875190dfd7c52395b50","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":6,"id":"a4bb49a2_4f8a18f5","updated":"2024-06-11 14:15:49.000000000","message":"LGTM, Thank you Abhishek!\nApproving the spec as we have got minimum required approvals.","commit_id":"5940c59d44528b1af747d93c7b9cbf71a3463681"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"d7309e8494201319ebb69fae7d5cef19a6bd39e1","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":6,"id":"802a0313_5d040370","updated":"2024-05-20 16:19:45.000000000","message":"Making changes as per suggestion given by oslo team in next PS.\nThank you for review Dan and Cyril!","commit_id":"5940c59d44528b1af747d93c7b9cbf71a3463681"},{"author":{"_account_id":8122,"name":"Cyril Roelandt","email":"cyril@redhat.com","username":"cyril.roelandt.enovance"},"change_message_id":"a9c338badb112c5ed250a71d3c3eb8731c7e798d","unresolved":false,"context_lines":[],"source_content_type":"","patch_set":6,"id":"d9ecff0a_90dea846","updated":"2024-05-30 00:59:15.000000000","message":"Thanks Dan for answering my concerns, this looks good to me now!","commit_id":"5940c59d44528b1af747d93c7b9cbf71a3463681"}],"specs/2024.2/approved/glance/improve-filesystem-driver.rst":[{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"8d85ebadfbc9f933636e4fa8921c2fbe5c3c7b22","unresolved":true,"context_lines":[{"line_number":121,"context_line":"detect if the NFS share is unmounted from underneath the Glance service. We"},{"line_number":122,"context_line":"proposed to introduce below new configuration options for the same:"},{"line_number":123,"context_line":""},{"line_number":124,"context_line":"* \u0027filesystem_is_nfs_configured\u0027 - boolean, verify if NFS is configured or not"},{"line_number":125,"context_line":"* \u0027filesystem_nfs_host\u0027 - IP address of NFS server"},{"line_number":126,"context_line":"* \u0027filesystem_nfs_share_path\u0027 - Mount path of NFS mapped with local filesystem"},{"line_number":127,"context_line":"* \u0027filesystem_nfs_mount_options\u0027 - Mount options to be passed to NFS client"}],"source_content_type":"text/x-rst","patch_set":4,"id":"e825c1a5_30f34c34","line":124,"updated":"2024-05-17 13:45:35.000000000","message":"These config names should be in `backticks` for readability.","commit_id":"bfe7265b93bb2dda70bbf0b4fa3b95a0e2aeb15b"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"0f42646347c5e963da50285168e54f5db92e2b7e","unresolved":false,"context_lines":[{"line_number":121,"context_line":"detect if the NFS share is unmounted from underneath the Glance service. We"},{"line_number":122,"context_line":"proposed to introduce below new configuration options for the same:"},{"line_number":123,"context_line":""},{"line_number":124,"context_line":"* \u0027filesystem_is_nfs_configured\u0027 - boolean, verify if NFS is configured or not"},{"line_number":125,"context_line":"* \u0027filesystem_nfs_host\u0027 - IP address of NFS server"},{"line_number":126,"context_line":"* \u0027filesystem_nfs_share_path\u0027 - Mount path of NFS mapped with local filesystem"},{"line_number":127,"context_line":"* \u0027filesystem_nfs_mount_options\u0027 - Mount options to be passed to NFS client"}],"source_content_type":"text/x-rst","patch_set":4,"id":"9892fef2_3e5afa47","line":124,"in_reply_to":"e825c1a5_30f34c34","updated":"2024-05-17 13:53:07.000000000","message":"Acknowledged","commit_id":"bfe7265b93bb2dda70bbf0b4fa3b95a0e2aeb15b"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"8d85ebadfbc9f933636e4fa8921c2fbe5c3c7b22","unresolved":true,"context_lines":[{"line_number":216,"context_line":""},{"line_number":217,"context_line":"* Unit Tests"},{"line_number":218,"context_line":"* Functional Tests"},{"line_number":219,"context_line":"* Tempest Tests"},{"line_number":220,"context_line":""},{"line_number":221,"context_line":"Documentation Impact"},{"line_number":222,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":4,"id":"798f3574_6618729a","line":219,"updated":"2024-05-17 13:45:35.000000000","message":"Do we have any other tempest tests where we check the healthcheck? I think you\u0027ll be able to test the happy path, but not the case where the healthcheck fails (except maybe in whitebox if you want to write a test there).","commit_id":"bfe7265b93bb2dda70bbf0b4fa3b95a0e2aeb15b"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"0f42646347c5e963da50285168e54f5db92e2b7e","unresolved":true,"context_lines":[{"line_number":216,"context_line":""},{"line_number":217,"context_line":"* Unit Tests"},{"line_number":218,"context_line":"* Functional Tests"},{"line_number":219,"context_line":"* Tempest Tests"},{"line_number":220,"context_line":""},{"line_number":221,"context_line":"Documentation Impact"},{"line_number":222,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":4,"id":"c1435d5e_e8682d6f","line":219,"in_reply_to":"798f3574_6618729a","updated":"2024-05-17 13:53:07.000000000","message":"AFAIK, we don\u0027t have anything there at the moment. I will try to write a possible test for it.","commit_id":"bfe7265b93bb2dda70bbf0b4fa3b95a0e2aeb15b"},{"author":{"_account_id":8122,"name":"Cyril Roelandt","email":"cyril@redhat.com","username":"cyril.roelandt.enovance"},"change_message_id":"a5c69f73ebe64cc5edab4c8346b1b301f2cbfe05","unresolved":true,"context_lines":[{"line_number":14,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":15,"context_line":""},{"line_number":16,"context_line":"The filesystem backend of glance can be used to mount NFS share as local"},{"line_number":17,"context_line":"filesystem, so it does not required to store any special configs at"},{"line_number":18,"context_line":"glance side. Glance does not care about NFS server address or NFS share"},{"line_number":19,"context_line":"path at all, it just assumes that each image is stored in the local"},{"line_number":20,"context_line":"filesystem. The downside of this assumption is that glance is not"}],"source_content_type":"text/x-rst","patch_set":5,"id":"8b7bada4_6c46c1e3","line":17,"range":{"start_line":17,"start_character":35,"end_line":17,"end_character":36},"updated":"2024-05-20 15:37:10.000000000","message":"I think you meant \"it does not require\" (not \"requireD\" or \"it is not required\").","commit_id":"af6b71401bcce7c505728b59e682131b806317ce"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"d7309e8494201319ebb69fae7d5cef19a6bd39e1","unresolved":false,"context_lines":[{"line_number":14,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":15,"context_line":""},{"line_number":16,"context_line":"The filesystem backend of glance can be used to mount NFS share as local"},{"line_number":17,"context_line":"filesystem, so it does not required to store any special configs at"},{"line_number":18,"context_line":"glance side. Glance does not care about NFS server address or NFS share"},{"line_number":19,"context_line":"path at all, it just assumes that each image is stored in the local"},{"line_number":20,"context_line":"filesystem. The downside of this assumption is that glance is not"}],"source_content_type":"text/x-rst","patch_set":5,"id":"91811524_a0b7108f","line":17,"range":{"start_line":17,"start_character":35,"end_line":17,"end_character":36},"in_reply_to":"8b7bada4_6c46c1e3","updated":"2024-05-20 16:19:45.000000000","message":"Acknowledged","commit_id":"af6b71401bcce7c505728b59e682131b806317ce"},{"author":{"_account_id":8122,"name":"Cyril Roelandt","email":"cyril@redhat.com","username":"cyril.roelandt.enovance"},"change_message_id":"a5c69f73ebe64cc5edab4c8346b1b301f2cbfe05","unresolved":true,"context_lines":[{"line_number":71,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":72,"context_line":""},{"line_number":73,"context_line":"We are planning to add new plugin `disable_if_missing` to `healthcheck`"},{"line_number":74,"context_line":"wsgi middleware in `oslo.middleware` which can be used by all openstack"},{"line_number":75,"context_line":"components to check if desired path is not present then report"},{"line_number":76,"context_line":"`503 \u003cREASON\u003e` error or `200 OK` if everything is OK."},{"line_number":77,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"6c540d1a_0568e11d","line":74,"range":{"start_line":74,"start_character":25,"end_line":74,"end_character":35},"updated":"2024-05-20 15:37:10.000000000","message":"Has this been discussed with oslo cores?","commit_id":"af6b71401bcce7c505728b59e682131b806317ce"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"c42dc2dadf2679e9b22a3c4d55c1ae0942470bb1","unresolved":true,"context_lines":[{"line_number":71,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":72,"context_line":""},{"line_number":73,"context_line":"We are planning to add new plugin `disable_if_missing` to `healthcheck`"},{"line_number":74,"context_line":"wsgi middleware in `oslo.middleware` which can be used by all openstack"},{"line_number":75,"context_line":"components to check if desired path is not present then report"},{"line_number":76,"context_line":"`503 \u003cREASON\u003e` error or `200 OK` if everything is OK."},{"line_number":77,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"bddb4dc9_6488f1ce","line":74,"range":{"start_line":74,"start_character":25,"end_line":74,"end_character":35},"in_reply_to":"6c540d1a_0568e11d","updated":"2024-05-20 15:44:55.000000000","message":"Yep: https://review.opendev.org/c/openstack/oslo-specs/+/919963","commit_id":"af6b71401bcce7c505728b59e682131b806317ce"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"d7309e8494201319ebb69fae7d5cef19a6bd39e1","unresolved":false,"context_lines":[{"line_number":71,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":72,"context_line":""},{"line_number":73,"context_line":"We are planning to add new plugin `disable_if_missing` to `healthcheck`"},{"line_number":74,"context_line":"wsgi middleware in `oslo.middleware` which can be used by all openstack"},{"line_number":75,"context_line":"components to check if desired path is not present then report"},{"line_number":76,"context_line":"`503 \u003cREASON\u003e` error or `200 OK` if everything is OK."},{"line_number":77,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"bcb0c738_f3361358","line":74,"range":{"start_line":74,"start_character":25,"end_line":74,"end_character":35},"in_reply_to":"bddb4dc9_6488f1ce","updated":"2024-05-20 16:19:45.000000000","message":"Done","commit_id":"af6b71401bcce7c505728b59e682131b806317ce"},{"author":{"_account_id":8122,"name":"Cyril Roelandt","email":"cyril@redhat.com","username":"cyril.roelandt.enovance"},"change_message_id":"a5c69f73ebe64cc5edab4c8346b1b301f2cbfe05","unresolved":true,"context_lines":[{"line_number":84,"context_line":"  paste.app_factory \u003d oslo_middleware:Healthcheck.app_factory"},{"line_number":85,"context_line":"  backends \u003d disable_if_missing (optional, default: empty)"},{"line_number":86,"context_line":"  # used by the \u0027disable_if_missing\u0027 backend"},{"line_number":87,"context_line":"  available_file_paths \u003d /var/lib/glance/images,/var/lib/glance/cache (optional, default: empty)"},{"line_number":88,"context_line":""},{"line_number":89,"context_line":"  # Use this composite for keystone auth with caching and cache management"},{"line_number":90,"context_line":"  [composite:glance-api-keystone+cachemanagement]"}],"source_content_type":"text/x-rst","patch_set":5,"id":"a30bd45b_75b48815","line":87,"range":{"start_line":87,"start_character":41,"end_line":87,"end_character":47},"updated":"2024-05-20 15:37:10.000000000","message":"So, \"/var/lib/glance/images\" is the default for the \"filesystem_store_datadir\" option. Shouldn\u0027t we somehow use the value of this option as a default value for \"available_file_paths\"?","commit_id":"af6b71401bcce7c505728b59e682131b806317ce"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"c42dc2dadf2679e9b22a3c4d55c1ae0942470bb1","unresolved":true,"context_lines":[{"line_number":84,"context_line":"  paste.app_factory \u003d oslo_middleware:Healthcheck.app_factory"},{"line_number":85,"context_line":"  backends \u003d disable_if_missing (optional, default: empty)"},{"line_number":86,"context_line":"  # used by the \u0027disable_if_missing\u0027 backend"},{"line_number":87,"context_line":"  available_file_paths \u003d /var/lib/glance/images,/var/lib/glance/cache (optional, default: empty)"},{"line_number":88,"context_line":""},{"line_number":89,"context_line":"  # Use this composite for keystone auth with caching and cache management"},{"line_number":90,"context_line":"  [composite:glance-api-keystone+cachemanagement]"}],"source_content_type":"text/x-rst","patch_set":5,"id":"e0feb970_7871b58c","line":87,"range":{"start_line":87,"start_character":41,"end_line":87,"end_character":47},"in_reply_to":"a30bd45b_75b48815","updated":"2024-05-20 15:44:55.000000000","message":"I don\u0027t think we should because we don\u0027t know that it\u0027s NFS, and there could be multiple file stores, some of which we want (or don\u0027t) want monitoring for. I think this should be opt-in for specific paths if you want this behavior. There are probably plenty of scenarios where someone deploying glance will not want their container system to restart glance endlessly trying to fix a missing mount if they\u0027re not related. Like, if you setup the NFS mount as part of starting the container/pod, then restarting may help. If you don\u0027t, restarting won\u0027t do anything to help.","commit_id":"af6b71401bcce7c505728b59e682131b806317ce"},{"author":{"_account_id":8122,"name":"Cyril Roelandt","email":"cyril@redhat.com","username":"cyril.roelandt.enovance"},"change_message_id":"a5c69f73ebe64cc5edab4c8346b1b301f2cbfe05","unresolved":true,"context_lines":[{"line_number":99,"context_line":"\"oslo.middleware.healthcheck\"."},{"line_number":100,"context_line":""},{"line_number":101,"context_line":"In glance, if local filesystem path is mounted on NFS share then we"},{"line_number":102,"context_line":"propose to add one marker file named `.glance` to NFS share and then"},{"line_number":103,"context_line":"use that file path to configure `disable_if_missing` healthcheck"},{"line_number":104,"context_line":"middleware plugin as shown below:"},{"line_number":105,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"237f5314_0003c2a6","line":102,"range":{"start_line":102,"start_character":31,"end_line":102,"end_character":36},"updated":"2024-05-20 15:37:10.000000000","message":"OK, we\u0027ll need to document that admins must not delete that file from the share :)\n\nWho is going to be in charge of writing that file to the share?","commit_id":"af6b71401bcce7c505728b59e682131b806317ce"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"c42dc2dadf2679e9b22a3c4d55c1ae0942470bb1","unresolved":true,"context_lines":[{"line_number":99,"context_line":"\"oslo.middleware.healthcheck\"."},{"line_number":100,"context_line":""},{"line_number":101,"context_line":"In glance, if local filesystem path is mounted on NFS share then we"},{"line_number":102,"context_line":"propose to add one marker file named `.glance` to NFS share and then"},{"line_number":103,"context_line":"use that file path to configure `disable_if_missing` healthcheck"},{"line_number":104,"context_line":"middleware plugin as shown below:"},{"line_number":105,"context_line":""}],"source_content_type":"text/x-rst","patch_set":5,"id":"8588873f_1c2434c8","line":102,"range":{"start_line":102,"start_character":31,"end_line":102,"end_character":36},"in_reply_to":"237f5314_0003c2a6","updated":"2024-05-20 15:44:55.000000000","message":"The admin or deployment tool needs to do this, and yes, it needs to be documented of course. Glance can\u0027t automatically do it because it won\u0027t know when it\u0027s appropriate to write/create it. You certainly don\u0027t want it to always create if missing because it\u0027ll just end up writing it to the unmounted mount point directory and then defeat the whole point.\n\nSo, if you (or your deployment tooling) is enabling the feature, you\u0027d also create the file. The way Abhi has prescribed the oslo middleware feature, the admin can name the file whatever they want to minimize confusion.","commit_id":"af6b71401bcce7c505728b59e682131b806317ce"}],"specs/2024.2/approved/glance_store/improve-filesystem-driver.rst":[{"author":{"_account_id":27615,"name":"Rajat Dhasmana","email":"rajatdhasmana@gmail.com","username":"whoami-rajat"},"change_message_id":"0886ef5b61aec338ea3d70a6ba36189868265f34","unresolved":true,"context_lines":[{"line_number":17,"context_line":"filesystem, so it does not required to store any special configs at"},{"line_number":18,"context_line":"glance side. Glance does not care about NFS server address or NFS share"},{"line_number":19,"context_line":"path at all, it just assumes that each image is stored in the local"},{"line_number":20,"context_line":"filesystem. The downside of this assumption is that glance does not"},{"line_number":21,"context_line":"aware whether NFS server is connected/available or not, NFS share"},{"line_number":22,"context_line":"is mounted or not and just keeps performing add/delete operations"},{"line_number":23,"context_line":"on local filesystem directory which later might causes problem"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7c0cbbfd_fad4b0c8","line":20,"range":{"start_line":20,"start_character":59,"end_line":20,"end_character":63},"updated":"2024-04-29 08:36:08.000000000","message":"nit: is","commit_id":"158ac627364485b553d76a19adaf355f21bc9ec7"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"7859bb6a85e44cd3aaaf3edff397a2b75bf4cb3f","unresolved":false,"context_lines":[{"line_number":17,"context_line":"filesystem, so it does not required to store any special configs at"},{"line_number":18,"context_line":"glance side. Glance does not care about NFS server address or NFS share"},{"line_number":19,"context_line":"path at all, it just assumes that each image is stored in the local"},{"line_number":20,"context_line":"filesystem. The downside of this assumption is that glance does not"},{"line_number":21,"context_line":"aware whether NFS server is connected/available or not, NFS share"},{"line_number":22,"context_line":"is mounted or not and just keeps performing add/delete operations"},{"line_number":23,"context_line":"on local filesystem directory which later might causes problem"}],"source_content_type":"text/x-rst","patch_set":2,"id":"8bbfa569_31ab5f80","line":20,"range":{"start_line":20,"start_character":59,"end_line":20,"end_character":63},"in_reply_to":"7c0cbbfd_fad4b0c8","updated":"2024-05-17 06:18:21.000000000","message":"Done","commit_id":"158ac627364485b553d76a19adaf355f21bc9ec7"},{"author":{"_account_id":25402,"name":"Francesco Pantano","email":"fpantano@redhat.com","username":"fmount"},"change_message_id":"f67092bda588c9e3a14859db69adfda8e7891180","unresolved":true,"context_lines":[{"line_number":26,"context_line":"Proposed change"},{"line_number":27,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":28,"context_line":""},{"line_number":29,"context_line":"Introduce few configuration options for filesystem driver which will help to"},{"line_number":30,"context_line":"detect if the NFS share is unmounted from underneath the Glance service. We"},{"line_number":31,"context_line":"proposed to introduce below new configuration options for the same:"},{"line_number":32,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"a6d31cc2_5516514f","line":29,"updated":"2024-05-02 17:05:16.000000000","message":"Are these going to be optional for the filesystem driver?\ne.g. you can enable the `NFS` mode and do some additional check, but let also the `File` driver keep the old behavior so that a different entity (k8s) can take care about ensuring that NFS is always available (e.g. using liveness probes or similar mechanisms).","commit_id":"158ac627364485b553d76a19adaf355f21bc9ec7"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"5590dd491216921d59460ffca91c54375976d4bd","unresolved":true,"context_lines":[{"line_number":26,"context_line":"Proposed change"},{"line_number":27,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":28,"context_line":""},{"line_number":29,"context_line":"Introduce few configuration options for filesystem driver which will help to"},{"line_number":30,"context_line":"detect if the NFS share is unmounted from underneath the Glance service. We"},{"line_number":31,"context_line":"proposed to introduce below new configuration options for the same:"},{"line_number":32,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"f6861fec_6b0582d5","line":29,"in_reply_to":"a6d31cc2_5516514f","updated":"2024-05-03 05:02:37.000000000","message":"Yes, those will be optional for local filesystem driver.","commit_id":"158ac627364485b553d76a19adaf355f21bc9ec7"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"25a83f4672869346a3f8d07142aa919709fe39b5","unresolved":false,"context_lines":[{"line_number":26,"context_line":"Proposed change"},{"line_number":27,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"},{"line_number":28,"context_line":""},{"line_number":29,"context_line":"Introduce few configuration options for filesystem driver which will help to"},{"line_number":30,"context_line":"detect if the NFS share is unmounted from underneath the Glance service. We"},{"line_number":31,"context_line":"proposed to introduce below new configuration options for the same:"},{"line_number":32,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"431a658a_0628002b","line":29,"in_reply_to":"f6861fec_6b0582d5","updated":"2024-05-03 06:18:29.000000000","message":"Done","commit_id":"158ac627364485b553d76a19adaf355f21bc9ec7"},{"author":{"_account_id":4523,"name":"Eric Harney","email":"eharney@redhat.com","username":"eharney"},"change_message_id":"962cacc60927ae14d9037b42475a91d37529d959","unresolved":true,"context_lines":[{"line_number":29,"context_line":"Introduce few configuration options for filesystem driver which will help to"},{"line_number":30,"context_line":"detect if the NFS share is unmounted from underneath the Glance service. We"},{"line_number":31,"context_line":"proposed to introduce below new configuration options for the same:"},{"line_number":32,"context_line":""},{"line_number":33,"context_line":"* \u0027filesystem_is_nfs_configured\u0027 - boolean, verify if NFS is configured or not"},{"line_number":34,"context_line":"* \u0027filesystem_nfs_host\u0027 - IP address of NFS server"},{"line_number":35,"context_line":"* \u0027filesystem_nfs_share_path\u0027 - Mount path of NFS mapped with local filesystem"}],"source_content_type":"text/x-rst","patch_set":2,"id":"11a7e127_799894ae","line":32,"updated":"2024-04-29 13:09:05.000000000","message":"I don\u0027t think you need to add nfs config options to glance to achieve this.  You can call os.statvfs and use the f_fsid field to get the file system id.  Doing this at startup, then on subsequent operations, and comparing the two, will tell you if the FS is was unmounted.\n\n(This also has the benefit of not being NFS-specific.)","commit_id":"158ac627364485b553d76a19adaf355f21bc9ec7"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"5590dd491216921d59460ffca91c54375976d4bd","unresolved":false,"context_lines":[{"line_number":29,"context_line":"Introduce few configuration options for filesystem driver which will help to"},{"line_number":30,"context_line":"detect if the NFS share is unmounted from underneath the Glance service. We"},{"line_number":31,"context_line":"proposed to introduce below new configuration options for the same:"},{"line_number":32,"context_line":""},{"line_number":33,"context_line":"* \u0027filesystem_is_nfs_configured\u0027 - boolean, verify if NFS is configured or not"},{"line_number":34,"context_line":"* \u0027filesystem_nfs_host\u0027 - IP address of NFS server"},{"line_number":35,"context_line":"* \u0027filesystem_nfs_share_path\u0027 - Mount path of NFS mapped with local filesystem"}],"source_content_type":"text/x-rst","patch_set":2,"id":"52fad924_b1c20744","line":32,"in_reply_to":"07645db7_b6444843","updated":"2024-05-03 05:02:37.000000000","message":"Acknowledged","commit_id":"158ac627364485b553d76a19adaf355f21bc9ec7"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"9d6df45941416ef25810da196844cc004889e2f5","unresolved":true,"context_lines":[{"line_number":29,"context_line":"Introduce few configuration options for filesystem driver which will help to"},{"line_number":30,"context_line":"detect if the NFS share is unmounted from underneath the Glance service. We"},{"line_number":31,"context_line":"proposed to introduce below new configuration options for the same:"},{"line_number":32,"context_line":""},{"line_number":33,"context_line":"* \u0027filesystem_is_nfs_configured\u0027 - boolean, verify if NFS is configured or not"},{"line_number":34,"context_line":"* \u0027filesystem_nfs_host\u0027 - IP address of NFS server"},{"line_number":35,"context_line":"* \u0027filesystem_nfs_share_path\u0027 - Mount path of NFS mapped with local filesystem"}],"source_content_type":"text/x-rst","patch_set":2,"id":"a85cddd5_b340be8e","line":32,"in_reply_to":"11a7e127_799894ae","updated":"2024-04-29 13:58:29.000000000","message":"os.statvfs(path) does not return f_fsid.\n\nos.statvfs(\u0027/opt/stack/data/glance/images\u0027)\nos.statvfs_result(f_bsize\u003d1048576, f_frsize\u003d1048576, f_blocks\u003d158600, f_bfree\u003d147841, f_bavail\u003d147825, f_files\u003d20643840, f_ffree\u003d20361876, f_favail\u003d20361876, f_flag\u003d4096, f_namemax\u003d255)\n\nAm i missing something here?\n\ndf -h\n10.0.109.178:/mnt/nfsshare_glance  155G   11G  145G   7% /opt/stack/data/glance/images","commit_id":"158ac627364485b553d76a19adaf355f21bc9ec7"},{"author":{"_account_id":4523,"name":"Eric Harney","email":"eharney@redhat.com","username":"eharney"},"change_message_id":"b631d1ad61f84ce69866608d28c6526e0cd5a905","unresolved":true,"context_lines":[{"line_number":29,"context_line":"Introduce few configuration options for filesystem driver which will help to"},{"line_number":30,"context_line":"detect if the NFS share is unmounted from underneath the Glance service. We"},{"line_number":31,"context_line":"proposed to introduce below new configuration options for the same:"},{"line_number":32,"context_line":""},{"line_number":33,"context_line":"* \u0027filesystem_is_nfs_configured\u0027 - boolean, verify if NFS is configured or not"},{"line_number":34,"context_line":"* \u0027filesystem_nfs_host\u0027 - IP address of NFS server"},{"line_number":35,"context_line":"* \u0027filesystem_nfs_share_path\u0027 - Mount path of NFS mapped with local filesystem"}],"source_content_type":"text/x-rst","patch_set":2,"id":"ff4a93b3_96dd0e14","line":32,"in_reply_to":"1710012f_fe7c5d1b","updated":"2024-04-30 20:00:44.000000000","message":"This is a different thing and isn\u0027t needed, I\u0027m referring to the fsid that identifies any linux FS, not an NFS-specific field.\n\nThe os.statvfs result has it on Python 3.7+, but it isn\u0027t present in the string representation of the result:\n\n    \u003e\u003e\u003e import os\n    \u003e\u003e\u003e a \u003d os.statvfs(\u0027/home/eharney\u0027)\n    \u003e\u003e\u003e print(a)\n    os.statvfs_result(f_bsize\u003d4096, f_frsize\u003d4096, f_blocks\u003d82289991, f_bfree\u003d12370212, f_bavail\u003d8171812, f_files\u003d20971520, f_ffree\u003d16287191, f_favail\u003d16287191, f_flag\u003d4096, f_namemax\u003d255)\n    \u003e\u003e\u003e print(a.f_fsid)\n    11174108811353590924\n    \nhttps://docs.python.org/3/library/os.html#os.statvfs","commit_id":"158ac627364485b553d76a19adaf355f21bc9ec7"},{"author":{"_account_id":4523,"name":"Eric Harney","email":"eharney@redhat.com","username":"eharney"},"change_message_id":"44b0f9ae9e0c13689a7a000053c96603592f5339","unresolved":true,"context_lines":[{"line_number":29,"context_line":"Introduce few configuration options for filesystem driver which will help to"},{"line_number":30,"context_line":"detect if the NFS share is unmounted from underneath the Glance service. We"},{"line_number":31,"context_line":"proposed to introduce below new configuration options for the same:"},{"line_number":32,"context_line":""},{"line_number":33,"context_line":"* \u0027filesystem_is_nfs_configured\u0027 - boolean, verify if NFS is configured or not"},{"line_number":34,"context_line":"* \u0027filesystem_nfs_host\u0027 - IP address of NFS server"},{"line_number":35,"context_line":"* \u0027filesystem_nfs_share_path\u0027 - Mount path of NFS mapped with local filesystem"}],"source_content_type":"text/x-rst","patch_set":2,"id":"07645db7_b6444843","line":32,"in_reply_to":"7e59a080_466a259f","updated":"2024-05-02 13:41:26.000000000","message":"This is still enough to distinguish between a mounted /opt/stack/data/glance/images and an unmounted one, because it will still return different results in those two cases, right?","commit_id":"158ac627364485b553d76a19adaf355f21bc9ec7"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"a6fc624053f48e86b1fc51af703962255fc9ae4c","unresolved":true,"context_lines":[{"line_number":29,"context_line":"Introduce few configuration options for filesystem driver which will help to"},{"line_number":30,"context_line":"detect if the NFS share is unmounted from underneath the Glance service. We"},{"line_number":31,"context_line":"proposed to introduce below new configuration options for the same:"},{"line_number":32,"context_line":""},{"line_number":33,"context_line":"* \u0027filesystem_is_nfs_configured\u0027 - boolean, verify if NFS is configured or not"},{"line_number":34,"context_line":"* \u0027filesystem_nfs_host\u0027 - IP address of NFS server"},{"line_number":35,"context_line":"* \u0027filesystem_nfs_share_path\u0027 - Mount path of NFS mapped with local filesystem"}],"source_content_type":"text/x-rst","patch_set":2,"id":"1710012f_fe7c5d1b","line":32,"in_reply_to":"a85cddd5_b340be8e","updated":"2024-04-30 07:08:17.000000000","message":"After digging more about fsid, it needs to be defined in /etc/exports where you define access control for your share;\n\n/mnt/nfsshare_glance *(rw,no_root_squash)\n/mnt/example1 *(rw,fsid\u003d10001)\n\nIMO this will be backward incompatible change if some deployers haven\u0027t defined the fsid for their share in this file.","commit_id":"158ac627364485b553d76a19adaf355f21bc9ec7"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"bcec09244006a8a2c587a2bd16cfabc999c5ad24","unresolved":true,"context_lines":[{"line_number":29,"context_line":"Introduce few configuration options for filesystem driver which will help to"},{"line_number":30,"context_line":"detect if the NFS share is unmounted from underneath the Glance service. We"},{"line_number":31,"context_line":"proposed to introduce below new configuration options for the same:"},{"line_number":32,"context_line":""},{"line_number":33,"context_line":"* \u0027filesystem_is_nfs_configured\u0027 - boolean, verify if NFS is configured or not"},{"line_number":34,"context_line":"* \u0027filesystem_nfs_host\u0027 - IP address of NFS server"},{"line_number":35,"context_line":"* \u0027filesystem_nfs_share_path\u0027 - Mount path of NFS mapped with local filesystem"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7e59a080_466a259f","line":32,"in_reply_to":"ff4a93b3_96dd0e14","updated":"2024-05-01 03:46:01.000000000","message":"in my case it returns 0\nopt \u003d os.statvfs(\u0027/opt/stack/data/glance/images\u0027)\nprint(opt.f_fsid)\n0","commit_id":"158ac627364485b553d76a19adaf355f21bc9ec7"},{"author":{"_account_id":27615,"name":"Rajat Dhasmana","email":"rajatdhasmana@gmail.com","username":"whoami-rajat"},"change_message_id":"0886ef5b61aec338ea3d70a6ba36189868265f34","unresolved":true,"context_lines":[{"line_number":41,"context_line":"config options in glance-api.conf otherwise the respective glance store will"},{"line_number":42,"context_line":"be disabled and will not be used for any operation."},{"line_number":43,"context_line":""},{"line_number":44,"context_line":"We are planning to use existing os-brick library (already used by cinder driver"},{"line_number":45,"context_line":"of glance_store) to create the NFS client with the help of above configuration"},{"line_number":46,"context_line":"options and check if NFS share is available or not during service"},{"line_number":47,"context_line":"initialization as well as before each image upload/import/delete operation. If"},{"line_number":48,"context_line":"NFS share is not available during service initialization then add and delete"},{"line_number":49,"context_line":"operations will be disabled but if NFS goes down afterwards we will raise"},{"line_number":50,"context_line":"HTTP 410 (HTTP GONE) response to the user."}],"source_content_type":"text/x-rst","patch_set":2,"id":"af999459_d0340fdc","line":47,"range":{"start_line":44,"start_character":0,"end_line":47,"end_character":75},"updated":"2024-04-29 08:36:08.000000000","message":"os-brick does have a mechanism of checking if a device is mounted[1] and mount it if not already mounted[2].\nThe question I have is, will the NFS share be mounted always? is there a case where we would like to unmount the nfs share? since that functionality isn\u0027t provided by os-brick.\n\n[1] https://github.com/openstack/os-brick/blob/master/os_brick/remotefs/remotefs.py#L104-L106\n[2] https://github.com/openstack/os-brick/blob/master/os_brick/remotefs/remotefs.py#L110","commit_id":"158ac627364485b553d76a19adaf355f21bc9ec7"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"5590dd491216921d59460ffca91c54375976d4bd","unresolved":false,"context_lines":[{"line_number":41,"context_line":"config options in glance-api.conf otherwise the respective glance store will"},{"line_number":42,"context_line":"be disabled and will not be used for any operation."},{"line_number":43,"context_line":""},{"line_number":44,"context_line":"We are planning to use existing os-brick library (already used by cinder driver"},{"line_number":45,"context_line":"of glance_store) to create the NFS client with the help of above configuration"},{"line_number":46,"context_line":"options and check if NFS share is available or not during service"},{"line_number":47,"context_line":"initialization as well as before each image upload/import/delete operation. If"},{"line_number":48,"context_line":"NFS share is not available during service initialization then add and delete"},{"line_number":49,"context_line":"operations will be disabled but if NFS goes down afterwards we will raise"},{"line_number":50,"context_line":"HTTP 410 (HTTP GONE) response to the user."}],"source_content_type":"text/x-rst","patch_set":2,"id":"9a1e3f45_ca638890","line":47,"range":{"start_line":44,"start_character":0,"end_line":47,"end_character":75},"in_reply_to":"af999459_d0340fdc","updated":"2024-05-03 05:02:37.000000000","message":"Acknowledged","commit_id":"158ac627364485b553d76a19adaf355f21bc9ec7"},{"author":{"_account_id":25402,"name":"Francesco Pantano","email":"fpantano@redhat.com","username":"fmount"},"change_message_id":"f67092bda588c9e3a14859db69adfda8e7891180","unresolved":true,"context_lines":[{"line_number":88,"context_line":"Performance Impact"},{"line_number":89,"context_line":"------------------"},{"line_number":90,"context_line":""},{"line_number":91,"context_line":"Performance will have little impact since each add/delete operation now query"},{"line_number":92,"context_line":"NFS server to check whether it is reachable or not."},{"line_number":93,"context_line":""},{"line_number":94,"context_line":"Other deployer impact"}],"source_content_type":"text/x-rst","patch_set":2,"id":"cddbf587_e2dec27c","line":91,"updated":"2024-05-02 17:05:16.000000000","message":"Is","commit_id":"158ac627364485b553d76a19adaf355f21bc9ec7"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"5590dd491216921d59460ffca91c54375976d4bd","unresolved":false,"context_lines":[{"line_number":88,"context_line":"Performance Impact"},{"line_number":89,"context_line":"------------------"},{"line_number":90,"context_line":""},{"line_number":91,"context_line":"Performance will have little impact since each add/delete operation now query"},{"line_number":92,"context_line":"NFS server to check whether it is reachable or not."},{"line_number":93,"context_line":""},{"line_number":94,"context_line":"Other deployer impact"}],"source_content_type":"text/x-rst","patch_set":2,"id":"40c36e81_64e6f56e","line":91,"in_reply_to":"cddbf587_e2dec27c","updated":"2024-05-03 05:02:37.000000000","message":"Acknowledged","commit_id":"158ac627364485b553d76a19adaf355f21bc9ec7"},{"author":{"_account_id":4523,"name":"Eric Harney","email":"eharney@redhat.com","username":"eharney"},"change_message_id":"962cacc60927ae14d9037b42475a91d37529d959","unresolved":true,"context_lines":[{"line_number":89,"context_line":"------------------"},{"line_number":90,"context_line":""},{"line_number":91,"context_line":"Performance will have little impact since each add/delete operation now query"},{"line_number":92,"context_line":"NFS server to check whether it is reachable or not."},{"line_number":93,"context_line":""},{"line_number":94,"context_line":"Other deployer impact"},{"line_number":95,"context_line":"---------------------"}],"source_content_type":"text/x-rst","patch_set":2,"id":"a005480f_1a908b3d","line":92,"updated":"2024-04-29 13:09:05.000000000","message":"Perf impact would be trivial with the approach I mentioned above.","commit_id":"158ac627364485b553d76a19adaf355f21bc9ec7"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"5590dd491216921d59460ffca91c54375976d4bd","unresolved":false,"context_lines":[{"line_number":89,"context_line":"------------------"},{"line_number":90,"context_line":""},{"line_number":91,"context_line":"Performance will have little impact since each add/delete operation now query"},{"line_number":92,"context_line":"NFS server to check whether it is reachable or not."},{"line_number":93,"context_line":""},{"line_number":94,"context_line":"Other deployer impact"},{"line_number":95,"context_line":"---------------------"}],"source_content_type":"text/x-rst","patch_set":2,"id":"980c509b_70f535dd","line":92,"in_reply_to":"a005480f_1a908b3d","updated":"2024-05-03 05:02:37.000000000","message":"Acknowledged","commit_id":"158ac627364485b553d76a19adaf355f21bc9ec7"},{"author":{"_account_id":25402,"name":"Francesco Pantano","email":"fpantano@redhat.com","username":"fmount"},"change_message_id":"f67092bda588c9e3a14859db69adfda8e7891180","unresolved":true,"context_lines":[{"line_number":94,"context_line":"Other deployer impact"},{"line_number":95,"context_line":"---------------------"},{"line_number":96,"context_line":""},{"line_number":97,"context_line":"Deployer need to ensure that newly introduced configuration options are set"},{"line_number":98,"context_line":"in glance-api.conf file for filesystem store driver if NFS is used."},{"line_number":99,"context_line":""},{"line_number":100,"context_line":"Developer impact"}],"source_content_type":"text/x-rst","patch_set":2,"id":"55b96006_115c571b","line":97,"updated":"2024-05-02 17:05:16.000000000","message":"In a k8s related environment, I think we do not need those parameters in the config because Glance won\u0027t even start if the share is not available, so I assume this is required for a deployment in a context !\u003d k8s.","commit_id":"158ac627364485b553d76a19adaf355f21bc9ec7"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"5590dd491216921d59460ffca91c54375976d4bd","unresolved":false,"context_lines":[{"line_number":94,"context_line":"Other deployer impact"},{"line_number":95,"context_line":"---------------------"},{"line_number":96,"context_line":""},{"line_number":97,"context_line":"Deployer need to ensure that newly introduced configuration options are set"},{"line_number":98,"context_line":"in glance-api.conf file for filesystem store driver if NFS is used."},{"line_number":99,"context_line":""},{"line_number":100,"context_line":"Developer impact"}],"source_content_type":"text/x-rst","patch_set":2,"id":"7d400e06_178f149a","line":97,"in_reply_to":"55b96006_115c571b","updated":"2024-05-03 05:02:37.000000000","message":"Acknowledged","commit_id":"158ac627364485b553d76a19adaf355f21bc9ec7"},{"author":{"_account_id":25402,"name":"Francesco Pantano","email":"fpantano@redhat.com","username":"fmount"},"change_message_id":"f67092bda588c9e3a14859db69adfda8e7891180","unresolved":true,"context_lines":[{"line_number":119,"context_line":""},{"line_number":120,"context_line":"* Introduce new configuration options"},{"line_number":121,"context_line":""},{"line_number":122,"context_line":"* Use os-brick library to detect if the NFS share is available"},{"line_number":123,"context_line":""},{"line_number":124,"context_line":"* Unit/Functional tests for coverage"},{"line_number":125,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"eb03f4c8_ad333510","line":122,"updated":"2024-05-02 17:05:16.000000000","message":"is the idea to adopt the same cinder mechanism for NFS?","commit_id":"158ac627364485b553d76a19adaf355f21bc9ec7"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"25a83f4672869346a3f8d07142aa919709fe39b5","unresolved":false,"context_lines":[{"line_number":119,"context_line":""},{"line_number":120,"context_line":"* Introduce new configuration options"},{"line_number":121,"context_line":""},{"line_number":122,"context_line":"* Use os-brick library to detect if the NFS share is available"},{"line_number":123,"context_line":""},{"line_number":124,"context_line":"* Unit/Functional tests for coverage"},{"line_number":125,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"34e9e4af_9d4d6169","line":122,"in_reply_to":"22985ef4_21c0b9ce","updated":"2024-05-03 06:18:29.000000000","message":"Done","commit_id":"158ac627364485b553d76a19adaf355f21bc9ec7"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"5590dd491216921d59460ffca91c54375976d4bd","unresolved":true,"context_lines":[{"line_number":119,"context_line":""},{"line_number":120,"context_line":"* Introduce new configuration options"},{"line_number":121,"context_line":""},{"line_number":122,"context_line":"* Use os-brick library to detect if the NFS share is available"},{"line_number":123,"context_line":""},{"line_number":124,"context_line":"* Unit/Functional tests for coverage"},{"line_number":125,"context_line":""}],"source_content_type":"text/x-rst","patch_set":2,"id":"22985ef4_21c0b9ce","line":122,"in_reply_to":"eb03f4c8_ad333510","updated":"2024-05-03 05:02:37.000000000","message":"Likely!","commit_id":"158ac627364485b553d76a19adaf355f21bc9ec7"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"0e22231a5a691e670dfdd4b8be256d1e523d718f","unresolved":true,"context_lines":[{"line_number":33,"context_line":"    ConflictException: 409: Client Error for url: https://glance-default-public-openstack.apps-crc.testing/v2/images/0ce1f894-5af7-44fa-987d-f4c47c77d0cf/file, Conflict"},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"Even though the Glance Pod is still up, `liveness` and `readiness` probes"},{"line_number":36,"context_line":"starts failing and as a result the Glance Pods are marked as `Unhealthy`::"},{"line_number":37,"context_line":""},{"line_number":38,"context_line":"    Normal   Started         12m                    kubelet            Started container glance-api"},{"line_number":39,"context_line":"      Warning  Unhealthy       5m24s (x2 over 9m24s)  kubelet            Liveness probe failed: Get \"https://10.217.0.247:9292/healthcheck\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"}],"source_content_type":"text/x-rst","patch_set":3,"id":"4d6780cd_b716f560","line":36,"updated":"2024-05-09 14:26:37.000000000","message":"Okay, but we control the healthcheck\u0027s behavior. I haven\u0027t gone looking for the healthcheck implementation in glance, but what is it checking currently?","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"0edfac0f064c738fa20008d3c31111113a34f079","unresolved":true,"context_lines":[{"line_number":33,"context_line":"    ConflictException: 409: Client Error for url: https://glance-default-public-openstack.apps-crc.testing/v2/images/0ce1f894-5af7-44fa-987d-f4c47c77d0cf/file, Conflict"},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"Even though the Glance Pod is still up, `liveness` and `readiness` probes"},{"line_number":36,"context_line":"starts failing and as a result the Glance Pods are marked as `Unhealthy`::"},{"line_number":37,"context_line":""},{"line_number":38,"context_line":"    Normal   Started         12m                    kubelet            Started container glance-api"},{"line_number":39,"context_line":"      Warning  Unhealthy       5m24s (x2 over 9m24s)  kubelet            Liveness probe failed: Get \"https://10.217.0.247:9292/healthcheck\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"}],"source_content_type":"text/x-rst","patch_set":3,"id":"df6286a7_fd1cec19","line":36,"in_reply_to":"01d8942d_fe4d3534","updated":"2024-05-10 14:00:23.000000000","message":"Based on the answer(s) below, yes. If you go the file marker route, a very simple \"check for existence of file means healthy\" plugin added to oslo could make an easy thing for any service that has similar needs to glance.","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"2eeac230d198c15bbe3053b4aa9423111131aeb8","unresolved":true,"context_lines":[{"line_number":33,"context_line":"    ConflictException: 409: Client Error for url: https://glance-default-public-openstack.apps-crc.testing/v2/images/0ce1f894-5af7-44fa-987d-f4c47c77d0cf/file, Conflict"},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"Even though the Glance Pod is still up, `liveness` and `readiness` probes"},{"line_number":36,"context_line":"starts failing and as a result the Glance Pods are marked as `Unhealthy`::"},{"line_number":37,"context_line":""},{"line_number":38,"context_line":"    Normal   Started         12m                    kubelet            Started container glance-api"},{"line_number":39,"context_line":"      Warning  Unhealthy       5m24s (x2 over 9m24s)  kubelet            Liveness probe failed: Get \"https://10.217.0.247:9292/healthcheck\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"}],"source_content_type":"text/x-rst","patch_set":3,"id":"01d8942d_fe4d3534","line":36,"in_reply_to":"4d6780cd_b716f560","updated":"2024-05-10 05:55:50.000000000","message":"AFAIK, healthcheck middleware can check if specified disabled path exists or not [1], and glance doesn\u0027t have its own implementation to override custom behavior.\n\n[1] https://github.com/openstack/oslo.middleware/blob/master/oslo_middleware/healthcheck/disable_by_file.py#L82\n\n\nSo are you suggesting we can define our own plugin using helathcheck middleware for this purpose?","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"7859bb6a85e44cd3aaaf3edff397a2b75bf4cb3f","unresolved":false,"context_lines":[{"line_number":33,"context_line":"    ConflictException: 409: Client Error for url: https://glance-default-public-openstack.apps-crc.testing/v2/images/0ce1f894-5af7-44fa-987d-f4c47c77d0cf/file, Conflict"},{"line_number":34,"context_line":""},{"line_number":35,"context_line":"Even though the Glance Pod is still up, `liveness` and `readiness` probes"},{"line_number":36,"context_line":"starts failing and as a result the Glance Pods are marked as `Unhealthy`::"},{"line_number":37,"context_line":""},{"line_number":38,"context_line":"    Normal   Started         12m                    kubelet            Started container glance-api"},{"line_number":39,"context_line":"      Warning  Unhealthy       5m24s (x2 over 9m24s)  kubelet            Liveness probe failed: Get \"https://10.217.0.247:9292/healthcheck\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"}],"source_content_type":"text/x-rst","patch_set":3,"id":"09202088_b8f786f4","line":36,"in_reply_to":"df6286a7_fd1cec19","updated":"2024-05-17 06:18:21.000000000","message":"Acknowledged","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"0e22231a5a691e670dfdd4b8be256d1e523d718f","unresolved":true,"context_lines":[{"line_number":65,"context_line":"from local filesystem. In this case we can definitely say that NFS share is"},{"line_number":66,"context_line":"not available, the Glance won\u0027t be able to upload any image in the"},{"line_number":67,"context_line":"filesystem local to the container and the Pod will be marked as failed and"},{"line_number":68,"context_line":"it fails to be recreated."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"Proposed change"},{"line_number":71,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":3,"id":"ea75c3de_b5807e15","line":68,"updated":"2024-05-09 14:26:37.000000000","message":"I\u0027m a bit confused about what you want. You want glance to refuse to start if its mount is not available, right? I think it seems like it should also go to \"unhealthy\" state if the mount disappears, right?","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"0edfac0f064c738fa20008d3c31111113a34f079","unresolved":true,"context_lines":[{"line_number":65,"context_line":"from local filesystem. In this case we can definitely say that NFS share is"},{"line_number":66,"context_line":"not available, the Glance won\u0027t be able to upload any image in the"},{"line_number":67,"context_line":"filesystem local to the container and the Pod will be marked as failed and"},{"line_number":68,"context_line":"it fails to be recreated."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"Proposed change"},{"line_number":71,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":3,"id":"f35fb2d1_7bd1837b","line":68,"in_reply_to":"0064c1a7_9eee1658","updated":"2024-05-10 14:00:23.000000000","message":"That\u0027s the desired behavior, not the current behavior, right?","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"c5b02a12850e4533c38c658f66c1ec32e51bf4c9","unresolved":true,"context_lines":[{"line_number":65,"context_line":"from local filesystem. In this case we can definitely say that NFS share is"},{"line_number":66,"context_line":"not available, the Glance won\u0027t be able to upload any image in the"},{"line_number":67,"context_line":"filesystem local to the container and the Pod will be marked as failed and"},{"line_number":68,"context_line":"it fails to be recreated."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"Proposed change"},{"line_number":71,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":3,"id":"d9702bc9_e5aa943f","line":68,"in_reply_to":"3de07db8_4780f954","updated":"2024-05-10 16:50:55.000000000","message":"How does the healthcheck currently know/report unhealthy if the mount goes away?\n\nIf that\u0027s actually currently working, then I guess I\u0027m not sure what the problem is - that seems like the ideal situation to me:\n\n - Mount available \u003d\u003d healthy\n - Mount not available \u003d\u003d unhealthy","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"7859bb6a85e44cd3aaaf3edff397a2b75bf4cb3f","unresolved":false,"context_lines":[{"line_number":65,"context_line":"from local filesystem. In this case we can definitely say that NFS share is"},{"line_number":66,"context_line":"not available, the Glance won\u0027t be able to upload any image in the"},{"line_number":67,"context_line":"filesystem local to the container and the Pod will be marked as failed and"},{"line_number":68,"context_line":"it fails to be recreated."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"Proposed change"},{"line_number":71,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":3,"id":"23f7b61b_b617abc1","line":68,"in_reply_to":"d9702bc9_e5aa943f","updated":"2024-05-17 06:18:21.000000000","message":"Acknowledged","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"2eeac230d198c15bbe3053b4aa9423111131aeb8","unresolved":true,"context_lines":[{"line_number":65,"context_line":"from local filesystem. In this case we can definitely say that NFS share is"},{"line_number":66,"context_line":"not available, the Glance won\u0027t be able to upload any image in the"},{"line_number":67,"context_line":"filesystem local to the container and the Pod will be marked as failed and"},{"line_number":68,"context_line":"it fails to be recreated."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"Proposed change"},{"line_number":71,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":3,"id":"0064c1a7_9eee1658","line":68,"in_reply_to":"ea75c3de_b5807e15","updated":"2024-05-10 05:55:50.000000000","message":"So, in this (k8s) deployment, while starting the service if NFS is available, service starts and reports healthy, but after that if mount disappears due to some unknown reasons it goes to unhealthy state.","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"11eeb5dbfdb1349e1e0266ea920914468c39b6ab","unresolved":true,"context_lines":[{"line_number":65,"context_line":"from local filesystem. In this case we can definitely say that NFS share is"},{"line_number":66,"context_line":"not available, the Glance won\u0027t be able to upload any image in the"},{"line_number":67,"context_line":"filesystem local to the container and the Pod will be marked as failed and"},{"line_number":68,"context_line":"it fails to be recreated."},{"line_number":69,"context_line":""},{"line_number":70,"context_line":"Proposed change"},{"line_number":71,"context_line":"\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d"}],"source_content_type":"text/x-rst","patch_set":3,"id":"3de07db8_4780f954","line":68,"in_reply_to":"f35fb2d1_7bd1837b","updated":"2024-05-10 15:31:20.000000000","message":"NO, this is the current behavior","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"0e22231a5a691e670dfdd4b8be256d1e523d718f","unresolved":true,"context_lines":[{"line_number":74,"context_line":"the `f_fsid` attribute at the start of the glance-api service when"},{"line_number":75,"context_line":"filesystem store is initialized. In case of local directory is mounted"},{"line_number":76,"context_line":"to NFS share then `statvfs(path)` function returns Zero otherwise it"},{"line_number":77,"context_line":"will return Non Zero value for the same."},{"line_number":78,"context_line":""},{"line_number":79,"context_line":"For example, if local FS `/opt/stack/data/glance/images` is mounted"},{"line_number":80,"context_line":"to NFS share::"}],"source_content_type":"text/x-rst","patch_set":3,"id":"3d6dc77b_971bfeae","line":77,"updated":"2024-05-09 14:26:37.000000000","message":"I see that it does (checked locally) but I don\u0027t see anything that says that\u0027s a hard and fast rule. NFSv4 lets you set (actually requires you to set) an fsid for the root of a multi-tree export so that it can tell the children apart from the parents. I don\u0027t know if that makes its way into the statvfs output or not.\n\nIf we\u0027re going to rely on this, I think we should have some documentation we can point to.\n\nOther than that, why not do something specific to nfs and just check the mount table? Also, presumably the same behavior could happen with any other mounted filesystem, even a local one, or another type of network one. Seems like this is too nfs-specific to me.","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"c5b02a12850e4533c38c658f66c1ec32e51bf4c9","unresolved":true,"context_lines":[{"line_number":74,"context_line":"the `f_fsid` attribute at the start of the glance-api service when"},{"line_number":75,"context_line":"filesystem store is initialized. In case of local directory is mounted"},{"line_number":76,"context_line":"to NFS share then `statvfs(path)` function returns Zero otherwise it"},{"line_number":77,"context_line":"will return Non Zero value for the same."},{"line_number":78,"context_line":""},{"line_number":79,"context_line":"For example, if local FS `/opt/stack/data/glance/images` is mounted"},{"line_number":80,"context_line":"to NFS share::"}],"source_content_type":"text/x-rst","patch_set":3,"id":"3079603f_3b56d8e6","line":77,"in_reply_to":"008ede29_9b547e9b","updated":"2024-05-10 16:50:55.000000000","message":"I think you want the opposite of the disable-by-file, but that should be pretty trivial. You want \"healthy if file is present\" not \"unhealthy if file present\" which is what the current plugin does, AFAICT:\n\nhttps://github.com/openstack/oslo.middleware/blob/master/oslo_middleware/healthcheck/disable_by_file.py#L115","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"836812903a6a5f78ea89645d9e2206c76ecaf960","unresolved":true,"context_lines":[{"line_number":74,"context_line":"the `f_fsid` attribute at the start of the glance-api service when"},{"line_number":75,"context_line":"filesystem store is initialized. In case of local directory is mounted"},{"line_number":76,"context_line":"to NFS share then `statvfs(path)` function returns Zero otherwise it"},{"line_number":77,"context_line":"will return Non Zero value for the same."},{"line_number":78,"context_line":""},{"line_number":79,"context_line":"For example, if local FS `/opt/stack/data/glance/images` is mounted"},{"line_number":80,"context_line":"to NFS share::"}],"source_content_type":"text/x-rst","patch_set":3,"id":"d02e2619_8daa01f0","line":77,"in_reply_to":"291d1091_beb9d986","updated":"2024-05-14 13:51:53.000000000","message":"Yeah, this is how healthcheck should behave - not a filter, but an application (or completely sepaprate API). It should be distinct from the regular API, and it has a different consumer. The healthcheck is used by k8s, docker, or something like systemd.  It\u0027s often even a different port, bound to localhost, etc.","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"3549b769e1cbda39cd8ba8a1d1744752b4855edd","unresolved":true,"context_lines":[{"line_number":74,"context_line":"the `f_fsid` attribute at the start of the glance-api service when"},{"line_number":75,"context_line":"filesystem store is initialized. In case of local directory is mounted"},{"line_number":76,"context_line":"to NFS share then `statvfs(path)` function returns Zero otherwise it"},{"line_number":77,"context_line":"will return Non Zero value for the same."},{"line_number":78,"context_line":""},{"line_number":79,"context_line":"For example, if local FS `/opt/stack/data/glance/images` is mounted"},{"line_number":80,"context_line":"to NFS share::"}],"source_content_type":"text/x-rst","patch_set":3,"id":"291d1091_beb9d986","line":77,"in_reply_to":"3079603f_3b56d8e6","updated":"2024-05-14 05:08:41.000000000","message":"AFAIK, with recent changes in oslo_middleware we need to configure healthcheck middleware as an application and not as filter, which means we will not be able to get desired result for each API call. \n\nSince this is now required to configure as an application we need a cron job which will hit the http://ip/image/healthcheck request and monitor the response for failure and notify admin about the failure, but it will not have any impact on service (i.e. we can restrict service from handling other api calls).","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"2eeac230d198c15bbe3053b4aa9423111131aeb8","unresolved":true,"context_lines":[{"line_number":74,"context_line":"the `f_fsid` attribute at the start of the glance-api service when"},{"line_number":75,"context_line":"filesystem store is initialized. In case of local directory is mounted"},{"line_number":76,"context_line":"to NFS share then `statvfs(path)` function returns Zero otherwise it"},{"line_number":77,"context_line":"will return Non Zero value for the same."},{"line_number":78,"context_line":""},{"line_number":79,"context_line":"For example, if local FS `/opt/stack/data/glance/images` is mounted"},{"line_number":80,"context_line":"to NFS share::"}],"source_content_type":"text/x-rst","patch_set":3,"id":"e9ef2ffb_0b99be67","line":77,"in_reply_to":"3d6dc77b_971bfeae","updated":"2024-05-10 05:55:50.000000000","message":"In short are you suggesting to use os.path.ismount() method or explicitly read /proc/mount and check mount path in it?","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"11eeb5dbfdb1349e1e0266ea920914468c39b6ab","unresolved":true,"context_lines":[{"line_number":74,"context_line":"the `f_fsid` attribute at the start of the glance-api service when"},{"line_number":75,"context_line":"filesystem store is initialized. In case of local directory is mounted"},{"line_number":76,"context_line":"to NFS share then `statvfs(path)` function returns Zero otherwise it"},{"line_number":77,"context_line":"will return Non Zero value for the same."},{"line_number":78,"context_line":""},{"line_number":79,"context_line":"For example, if local FS `/opt/stack/data/glance/images` is mounted"},{"line_number":80,"context_line":"to NFS share::"}],"source_content_type":"text/x-rst","patch_set":3,"id":"008ede29_9b547e9b","line":77,"in_reply_to":"59c2f8f9_2f32b447","updated":"2024-05-10 15:31:20.000000000","message":"marker file sounds good and we can override existing disable_by_file healthcheck plugin for the same.","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"754a37bd8c1cd64ba42ccea2aa967ac288611691","unresolved":true,"context_lines":[{"line_number":74,"context_line":"the `f_fsid` attribute at the start of the glance-api service when"},{"line_number":75,"context_line":"filesystem store is initialized. In case of local directory is mounted"},{"line_number":76,"context_line":"to NFS share then `statvfs(path)` function returns Zero otherwise it"},{"line_number":77,"context_line":"will return Non Zero value for the same."},{"line_number":78,"context_line":""},{"line_number":79,"context_line":"For example, if local FS `/opt/stack/data/glance/images` is mounted"},{"line_number":80,"context_line":"to NFS share::"}],"source_content_type":"text/x-rst","patch_set":3,"id":"ec0b6b86_a8754680","line":77,"in_reply_to":"d02e2619_8daa01f0","updated":"2024-05-15 05:18:20.000000000","message":"Ack, I have little confusion here, if possible for you we can discuss this in weekly meeting or upstream glance irc channel.","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"0edfac0f064c738fa20008d3c31111113a34f079","unresolved":true,"context_lines":[{"line_number":74,"context_line":"the `f_fsid` attribute at the start of the glance-api service when"},{"line_number":75,"context_line":"filesystem store is initialized. In case of local directory is mounted"},{"line_number":76,"context_line":"to NFS share then `statvfs(path)` function returns Zero otherwise it"},{"line_number":77,"context_line":"will return Non Zero value for the same."},{"line_number":78,"context_line":""},{"line_number":79,"context_line":"For example, if local FS `/opt/stack/data/glance/images` is mounted"},{"line_number":80,"context_line":"to NFS share::"}],"source_content_type":"text/x-rst","patch_set":3,"id":"59c2f8f9_2f32b447","line":77,"in_reply_to":"e9ef2ffb_0b99be67","updated":"2024-05-10 14:00:23.000000000","message":"Well, I don\u0027t really think you can *just* do either that or what you\u0027re proposing here. I think you need some sort of config from the user to know their intent. This seems to imply that it will always be started with the proper arrangement, but the container could just as easily be started without NFS available and then it will assume that no such mount is expected. There might even be a race between setup of the pod and the nfs mount and the glance startup check to determine the canonical disk config.\n\nThe reason containers don\u0027t *start* healthy automatically is so that they can survey the environment and then go from \"starting\" to \"healthy\" (or whatever the startup state is) after they determine that things are good. So I really think you need some \"intent\" from the operator other than just \"my dir was a mount when I started so that must be correct.\"\n\nI\u0027m not really sure what to suggest here, other than some config that says something very specific about the mount that we can check. Just checking \"ismount\" will probably return true for almost every container environment, since in almost all cases glance\u0027s images directory will at *least* be bind-mounted from the host.\n\nI think maybe something better might be a config knob that tells glance to look for some \"marker file\" in the images directory. Something (kinda like a tombstone) that should always be in the images directory to know it\u0027s the images directory. Something as simple as `/var/lib/glance/images/GLANCE` or `/var/lib/glance/images/.glance`. Then if that file is there (which much be provisioned the first time) we know we\u0027re looking at our images dir, regardless of where/how it\u0027s mounted.","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"7859bb6a85e44cd3aaaf3edff397a2b75bf4cb3f","unresolved":false,"context_lines":[{"line_number":74,"context_line":"the `f_fsid` attribute at the start of the glance-api service when"},{"line_number":75,"context_line":"filesystem store is initialized. In case of local directory is mounted"},{"line_number":76,"context_line":"to NFS share then `statvfs(path)` function returns Zero otherwise it"},{"line_number":77,"context_line":"will return Non Zero value for the same."},{"line_number":78,"context_line":""},{"line_number":79,"context_line":"For example, if local FS `/opt/stack/data/glance/images` is mounted"},{"line_number":80,"context_line":"to NFS share::"}],"source_content_type":"text/x-rst","patch_set":3,"id":"156b3de2_1c201fb9","line":77,"in_reply_to":"ec0b6b86_a8754680","updated":"2024-05-17 06:18:21.000000000","message":"Done","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"c0acd38b0c6360d565145a51e74a1d0eb84a7e39","unresolved":true,"context_lines":[{"line_number":99,"context_line":"assume that NFS share is configured for glance. We will again retrieve this"},{"line_number":100,"context_line":"value while adding image data to filesystem store or deleting the data from"},{"line_number":101,"context_line":"the filesystem store. If newly retrieved value is Non Zero then we will"},{"line_number":102,"context_line":"simply abort the operation and return HTTP 400 to end user."},{"line_number":103,"context_line":""},{"line_number":104,"context_line":"If the `f_fsid` is found Non Zero at service start then we will ignore it"},{"line_number":105,"context_line":"considering local filesystem is used for Glance storage."}],"source_content_type":"text/x-rst","patch_set":3,"id":"cf7b5b55_0a2f432a","line":102,"range":{"start_line":102,"start_character":38,"end_line":102,"end_character":46},"updated":"2024-05-06 14:03:15.000000000","message":"Should it be HTTP 410 HTTP GONE","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"7859bb6a85e44cd3aaaf3edff397a2b75bf4cb3f","unresolved":false,"context_lines":[{"line_number":99,"context_line":"assume that NFS share is configured for glance. We will again retrieve this"},{"line_number":100,"context_line":"value while adding image data to filesystem store or deleting the data from"},{"line_number":101,"context_line":"the filesystem store. If newly retrieved value is Non Zero then we will"},{"line_number":102,"context_line":"simply abort the operation and return HTTP 400 to end user."},{"line_number":103,"context_line":""},{"line_number":104,"context_line":"If the `f_fsid` is found Non Zero at service start then we will ignore it"},{"line_number":105,"context_line":"considering local filesystem is used for Glance storage."}],"source_content_type":"text/x-rst","patch_set":3,"id":"fb1ffc70_d1d41468","line":102,"range":{"start_line":102,"start_character":38,"end_line":102,"end_character":46},"in_reply_to":"0d4a8883_856c9de0","updated":"2024-05-17 06:18:21.000000000","message":"Done","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"11eeb5dbfdb1349e1e0266ea920914468c39b6ab","unresolved":true,"context_lines":[{"line_number":99,"context_line":"assume that NFS share is configured for glance. We will again retrieve this"},{"line_number":100,"context_line":"value while adding image data to filesystem store or deleting the data from"},{"line_number":101,"context_line":"the filesystem store. If newly retrieved value is Non Zero then we will"},{"line_number":102,"context_line":"simply abort the operation and return HTTP 400 to end user."},{"line_number":103,"context_line":""},{"line_number":104,"context_line":"If the `f_fsid` is found Non Zero at service start then we will ignore it"},{"line_number":105,"context_line":"considering local filesystem is used for Glance storage."}],"source_content_type":"text/x-rst","patch_set":3,"id":"8982ff86_f164ed2a","line":102,"range":{"start_line":102,"start_character":38,"end_line":102,"end_character":46},"in_reply_to":"546b033a_1e8d2af8","updated":"2024-05-10 15:31:20.000000000","message":"Agree for not returning 4xx, I wanted to say need to verify configuring healthcheck middleware disable_by_file and how it behaves on startup and for api call after service started.\n\nSurprisingly I configured the same but it doesn\u0027t made any difference (not reported any issue)\n\n\nlocal api-paste.ini changes\n\n[app:healthcheck]\npaste.app_factory \u003d oslo_middleware:Healthcheck.app_factory\nbackends \u003d disabled_by_file\ndisable_by_file_path \u003d /opt/stack/data/glance/images\n\n[filter:healthcheck]\npaste.filter_factory \u003d oslo_middleware:Healthcheck.factory\npath \u003d /healthcheck\nbackends \u003d disabled_by_file\ndisable_by_file_path \u003d /opt/stack/data/glance/images\n#### set to True to enable detailed output, False is the default\ndetailed \u003d False\n\n\n[composite:api]\nkeystone+cachemanagement \u003d cors http_proxy_to_wsgi versionnegotiation osprofiler authtoken context healthcheck cache cachemanage rootapp","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"c5b02a12850e4533c38c658f66c1ec32e51bf4c9","unresolved":true,"context_lines":[{"line_number":99,"context_line":"assume that NFS share is configured for glance. We will again retrieve this"},{"line_number":100,"context_line":"value while adding image data to filesystem store or deleting the data from"},{"line_number":101,"context_line":"the filesystem store. If newly retrieved value is Non Zero then we will"},{"line_number":102,"context_line":"simply abort the operation and return HTTP 400 to end user."},{"line_number":103,"context_line":""},{"line_number":104,"context_line":"If the `f_fsid` is found Non Zero at service start then we will ignore it"},{"line_number":105,"context_line":"considering local filesystem is used for Glance storage."}],"source_content_type":"text/x-rst","patch_set":3,"id":"0d4a8883_856c9de0","line":102,"range":{"start_line":102,"start_character":38,"end_line":102,"end_character":46},"in_reply_to":"8982ff86_f164ed2a","updated":"2024-05-10 16:50:55.000000000","message":"It depends on if the mount is at the `glance` level or the `images` level. If the latter, then `images` will always exist, even if nothing is mounted there. If at the `glance` level, then checking for the presence of `images` may be good enough. However, you don\u0027t ever want glance to create the images/ directory if missing, otherwise an empty mount will create the dir when it shouldn\u0027t and then think it\u0027s healthy.","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":9303,"name":"Abhishek Kekane","email":"akekane@redhat.com","username":"abhishekkekane"},"change_message_id":"2eeac230d198c15bbe3053b4aa9423111131aeb8","unresolved":true,"context_lines":[{"line_number":99,"context_line":"assume that NFS share is configured for glance. We will again retrieve this"},{"line_number":100,"context_line":"value while adding image data to filesystem store or deleting the data from"},{"line_number":101,"context_line":"the filesystem store. If newly retrieved value is Non Zero then we will"},{"line_number":102,"context_line":"simply abort the operation and return HTTP 400 to end user."},{"line_number":103,"context_line":""},{"line_number":104,"context_line":"If the `f_fsid` is found Non Zero at service start then we will ignore it"},{"line_number":105,"context_line":"considering local filesystem is used for Glance storage."}],"source_content_type":"text/x-rst","patch_set":3,"id":"98e79966_b7a9d2cc","line":102,"range":{"start_line":102,"start_character":38,"end_line":102,"end_character":46},"in_reply_to":"9213de84_318172bd","updated":"2024-05-10 05:55:50.000000000","message":"Need to verify this behavior using healthcheck middleware","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"0edfac0f064c738fa20008d3c31111113a34f079","unresolved":true,"context_lines":[{"line_number":99,"context_line":"assume that NFS share is configured for glance. We will again retrieve this"},{"line_number":100,"context_line":"value while adding image data to filesystem store or deleting the data from"},{"line_number":101,"context_line":"the filesystem store. If newly retrieved value is Non Zero then we will"},{"line_number":102,"context_line":"simply abort the operation and return HTTP 400 to end user."},{"line_number":103,"context_line":""},{"line_number":104,"context_line":"If the `f_fsid` is found Non Zero at service start then we will ignore it"},{"line_number":105,"context_line":"considering local filesystem is used for Glance storage."}],"source_content_type":"text/x-rst","patch_set":3,"id":"546b033a_1e8d2af8","line":102,"range":{"start_line":102,"start_character":38,"end_line":102,"end_character":46},"in_reply_to":"98e79966_b7a9d2cc","updated":"2024-05-10 14:00:23.000000000","message":"Verify which behavior? The healthcheck is separate from the response code for actual image operations.\n\nIMHO, the way this should work is that if the mount disappears, the healthcheck goes to \"unhealthy\" and k8s stops sending requests to this worker.. it\u0027ll likely restart the pod (or some other remedy) and only once it goes back to healthy, start sending requests.\n\nA good reason not to return 4xx for image operations are where they might cause another service to do something wrong. Imagine if nova or cinder had some operation where a user deletes an instance (or volume) with a \"recurisive\" flag that means nova or cinder should also delete all snapshots in glance of that thing. Usually the way to do that is we list the images, then walk through and delete them. Any that return a 4xx (at least a 404 or 410) are ignored and don\u0027t trigger a failure because they must be \"already deleted.\" If glance is in a bad state and is returning these codes, it could cause nova to assume the snapshots are already deleted and continue deleting the instance, when it really should abort because it can\u0027t do the thing it was asked to do.\n\nAnother example might be some \"employee has left, delete all his data for legal/privacy reasons.\" If this glance pod happens to be in a bad state, the HR/legal automation script will think \"yep, all the private data has been securely deleted\" when it hasn\u0027t.","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"},{"author":{"_account_id":4393,"name":"Dan Smith","email":"dms@danplanet.com","username":"danms"},"change_message_id":"0e22231a5a691e670dfdd4b8be256d1e523d718f","unresolved":true,"context_lines":[{"line_number":99,"context_line":"assume that NFS share is configured for glance. We will again retrieve this"},{"line_number":100,"context_line":"value while adding image data to filesystem store or deleting the data from"},{"line_number":101,"context_line":"the filesystem store. If newly retrieved value is Non Zero then we will"},{"line_number":102,"context_line":"simply abort the operation and return HTTP 400 to end user."},{"line_number":103,"context_line":""},{"line_number":104,"context_line":"If the `f_fsid` is found Non Zero at service start then we will ignore it"},{"line_number":105,"context_line":"considering local filesystem is used for Glance storage."}],"source_content_type":"text/x-rst","patch_set":3,"id":"9213de84_318172bd","line":102,"range":{"start_line":102,"start_character":38,"end_line":102,"end_character":46},"in_reply_to":"cf7b5b55_0a2f432a","updated":"2024-05-09 14:26:37.000000000","message":"Why? 4xx means \"the user is at fault\" and neither 400 or 410 is correct for \"my storage is no longer accessible\". IMHO, the correct behavior is for the healthcheck to start to fail and for the pods to be restarted (ideally with fixed storage) no?","commit_id":"52923f9fdde9e2e63e3bf667f30900ca8f7281f5"}]}