S3 'blob unknown'/'blob unknown to registry'

[mdavid01]

We're running harbor 2.10 with Postgresql 13.5 and AWS S3 storage on VmWare TKG. Every day we get 250,000+ errors from the registry containers similar to what's shown below. Although most often, errors refer to proxy-cache items, many are non-proxy-cache. The s3 'blobs' folder shows that the digest exists as does the DB 'blobs' table and the DB 'artifact_blob' table. The digest does not appear in DB 'artifact',artifact_trash, artifact_reference or artifact_accessory tables. I do not see any references to this digest in the s3 repository records: manifest or layers folders.

1. Is this message from Harbor (as opposed to, say, a pre-processor app that we use to add our certificates)?
2. If so, what Harbor module is producing the error?
3. Is this something we can fix on our side. It's massively impacting our splunk and AWS cloudwatch volumes.
4. Is this a valid uri? http.request.uri = /v2/lmc.eo.swf.lmified/ext.hub.docker.com/library/docker/blobs/sha256:8ee87cef5aa57784339364a8f2212b7ad601dd6ae6de7f8928d4db492a7f79f2

Thanks

===================================================
ERROR MESSAGE EXAMPLE
registry time="2024-12-17T19:22:05.836413718Z" level=error msg="response completed with error" auth.user.name="harbor_registry_user" err.code="blob unknown" err.detail=sha256:8ee87cef5aa57784339364a8f2212b7ad601dd6ae6de7f8928d4db492a7f79f2 err.message="blob unknown to registry" go.version=go1.21.4 http.request.host=harbor.XXXXX http.request.id=71026439-1346-4d5a-806b-b735d9758989 http.request.method=GET http.request.remoteaddr=100.xx.xx.xx http.request.uri="/v2/XXXX.swf.lmified/ext.hub.docker.com/library/docker/blobs/sha256:8ee87cef5aa57784339364a8f2212b7ad601dd6ae6de7f8928d4db492a7f79f2"http.request.useragent="Go-http-client/1.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration=117.290868ms http.response.status=404 http.response.written=157 vars.digest="sha256:8ee87cef5aa57784339364a8f2212b7ad601dd6ae6de7f8928d4db492a7f79f2" vars.name="XXXX.XXXX/ext.hub.docker.com/library/docker"

[wy65701436]

The s3 'blobs' folder shows that the digest exists as does the DB 'blobs' table and the DB 'artifact_blob' table.

This means that the artifact has been moved from the harbor and has not been garbage collected. Have you ever executed a garbage collection in this harbor?

[reasonerjt]

@mdavid01

It looks like you're a VMware(by Broadcom) customer, I suggest you contact GSS support of VMware(by Broadcom) to have official support.

[mdavid01]

Thanks @wy65701436 and @reasonerjt. I should have mentioned that the digest appears in the project/repo UI.

GC feedback could be part of the issue. Pls see attached pptx to see state of GC. We have 4 instances of harbor running and only this instance has GC issues. This happens to be our prod instance that includes proxy-cache and other company projects.

Unfortunately, we have no clue where to look to to fix the problem. Could it be Sentinel-Redis? Does GC rely on AWS S3 versioning? We have AWS s3 versioning turned on. We run a script to remove non-current and deleted versions of AWS S3 objects.

HarborIssue21331NotFoundError.pptx

[wy65701436]

Thanks @wy65701436 and @reasonerjt. I should have mentioned that the digest appears in the project/repo UI.

GC feedback could be part of the issue. Pls see attached pptx to see state of GC. We have 4 instances of harbor running and only this instance has GC issues. This happens to be our prod instance that includes proxy-cache and other company projects.

Unfortunately, we have no clue where to look to to fix the problem. Could it be Sentinel-Redis? Does GC rely on AWS S3 versioning? We have AWS s3 versioning turned on. We run a script to remove non-current and deleted versions of AWS S3 objects.

HarborIssue21331NotFoundError.pptx

hi @mdavid01 as @reasonerjt mentioned above, if you're a VMware(by Broadcom) customer, you can reach out to GSS of VMware(by Broadcom) to get the official support.