Pushing Container with robot account ends with 500 (internal server error)

[reisman234]

I am trying to push a container image with a robot account created in Harbor, but the push fails with status code 500. I have created the robot-account as admin and is able to access 3 projects, 2 of them I have configured as proxy cache and address dockerhub and the nvidia container registry respectively, the robot-account can only read from these projects. A third project serves as a local storage location, so the account can read and write from it (pull and push).

The pull of a small test image via the proxy cache worked as intended. However, after the corresponding tagging and the push to save the image in the targeted project in the registry, it fails with status code 500.

The Version of Harbor is 2.7.1

The logs from the harbor-registry does not show any problems:

time="2023-11-11T17:28:42.863798883Z" level=info msg="authorized request" go.version=go1.19.4 http.request.host=HARBOR_REGISTRY http.request.id=84b32d77-6dae-4aec-ae0c-39766e443fe9 http.request.method=POST http.request.remoteaddr=127.0.0.1 http.request.uri="/v2/PROJECT3/alpine/blobs/uploads/" http.request.useragent="docker/24.0.7 go/go1.20.10 git-commit/311b9ff kernel/5.15.0-88-generic os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.7 \(linux\))" vars.name="PROJECT3/alpine" 
time="2023-11-11T17:28:42.866382933Z" level=info msg="response completed" go.version=go1.19.4 http.request.host=HARBOR_REGISTRY http.request.id=84b32d77-6dae-4aec-ae0c-39766e443fe9 http.request.method=POST http.request.remoteaddr=127.0.0.1 http.request.uri="/v2/PROJECT3/alpine/blobs/uploads/" http.request.useragent="docker/24.0.7 go/go1.20.10 git-commit/311b9ff kernel/5.15.0-88-generic os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.7 \(linux\))" http.response.duration=128.409778ms http.response.status=202 http.response.written=0 
10.1.58.99 - - [11/Nov/2023:17:28:42 +0000] "POST /v2/PROJECT3/alpine/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/24.0.7 go/go1.20.10 git-commit/311b9ff kernel/5.15.0-88-generic os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.7 \\(linux\\))"
time="2023-11-11T17:28:42.867490715Z" level=info msg="authorized request" go.version=go1.19.4 http.request.host=HARBOR_REGISTRY http.request.id=fea241b7-ec23-475c-84d4-48e77855f4a0 http.request.method=POST http.request.remoteaddr=127.0.0.1 http.request.uri="/v2/PROJECT3/alpine/blobs/uploads/" http.request.useragent="docker/24.0.7 go/go1.20.10 git-commit/311b9ff kernel/5.15.0-88-generic os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.7 \(linux\))" vars.name="PROJECT3/alpine" 
time="2023-11-11T17:28:42.873158412Z" level=info msg="response completed" go.version=go1.19.4 http.request.host=HARBOR_REGISTRY http.request.id=fea241b7-ec23-475c-84d4-48e77855f4a0 http.request.method=POST http.request.remoteaddr=127.0.0.1 http.request.uri="/v2/PROJECT3/alpine/blobs/uploads/" http.request.useragent="docker/24.0.7 go/go1.20.10 git-commit/311b9ff kernel/5.15.0-88-generic os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.7 \(linux\))" http.response.duration=133.105017ms http.response.status=202 http.response.written=0 
10.1.58.99 - - [11/Nov/2023:17:28:42 +0000] "POST /v2/PROJECT3/alpine/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/24.0.7 go/go1.20.10 git-commit/311b9ff kernel/5.15.0-88-generic os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.7 \\(linux\\))"
time="2023-11-11T17:28:43.423823059Z" level=info msg="authorized request" go.version=go1.19.4 http.request.host=HARBOR_REGISTRY http.request.id=2d58b86b-b932-4619-8ad0-00ea4d366d36 http.request.method=PATCH http.request.remoteaddr=127.0.0.1 http.request.uri="/v2/PROJECT3/alpine/blobs/uploads/b3403c59-4750-455b-8b79-1a7811f3807c?_state=z3F4C6AfQOHpteeemu9uRE24O6nF2MisHBRCHCRAfaJ7Ik5hbWUiOiJyaXNlbm1hbi9hbHBpbmUiLCJVVUlEIjoiYjM0MDNjNTktNDc1MC00NTViLThiNzktMWE3ODExZjM4MDdjIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDIzLTExLTExVDE3OjI4OjQyLjg2Mzg5NTAzMVoifQ%3D%3D" http.request.useragent="docker/24.0.7 go/go1.20.10 git-commit/311b9ff kernel/5.15.0-88-generic os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.7 \(linux\))" vars.name="PROJECT3/alpine" vars.uuid=b3403c59-4750-455b-8b79-1a7811f3807c 
time="2023-11-11T17:28:43.469088248Z" level=info msg="response completed" go.version=go1.19.4 http.request.host=HARBOR_REGISTRY http.request.id=2d58b86b-b932-4619-8ad0-00ea4d366d36 http.request.method=PATCH http.request.remoteaddr=127.0.0.1 http.request.uri="/v2/PROJECT3/alpine/blobs/uploads/b3403c59-4750-455b-8b79-1a7811f3807c?_state=z3F4C6AfQOHpteeemu9uRE24O6nF2MisHBRCHCRAfaJ7Ik5hbWUiOiJyaXNlbm1hbi9hbHBpbmUiLCJVVUlEIjoiYjM0MDNjNTktNDc1MC00NTViLThiNzktMWE3ODExZjM4MDdjIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDIzLTExLTExVDE3OjI4OjQyLjg2Mzg5NTAzMVoifQ%3D%3D" http.request.useragent="docker/24.0.7 go/go1.20.10 git-commit/311b9ff kernel/5.15.0-88-generic os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.7 \(linux\))" http.response.duration=142.345199ms http.response.status=202 http.response.written=0 
10.1.58.99 - - [11/Nov/2023:17:28:43 +0000] "PATCH /v2/PROJECT3/alpine/blobs/uploads/b3403c59-4750-455b-8b79-1a7811f3807c?_state=z3F4C6AfQOHpteeemu9uRE24O6nF2MisHBRCHCRAfaJ7Ik5hbWUiOiJyaXNlbm1hbi9hbHBpbmUiLCJVVUlEIjoiYjM0MDNjNTktNDc1MC00NTViLThiNzktMWE3ODExZjM4MDdjIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDIzLTExLTExVDE3OjI4OjQyLjg2Mzg5NTAzMVoifQ%3D%3D HTTP/1.1" 202 0 "" "docker/24.0.7 go/go1.20.10 git-commit/311b9ff kernel/5.15.0-88-generic os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.7 \\(linux\\))"
time="2023-11-11T17:28:43.471621361Z" level=info msg="authorized request" go.version=go1.19.4 http.request.host=HARBOR_REGISTRY http.request.id=057a6a9f-a9a4-477a-b1f1-ac2ee9c544d1 http.request.method=PATCH http.request.remoteaddr=127.0.0.1 http.request.uri="/v2/PROJECT3/alpine/blobs/uploads/b11f7e8b-282f-4c4f-bed8-c266da8ada8f?_state=ZvuEqJyisxL6VeJ_Y2A35mGRelCX0ud9An846Okm9Fh7Ik5hbWUiOiJyaXNlbm1hbi9hbHBpbmUiLCJVVUlEIjoiYjExZjdlOGItMjgyZi00YzRmLWJlZDgtYzI2NmRhOGFkYThmIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDIzLTExLTExVDE3OjI4OjQyLjg2NzU3OTdaIn0%3D" http.request.useragent="docker/24.0.7 go/go1.20.10 git-commit/311b9ff kernel/5.15.0-88-generic os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.7 \(linux\))" vars.name="PROJECT3/alpine" vars.uuid=b11f7e8b-282f-4c4f-bed8-c266da8ada8f 
time="2023-11-11T17:28:43.498572639Z" level=info msg="response completed" go.version=go1.19.4 http.request.host=HARBOR_REGISTRY http.request.id=057a6a9f-a9a4-477a-b1f1-ac2ee9c544d1 http.request.method=PATCH http.request.remoteaddr=127.0.0.1 http.request.uri="/v2/PROJECT3/alpine/blobs/uploads/b11f7e8b-282f-4c4f-bed8-c266da8ada8f?_state=ZvuEqJyisxL6VeJ_Y2A35mGRelCX0ud9An846Okm9Fh7Ik5hbWUiOiJyaXNlbm1hbi9hbHBpbmUiLCJVVUlEIjoiYjExZjdlOGItMjgyZi00YzRmLWJlZDgtYzI2NmRhOGFkYThmIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDIzLTExLTExVDE3OjI4OjQyLjg2NzU3OTdaIn0%3D" http.request.useragent="docker/24.0.7 go/go1.20.10 git-commit/311b9ff kernel/5.15.0-88-generic os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.7 \(linux\))" http.response.duration=126.807501ms http.response.status=202 http.response.written=0 
10.1.58.99 - - [11/Nov/2023:17:28:43 +0000] "PATCH /v2/PROJECT3/alpine/blobs/uploads/b11f7e8b-282f-4c4f-bed8-c266da8ada8f?_state=ZvuEqJyisxL6VeJ_Y2A35mGRelCX0ud9An846Okm9Fh7Ik5hbWUiOiJyaXNlbm1hbi9hbHBpbmUiLCJVVUlEIjoiYjExZjdlOGItMjgyZi00YzRmLWJlZDgtYzI2NmRhOGFkYThmIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDIzLTExLTExVDE3OjI4OjQyLjg2NzU3OTdaIn0%3D HTTP/1.1" 202 0 "" "docker/24.0.7 go/go1.20.10 git-commit/311b9ff kernel/5.15.0-88-generic os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.7 \\(linux\\))"
time="2023-11-11T17:28:43.594177994Z" level=info msg="authorized request" go.version=go1.19.4 http.request.host=HARBOR_REGISTRY http.request.id=33dc3138-eddc-43ef-9e66-348d7f47d8bb http.request.method=PUT http.request.remoteaddr=127.0.0.1 http.request.uri="/v2/PROJECT3/alpine/blobs/uploads/b3403c59-4750-455b-8b79-1a7811f3807c?_state=g1ExhVmh69-nLC9J0dyNPUo8gJq4S6vM1uhwX7E9wkJ7Ik5hbWUiOiJyaXNlbm1hbi9hbHBpbmUiLCJVVUlEIjoiYjM0MDNjNTktNDc1MC00NTViLThiNzktMWE3ODExZjM4MDdjIiwiT2Zmc2V0IjoyMTk2MTI5LCJTdGFydGVkQXQiOiIyMDIzLTExLTExVDE3OjI4OjQyWiJ9&digest=sha256%3A59d8ed68a22458f95b452bde3ecf2c5c6bf8dc48a5d10d0ae314fd4685a75619" http.request.useragent="docker/24.0.7 go/go1.20.10 git-commit/311b9ff kernel/5.15.0-88-generic os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.7 \(linux\))" vars.name="PROJECT3/alpine" vars.uuid=b3403c59-4750-455b-8b79-1a7811f3807c 
time="2023-11-11T17:28:43.599595114Z" level=info msg="redis: connect harbor-redis:6379" go.version=go1.19.4 instance.id=c338d66f-f93f-4e0d-a712-30d2beff4479 redis.connect.duration=961.577µs service=registry version=v2.8.0.m 
10.1.58.99 - - [11/Nov/2023:17:28:43 +0000] "PUT /v2/PROJECT3/alpine/blobs/uploads/b3403c59-4750-455b-8b79-1a7811f3807c?_state=g1ExhVmh69-nLC9J0dyNPUo8gJq4S6vM1uhwX7E9wkJ7Ik5hbWUiOiJyaXNlbm1hbi9hbHBpbmUiLCJVVUlEIjoiYjM0MDNjNTktNDc1MC00NTViLThiNzktMWE3ODExZjM4MDdjIiwiT2Zmc2V0IjoyMTk2MTI5LCJTdGFydGVkQXQiOiIyMDIzLTExLTExVDE3OjI4OjQyWiJ9&digest=sha256%3A59d8ed68a22458f95b452bde3ecf2c5c6bf8dc48a5d10d0ae314fd4685a75619 HTTP/1.1" 201 0 "" "docker/24.0.7 go/go1.20.10 git-commit/311b9ff kernel/5.15.0-88-generic os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.7 \\(linux\\))"
time="2023-11-11T17:28:43.600069563Z" level=info msg="response completed" go.version=go1.19.4 http.request.host=HARBOR_REGISTRY http.request.id=33dc3138-eddc-43ef-9e66-348d7f47d8bb http.request.method=PUT http.request.remoteaddr=127.0.0.1 http.request.uri="/v2/PROJECT3/alpine/blobs/uploads/b3403c59-4750-455b-8b79-1a7811f3807c?_state=g1ExhVmh69-nLC9J0dyNPUo8gJq4S6vM1uhwX7E9wkJ7Ik5hbWUiOiJyaXNlbm1hbi9hbHBpbmUiLCJVVUlEIjoiYjM0MDNjNTktNDc1MC00NTViLThiNzktMWE3ODExZjM4MDdjIiwiT2Zmc2V0IjoyMTk2MTI5LCJTdGFydGVkQXQiOiIyMDIzLTExLTExVDE3OjI4OjQyWiJ9&digest=sha256%3A59d8ed68a22458f95b452bde3ecf2c5c6bf8dc48a5d10d0ae314fd4685a75619" http.request.useragent="docker/24.0.7 go/go1.20.10 git-commit/311b9ff kernel/5.15.0-88-generic os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.7 \(linux\))" http.response.duration=104.014915ms http.response.status=201 http.response.written=0 

I can see some errors in the harbor-core container, but it doesn't help me.

2023-11-11T17:37:14Z [ERROR] [/server/middleware/blob/put_blob_upload.go:68][middleware="blob" requestID="2db30e1c1dc57ce9052b77b874e39198"]: ensure blob sha256:59d8ed68a22458f95b452bde3ecf2c5c6bf8dc48a5d10d0ae314fd4685a75619 failed, error: ERROR: unexpected data beyond EOF in block 5 of relation base/16388/16906 (SQLSTATE XX000)
2023-11-11T17:37:14Z [ERROR] [/lib/http/error.go:56]: {"errors":[{"code":"UNKNOWN","message":"unknown: ERROR: unexpected data beyond EOF in block 5 of relation base/16388/16906 (SQLSTATE XX000)"}]} 

[AllForNothing]

@reisman234

1. Does it work if you use an admin account?
2. Can you share the JSON payload of your robot account?

[AllForNothing]

@reisman234
Can you provide the log from the DB container?

[reisman234]

@reisman234

1. Does it work if you use an admin account?
2. Can you share the JSON payload of your robot account?

With the admin account, I had the same problem. What did you mean with JSON payload from the robot account ?

@reisman234 Can you provide the log from the DB container?
Before I saw your comment, I also head a look at the DB conainter and it showed also some errors with the Hint to update the Kernel and restart, which I did. But now, the redis-db has also a startup error, which I currently don't know how to resolve.

[offset 0] Checking RDB file dump.rdb
[offset 26] AUX FIELD redis-ver = '7.0.0'
[offset 40] AUX FIELD redis-bits = '64'
[offset 52] AUX FIELD ctime = '1699863641'
[offset 67] AUX FIELD used-mem = '1955224'
[offset 79] AUX FIELD aof-base = '0'
[offset 81] Selecting DB ID 0
[offset 961] Selecting DB ID 1
[offset 28576] Selecting DB ID 2
--- RDB ERROR DETECTED ---
[offset 91110] Internal error in RDB reading offset 0, function at rdb.c:529 -> Unknown RDB string encoding type 48
[additional info] While doing: read-object-value
[additional info] Reading key 'bl5@1689897600'
[additional info] Reading type 4 (hash-hashtable)
[info] 279 keys read
[info] 0 expires
[info] 0 already expired
19:C 13 Nov 2023 08:42:46.340 # Terminating server after rdb file reading failure.

To start redis again, I moved the dump.rdb file, which worked so fare that the containers (redis, core, job...) could start again. But now there are missing projects and users, which cannot be created again because the UI throwing error "they are already existing".

[AllForNothing]

@reisman234 So this issue has nothing to do with the robot account.

I guess there was something wrong with the DB container.

[reisman234]

Since nobody but me currently uses the registry and only containers were stored which were built by a CI pipeline, it is probably easier to simply reinstall Harbor than to tinker with databases. Also, the current error has nothing to do with the initial discussion.
Thanks for your time though