Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [........X..............................X...........XX..X....] duration:
3.645338828s trace:
https://jaeger.dev.storj.io/trace/10c47e6f0c350377
FAIL (2026-04-05 06:08:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.645365196s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.........X..............................X...........XX..X...] duration:
3.62049696s trace:
https://jaeger.dev.storj.io/trace/10c47e6f0c350377
FAIL (2026-04-05 06:07:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.620522089s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
4.719257137s trace:
https://jaeger.dev.storj.io/trace/1fbfbe664cccd119
FAIL (2026-04-05 06:07:29 UTC)
orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded
Duration: 4.719286464s
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [..........X..............................X...........XX..X..] duration:
3.710756296s trace:
https://jaeger.dev.storj.io/trace/10c47e6f0c350377
FAIL (2026-04-05 06:07:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.710779692s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
5.815748213s trace:
https://jaeger.dev.storj.io/trace/5a5341770822a3cc
FAIL (2026-04-05 06:07:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
ce_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.192807417 (714h20m27.105145058s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193033373 (714h20m27.105371014s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193198932 (714h20m27.105536564s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193549759 (714h20m27.10588739s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193572203 (714h20m27.105909834s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194022512 (714h20m27.106360143s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194104560 (714h20m27.106442191s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194389039 (714h20m27.10672667s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194717774 (714h20m27.107055415s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194730568 (714h20m27.107068199s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195059643 (714h20m27.107397274s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195234200 (714h20m27.107571831s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195436932 (714h20m27.107774563s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195714016 (714h20m27.108051657s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:45.988724037 (714h20m28.901061668s) [upload=685519]: Next returned nil segment
Apr 5 05:25:45.988736951 (714h20m28.901074582s) [upload=685519]: Waiting for error group managing segments...
Apr 5 05:25:46.278334571 (714h20m29.190672202s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.281965999 (714h20m29.19430363s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.292588509 (714h20m29.204926151s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.302631332 (714h20m29.214968973s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.305482154 (714h20m29.217819785s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:26:13.304943733 (714h20m56.217281364s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:58840->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.304978810 (714h20m56.217316442s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: rpc: dial tcp 145.79.227.42:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305052803 (714h20m56.217390434s) [noise=true , address=145.79.227.28:4729 , piece_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: read tcp 188.166.225.170:46538->145.79.227.28:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305085256 (714h20m56.217422877s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: read tcp 188.166.225.170:58958->145.79.227.39:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305144721 (714h20m56.217482352s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: read tcp 188.166.225.170:44166->145.79.227.3:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305173626 (714h20m56.217511257s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: read tcp 188.166.225.170:45470->145.79.227.29:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305241718 (714h20m56.217579349s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:127L5Mr53vCqKr4gy7N4rAaVCC9FvU8XdYKFrxWrJL728dt3LmR, address:145.79.227.32:4833): protocol: read tcp 188.166.225.170:42334->145.79.227.32:4833: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305260534 (714h20m56.217598165s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1sDggu72xPTG9o2K8sbPHbYfwXarrvqP98YySo7ZSEZ2ddYFEz, address:145.79.227.7:4298): protocol: read tcp 188.166.225.170:38480->145.79.227.7:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305329106 (714h20m56.217666737s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:46896->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305435973 (714h20m56.217773604s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1zDoLfA5bQWM7Uf479gfFVjG3yYvxG5EZS5yZSFGTvoFoPYEAo, address:145.79.227.46:4607): protocol: read tcp 188.166.225.170:49286->145.79.227.46:4607: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305531467 (714h20m56.217869098s) [noise=true , address=145.79.227.18:4010 , piece_id=XC2T7S72SFQ2Z45N7VLGY7CAMBLCZ4MOWU3BALSHACYGOPQPTEWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp, address:145.79.227.18:4010): protocol: read tcp 188.166.225.170:44936->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305549902 (714h20m56.217887534s) [seg_pos={0 0}, upload=685519]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Apr 5 05:26:13.536348486 (714h20m56.448686117s) [upload=685519]: Done uploading segments...
Apr 5 05:26:13.536382360 (714h20m56.448719991s) [upload=685519]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.815782229s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [...........X..............................X...........XX..X.] duration:
3.542632714s trace:
https://jaeger.dev.storj.io/trace/10c47e6f0c350377
FAIL (2026-04-05 06:06:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.542652742s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
4.72273919s trace:
https://jaeger.dev.storj.io/trace/1fbfbe664cccd119
FAIL (2026-04-05 06:06:38 UTC)
orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded
Duration: 4.722768937s
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [............X..............................X...........XX..X] duration:
5.001185265s trace:
https://jaeger.dev.storj.io/trace/10c47e6f0c350377
FAIL (2026-04-05 06:06:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001208048s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
5.618840962s trace:
https://jaeger.dev.storj.io/trace/5a5341770822a3cc
FAIL (2026-04-05 06:06:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
ce_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.192807417 (714h20m27.105145058s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193033373 (714h20m27.105371014s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193198932 (714h20m27.105536564s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193549759 (714h20m27.10588739s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193572203 (714h20m27.105909834s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194022512 (714h20m27.106360143s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194104560 (714h20m27.106442191s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194389039 (714h20m27.10672667s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194717774 (714h20m27.107055415s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194730568 (714h20m27.107068199s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195059643 (714h20m27.107397274s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195234200 (714h20m27.107571831s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195436932 (714h20m27.107774563s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195714016 (714h20m27.108051657s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:45.988724037 (714h20m28.901061668s) [upload=685519]: Next returned nil segment
Apr 5 05:25:45.988736951 (714h20m28.901074582s) [upload=685519]: Waiting for error group managing segments...
Apr 5 05:25:46.278334571 (714h20m29.190672202s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.281965999 (714h20m29.19430363s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.292588509 (714h20m29.204926151s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.302631332 (714h20m29.214968973s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.305482154 (714h20m29.217819785s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:26:13.304943733 (714h20m56.217281364s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:58840->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.304978810 (714h20m56.217316442s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: rpc: dial tcp 145.79.227.42:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305052803 (714h20m56.217390434s) [noise=true , address=145.79.227.28:4729 , piece_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: read tcp 188.166.225.170:46538->145.79.227.28:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305085256 (714h20m56.217422877s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: read tcp 188.166.225.170:58958->145.79.227.39:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305144721 (714h20m56.217482352s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: read tcp 188.166.225.170:44166->145.79.227.3:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305173626 (714h20m56.217511257s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: read tcp 188.166.225.170:45470->145.79.227.29:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305241718 (714h20m56.217579349s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:127L5Mr53vCqKr4gy7N4rAaVCC9FvU8XdYKFrxWrJL728dt3LmR, address:145.79.227.32:4833): protocol: read tcp 188.166.225.170:42334->145.79.227.32:4833: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305260534 (714h20m56.217598165s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1sDggu72xPTG9o2K8sbPHbYfwXarrvqP98YySo7ZSEZ2ddYFEz, address:145.79.227.7:4298): protocol: read tcp 188.166.225.170:38480->145.79.227.7:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305329106 (714h20m56.217666737s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:46896->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305435973 (714h20m56.217773604s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1zDoLfA5bQWM7Uf479gfFVjG3yYvxG5EZS5yZSFGTvoFoPYEAo, address:145.79.227.46:4607): protocol: read tcp 188.166.225.170:49286->145.79.227.46:4607: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305531467 (714h20m56.217869098s) [noise=true , address=145.79.227.18:4010 , piece_id=XC2T7S72SFQ2Z45N7VLGY7CAMBLCZ4MOWU3BALSHACYGOPQPTEWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp, address:145.79.227.18:4010): protocol: read tcp 188.166.225.170:44936->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305549902 (714h20m56.217887534s) [seg_pos={0 0}, upload=685519]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Apr 5 05:26:13.536348486 (714h20m56.448686117s) [upload=685519]: Done uploading segments...
Apr 5 05:26:13.536382360 (714h20m56.448719991s) [upload=685519]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.618863715s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.............X..............................X...........XX..] duration:
3.497744082s trace:
https://jaeger.dev.storj.io/trace/51e35f85cb541694
FAIL (2026-04-05 06:05:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.497764131s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
4.693739011s trace:
https://jaeger.dev.storj.io/trace/1fbfbe664cccd119
FAIL (2026-04-05 06:05:51 UTC)
orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded
Duration: 4.693768348s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
5.664766471s trace:
https://jaeger.dev.storj.io/trace/5a5341770822a3cc
FAIL (2026-04-05 06:05:32 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
ce_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.192807417 (714h20m27.105145058s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193033373 (714h20m27.105371014s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193198932 (714h20m27.105536564s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193549759 (714h20m27.10588739s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193572203 (714h20m27.105909834s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194022512 (714h20m27.106360143s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194104560 (714h20m27.106442191s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194389039 (714h20m27.10672667s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194717774 (714h20m27.107055415s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194730568 (714h20m27.107068199s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195059643 (714h20m27.107397274s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195234200 (714h20m27.107571831s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195436932 (714h20m27.107774563s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195714016 (714h20m27.108051657s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:45.988724037 (714h20m28.901061668s) [upload=685519]: Next returned nil segment
Apr 5 05:25:45.988736951 (714h20m28.901074582s) [upload=685519]: Waiting for error group managing segments...
Apr 5 05:25:46.278334571 (714h20m29.190672202s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.281965999 (714h20m29.19430363s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.292588509 (714h20m29.204926151s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.302631332 (714h20m29.214968973s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.305482154 (714h20m29.217819785s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:26:13.304943733 (714h20m56.217281364s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:58840->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.304978810 (714h20m56.217316442s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: rpc: dial tcp 145.79.227.42:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305052803 (714h20m56.217390434s) [noise=true , address=145.79.227.28:4729 , piece_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: read tcp 188.166.225.170:46538->145.79.227.28:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305085256 (714h20m56.217422877s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: read tcp 188.166.225.170:58958->145.79.227.39:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305144721 (714h20m56.217482352s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: read tcp 188.166.225.170:44166->145.79.227.3:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305173626 (714h20m56.217511257s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: read tcp 188.166.225.170:45470->145.79.227.29:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305241718 (714h20m56.217579349s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:127L5Mr53vCqKr4gy7N4rAaVCC9FvU8XdYKFrxWrJL728dt3LmR, address:145.79.227.32:4833): protocol: read tcp 188.166.225.170:42334->145.79.227.32:4833: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305260534 (714h20m56.217598165s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1sDggu72xPTG9o2K8sbPHbYfwXarrvqP98YySo7ZSEZ2ddYFEz, address:145.79.227.7:4298): protocol: read tcp 188.166.225.170:38480->145.79.227.7:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305329106 (714h20m56.217666737s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:46896->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305435973 (714h20m56.217773604s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1zDoLfA5bQWM7Uf479gfFVjG3yYvxG5EZS5yZSFGTvoFoPYEAo, address:145.79.227.46:4607): protocol: read tcp 188.166.225.170:49286->145.79.227.46:4607: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305531467 (714h20m56.217869098s) [noise=true , address=145.79.227.18:4010 , piece_id=XC2T7S72SFQ2Z45N7VLGY7CAMBLCZ4MOWU3BALSHACYGOPQPTEWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp, address:145.79.227.18:4010): protocol: read tcp 188.166.225.170:44936->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305549902 (714h20m56.217887534s) [seg_pos={0 0}, upload=685519]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Apr 5 05:26:13.536348486 (714h20m56.448686117s) [upload=685519]: Done uploading segments...
Apr 5 05:26:13.536382360 (714h20m56.448719991s) [upload=685519]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.664794515s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..............X..............................X...........XX.] duration:
3.502003692s trace:
https://jaeger.dev.storj.io/trace/51e35f85cb541694
FAIL (2026-04-05 06:05:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.502031174s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
5.213842029s trace:
https://jaeger.dev.storj.io/trace/1fbfbe664cccd119
FAIL (2026-04-05 06:05:04 UTC)
orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded
Duration: 5.213878488s
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...............X..............................X...........XX] duration:
5.001371806s trace:
https://jaeger.dev.storj.io/trace/51e35f85cb541694
FAIL (2026-04-05 06:05:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001390021s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
5.754509007s trace:
https://jaeger.dev.storj.io/trace/5a5341770822a3cc
FAIL (2026-04-05 06:04:43 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
ce_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.192807417 (714h20m27.105145058s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193033373 (714h20m27.105371014s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193198932 (714h20m27.105536564s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193549759 (714h20m27.10588739s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193572203 (714h20m27.105909834s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194022512 (714h20m27.106360143s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194104560 (714h20m27.106442191s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194389039 (714h20m27.10672667s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194717774 (714h20m27.107055415s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194730568 (714h20m27.107068199s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195059643 (714h20m27.107397274s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195234200 (714h20m27.107571831s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195436932 (714h20m27.107774563s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195714016 (714h20m27.108051657s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:45.988724037 (714h20m28.901061668s) [upload=685519]: Next returned nil segment
Apr 5 05:25:45.988736951 (714h20m28.901074582s) [upload=685519]: Waiting for error group managing segments...
Apr 5 05:25:46.278334571 (714h20m29.190672202s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.281965999 (714h20m29.19430363s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.292588509 (714h20m29.204926151s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.302631332 (714h20m29.214968973s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.305482154 (714h20m29.217819785s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:26:13.304943733 (714h20m56.217281364s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:58840->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.304978810 (714h20m56.217316442s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: rpc: dial tcp 145.79.227.42:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305052803 (714h20m56.217390434s) [noise=true , address=145.79.227.28:4729 , piece_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: read tcp 188.166.225.170:46538->145.79.227.28:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305085256 (714h20m56.217422877s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: read tcp 188.166.225.170:58958->145.79.227.39:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305144721 (714h20m56.217482352s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: read tcp 188.166.225.170:44166->145.79.227.3:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305173626 (714h20m56.217511257s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: read tcp 188.166.225.170:45470->145.79.227.29:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305241718 (714h20m56.217579349s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:127L5Mr53vCqKr4gy7N4rAaVCC9FvU8XdYKFrxWrJL728dt3LmR, address:145.79.227.32:4833): protocol: read tcp 188.166.225.170:42334->145.79.227.32:4833: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305260534 (714h20m56.217598165s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1sDggu72xPTG9o2K8sbPHbYfwXarrvqP98YySo7ZSEZ2ddYFEz, address:145.79.227.7:4298): protocol: read tcp 188.166.225.170:38480->145.79.227.7:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305329106 (714h20m56.217666737s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:46896->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305435973 (714h20m56.217773604s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1zDoLfA5bQWM7Uf479gfFVjG3yYvxG5EZS5yZSFGTvoFoPYEAo, address:145.79.227.46:4607): protocol: read tcp 188.166.225.170:49286->145.79.227.46:4607: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305531467 (714h20m56.217869098s) [noise=true , address=145.79.227.18:4010 , piece_id=XC2T7S72SFQ2Z45N7VLGY7CAMBLCZ4MOWU3BALSHACYGOPQPTEWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp, address:145.79.227.18:4010): protocol: read tcp 188.166.225.170:44936->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305549902 (714h20m56.217887534s) [seg_pos={0 0}, upload=685519]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Apr 5 05:26:13.536348486 (714h20m56.448686117s) [upload=685519]: Done uploading segments...
Apr 5 05:26:13.536382360 (714h20m56.448719991s) [upload=685519]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.754529977s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
4.968780046s trace:
https://jaeger.dev.storj.io/trace/1fbfbe664cccd119
FAIL (2026-04-05 06:04:32 UTC)
orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded
Duration: 4.968795776s
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [................X..............................X...........X] duration:
5.000730618s trace:
https://jaeger.dev.storj.io/trace/e48115d0e40cd67
FAIL (2026-04-05 06:04:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000757731s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.................X..............................X...........] duration:
3.350802306s trace:
https://jaeger.dev.storj.io/trace/77d099d715fa4f39
FAIL (2026-04-05 06:03:53 UTC)
orbiter warning: context deadline exceeded
Duration: 3.350827354s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
5.680935905s trace:
https://jaeger.dev.storj.io/trace/5a5341770822a3cc
FAIL (2026-04-05 06:03:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
ce_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.192807417 (714h20m27.105145058s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193033373 (714h20m27.105371014s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193198932 (714h20m27.105536564s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193549759 (714h20m27.10588739s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193572203 (714h20m27.105909834s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194022512 (714h20m27.106360143s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194104560 (714h20m27.106442191s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194389039 (714h20m27.10672667s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194717774 (714h20m27.107055415s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194730568 (714h20m27.107068199s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195059643 (714h20m27.107397274s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195234200 (714h20m27.107571831s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195436932 (714h20m27.107774563s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195714016 (714h20m27.108051657s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:45.988724037 (714h20m28.901061668s) [upload=685519]: Next returned nil segment
Apr 5 05:25:45.988736951 (714h20m28.901074582s) [upload=685519]: Waiting for error group managing segments...
Apr 5 05:25:46.278334571 (714h20m29.190672202s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.281965999 (714h20m29.19430363s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.292588509 (714h20m29.204926151s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.302631332 (714h20m29.214968973s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.305482154 (714h20m29.217819785s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:26:13.304943733 (714h20m56.217281364s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:58840->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.304978810 (714h20m56.217316442s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: rpc: dial tcp 145.79.227.42:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305052803 (714h20m56.217390434s) [noise=true , address=145.79.227.28:4729 , piece_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: read tcp 188.166.225.170:46538->145.79.227.28:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305085256 (714h20m56.217422877s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: read tcp 188.166.225.170:58958->145.79.227.39:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305144721 (714h20m56.217482352s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: read tcp 188.166.225.170:44166->145.79.227.3:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305173626 (714h20m56.217511257s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: read tcp 188.166.225.170:45470->145.79.227.29:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305241718 (714h20m56.217579349s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:127L5Mr53vCqKr4gy7N4rAaVCC9FvU8XdYKFrxWrJL728dt3LmR, address:145.79.227.32:4833): protocol: read tcp 188.166.225.170:42334->145.79.227.32:4833: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305260534 (714h20m56.217598165s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1sDggu72xPTG9o2K8sbPHbYfwXarrvqP98YySo7ZSEZ2ddYFEz, address:145.79.227.7:4298): protocol: read tcp 188.166.225.170:38480->145.79.227.7:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305329106 (714h20m56.217666737s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:46896->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305435973 (714h20m56.217773604s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1zDoLfA5bQWM7Uf479gfFVjG3yYvxG5EZS5yZSFGTvoFoPYEAo, address:145.79.227.46:4607): protocol: read tcp 188.166.225.170:49286->145.79.227.46:4607: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305531467 (714h20m56.217869098s) [noise=true , address=145.79.227.18:4010 , piece_id=XC2T7S72SFQ2Z45N7VLGY7CAMBLCZ4MOWU3BALSHACYGOPQPTEWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp, address:145.79.227.18:4010): protocol: read tcp 188.166.225.170:44936->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305549902 (714h20m56.217887534s) [seg_pos={0 0}, upload=685519]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Apr 5 05:26:13.536348486 (714h20m56.448686117s) [upload=685519]: Done uploading segments...
Apr 5 05:26:13.536382360 (714h20m56.448719991s) [upload=685519]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.680957326s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
4.701007919s trace:
https://jaeger.dev.storj.io/trace/1fbfbe664cccd119
FAIL (2026-04-05 06:03:47 UTC)
orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded
Duration: 4.70103424s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
4.63787725s trace:
https://jaeger.dev.storj.io/trace/1fbfbe664cccd119
FAIL (2026-04-05 06:03:31 UTC)
orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded
Duration: 4.63789229s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.000558715s trace:
https://jaeger.dev.storj.io/trace/1fbfbe664cccd119
FAIL (2026-04-05 06:02:50 UTC)
orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded
Duration: 30.000577541s
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..................X..............................X..........] duration:
3.423943551s trace:
https://jaeger.dev.storj.io/trace/77d099d715fa4f39
FAIL (2026-04-05 06:02:47 UTC)
orbiter warning: context deadline exceeded
Duration: 3.423973278s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
5.769189585s trace:
https://jaeger.dev.storj.io/trace/5a5341770822a3cc
FAIL (2026-04-05 06:02:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
ce_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.192807417 (714h20m27.105145058s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193033373 (714h20m27.105371014s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193198932 (714h20m27.105536564s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193549759 (714h20m27.10588739s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193572203 (714h20m27.105909834s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194022512 (714h20m27.106360143s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194104560 (714h20m27.106442191s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194389039 (714h20m27.10672667s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194717774 (714h20m27.107055415s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194730568 (714h20m27.107068199s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195059643 (714h20m27.107397274s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195234200 (714h20m27.107571831s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195436932 (714h20m27.107774563s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195714016 (714h20m27.108051657s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:45.988724037 (714h20m28.901061668s) [upload=685519]: Next returned nil segment
Apr 5 05:25:45.988736951 (714h20m28.901074582s) [upload=685519]: Waiting for error group managing segments...
Apr 5 05:25:46.278334571 (714h20m29.190672202s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.281965999 (714h20m29.19430363s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.292588509 (714h20m29.204926151s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.302631332 (714h20m29.214968973s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.305482154 (714h20m29.217819785s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:26:13.304943733 (714h20m56.217281364s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:58840->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.304978810 (714h20m56.217316442s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: rpc: dial tcp 145.79.227.42:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305052803 (714h20m56.217390434s) [noise=true , address=145.79.227.28:4729 , piece_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: read tcp 188.166.225.170:46538->145.79.227.28:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305085256 (714h20m56.217422877s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: read tcp 188.166.225.170:58958->145.79.227.39:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305144721 (714h20m56.217482352s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: read tcp 188.166.225.170:44166->145.79.227.3:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305173626 (714h20m56.217511257s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: read tcp 188.166.225.170:45470->145.79.227.29:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305241718 (714h20m56.217579349s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:127L5Mr53vCqKr4gy7N4rAaVCC9FvU8XdYKFrxWrJL728dt3LmR, address:145.79.227.32:4833): protocol: read tcp 188.166.225.170:42334->145.79.227.32:4833: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305260534 (714h20m56.217598165s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1sDggu72xPTG9o2K8sbPHbYfwXarrvqP98YySo7ZSEZ2ddYFEz, address:145.79.227.7:4298): protocol: read tcp 188.166.225.170:38480->145.79.227.7:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305329106 (714h20m56.217666737s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:46896->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305435973 (714h20m56.217773604s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1zDoLfA5bQWM7Uf479gfFVjG3yYvxG5EZS5yZSFGTvoFoPYEAo, address:145.79.227.46:4607): protocol: read tcp 188.166.225.170:49286->145.79.227.46:4607: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305531467 (714h20m56.217869098s) [noise=true , address=145.79.227.18:4010 , piece_id=XC2T7S72SFQ2Z45N7VLGY7CAMBLCZ4MOWU3BALSHACYGOPQPTEWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp, address:145.79.227.18:4010): protocol: read tcp 188.166.225.170:44936->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305549902 (714h20m56.217887534s) [seg_pos={0 0}, upload=685519]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Apr 5 05:26:13.536348486 (714h20m56.448686117s) [upload=685519]: Done uploading segments...
Apr 5 05:26:13.536382360 (714h20m56.448719991s) [upload=685519]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.769223129s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...................X..............................X.........] duration:
3.409951498s trace:
https://jaeger.dev.storj.io/trace/77d099d715fa4f39
FAIL (2026-04-05 06:02:09 UTC)
orbiter warning: context deadline exceeded
Duration: 3.409980614s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
5.827033948s trace:
https://jaeger.dev.storj.io/trace/5a5341770822a3cc
FAIL (2026-04-05 06:01:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
ce_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.192807417 (714h20m27.105145058s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193033373 (714h20m27.105371014s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193198932 (714h20m27.105536564s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193549759 (714h20m27.10588739s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193572203 (714h20m27.105909834s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194022512 (714h20m27.106360143s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194104560 (714h20m27.106442191s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194389039 (714h20m27.10672667s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194717774 (714h20m27.107055415s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194730568 (714h20m27.107068199s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195059643 (714h20m27.107397274s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195234200 (714h20m27.107571831s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195436932 (714h20m27.107774563s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195714016 (714h20m27.108051657s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:45.988724037 (714h20m28.901061668s) [upload=685519]: Next returned nil segment
Apr 5 05:25:45.988736951 (714h20m28.901074582s) [upload=685519]: Waiting for error group managing segments...
Apr 5 05:25:46.278334571 (714h20m29.190672202s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.281965999 (714h20m29.19430363s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.292588509 (714h20m29.204926151s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.302631332 (714h20m29.214968973s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.305482154 (714h20m29.217819785s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:26:13.304943733 (714h20m56.217281364s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:58840->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.304978810 (714h20m56.217316442s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: rpc: dial tcp 145.79.227.42:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305052803 (714h20m56.217390434s) [noise=true , address=145.79.227.28:4729 , piece_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: read tcp 188.166.225.170:46538->145.79.227.28:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305085256 (714h20m56.217422877s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: read tcp 188.166.225.170:58958->145.79.227.39:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305144721 (714h20m56.217482352s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: read tcp 188.166.225.170:44166->145.79.227.3:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305173626 (714h20m56.217511257s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: read tcp 188.166.225.170:45470->145.79.227.29:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305241718 (714h20m56.217579349s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:127L5Mr53vCqKr4gy7N4rAaVCC9FvU8XdYKFrxWrJL728dt3LmR, address:145.79.227.32:4833): protocol: read tcp 188.166.225.170:42334->145.79.227.32:4833: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305260534 (714h20m56.217598165s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1sDggu72xPTG9o2K8sbPHbYfwXarrvqP98YySo7ZSEZ2ddYFEz, address:145.79.227.7:4298): protocol: read tcp 188.166.225.170:38480->145.79.227.7:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305329106 (714h20m56.217666737s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:46896->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305435973 (714h20m56.217773604s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1zDoLfA5bQWM7Uf479gfFVjG3yYvxG5EZS5yZSFGTvoFoPYEAo, address:145.79.227.46:4607): protocol: read tcp 188.166.225.170:49286->145.79.227.46:4607: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305531467 (714h20m56.217869098s) [noise=true , address=145.79.227.18:4010 , piece_id=XC2T7S72SFQ2Z45N7VLGY7CAMBLCZ4MOWU3BALSHACYGOPQPTEWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp, address:145.79.227.18:4010): protocol: read tcp 188.166.225.170:44936->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305549902 (714h20m56.217887534s) [seg_pos={0 0}, upload=685519]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Apr 5 05:26:13.536348486 (714h20m56.448686117s) [upload=685519]: Done uploading segments...
Apr 5 05:26:13.536382360 (714h20m56.448719991s) [upload=685519]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.827060959s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
5.750538133s trace:
https://jaeger.dev.storj.io/trace/5a5341770822a3cc
FAIL (2026-04-05 06:01:11 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
ce_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.192807417 (714h20m27.105145058s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193033373 (714h20m27.105371014s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193198932 (714h20m27.105536564s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193549759 (714h20m27.10588739s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193572203 (714h20m27.105909834s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194022512 (714h20m27.106360143s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194104560 (714h20m27.106442191s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194389039 (714h20m27.10672667s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194717774 (714h20m27.107055415s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194730568 (714h20m27.107068199s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195059643 (714h20m27.107397274s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195234200 (714h20m27.107571831s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195436932 (714h20m27.107774563s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195714016 (714h20m27.108051657s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:45.988724037 (714h20m28.901061668s) [upload=685519]: Next returned nil segment
Apr 5 05:25:45.988736951 (714h20m28.901074582s) [upload=685519]: Waiting for error group managing segments...
Apr 5 05:25:46.278334571 (714h20m29.190672202s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.281965999 (714h20m29.19430363s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.292588509 (714h20m29.204926151s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.302631332 (714h20m29.214968973s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.305482154 (714h20m29.217819785s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:26:13.304943733 (714h20m56.217281364s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:58840->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.304978810 (714h20m56.217316442s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: rpc: dial tcp 145.79.227.42:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305052803 (714h20m56.217390434s) [noise=true , address=145.79.227.28:4729 , piece_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: read tcp 188.166.225.170:46538->145.79.227.28:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305085256 (714h20m56.217422877s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: read tcp 188.166.225.170:58958->145.79.227.39:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305144721 (714h20m56.217482352s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: read tcp 188.166.225.170:44166->145.79.227.3:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305173626 (714h20m56.217511257s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: read tcp 188.166.225.170:45470->145.79.227.29:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305241718 (714h20m56.217579349s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:127L5Mr53vCqKr4gy7N4rAaVCC9FvU8XdYKFrxWrJL728dt3LmR, address:145.79.227.32:4833): protocol: read tcp 188.166.225.170:42334->145.79.227.32:4833: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305260534 (714h20m56.217598165s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1sDggu72xPTG9o2K8sbPHbYfwXarrvqP98YySo7ZSEZ2ddYFEz, address:145.79.227.7:4298): protocol: read tcp 188.166.225.170:38480->145.79.227.7:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305329106 (714h20m56.217666737s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:46896->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305435973 (714h20m56.217773604s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1zDoLfA5bQWM7Uf479gfFVjG3yYvxG5EZS5yZSFGTvoFoPYEAo, address:145.79.227.46:4607): protocol: read tcp 188.166.225.170:49286->145.79.227.46:4607: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305531467 (714h20m56.217869098s) [noise=true , address=145.79.227.18:4010 , piece_id=XC2T7S72SFQ2Z45N7VLGY7CAMBLCZ4MOWU3BALSHACYGOPQPTEWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp, address:145.79.227.18:4010): protocol: read tcp 188.166.225.170:44936->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305549902 (714h20m56.217887534s) [seg_pos={0 0}, upload=685519]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Apr 5 05:26:13.536348486 (714h20m56.448686117s) [upload=685519]: Done uploading segments...
Apr 5 05:26:13.536382360 (714h20m56.448719991s) [upload=685519]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.750563211s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [....................X..............................X........] duration:
3.730565588s trace:
https://jaeger.dev.storj.io/trace/77d099d715fa4f39
FAIL (2026-04-05 06:01:01 UTC)
orbiter warning: context deadline exceeded
Duration: 3.730589964s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
5.686807329s trace:
https://jaeger.dev.storj.io/trace/5a5341770822a3cc
FAIL (2026-04-05 06:00:18 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
ce_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.192807417 (714h20m27.105145058s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193033373 (714h20m27.105371014s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193198932 (714h20m27.105536564s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193549759 (714h20m27.10588739s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193572203 (714h20m27.105909834s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194022512 (714h20m27.106360143s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194104560 (714h20m27.106442191s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194389039 (714h20m27.10672667s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194717774 (714h20m27.107055415s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194730568 (714h20m27.107068199s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195059643 (714h20m27.107397274s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195234200 (714h20m27.107571831s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195436932 (714h20m27.107774563s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195714016 (714h20m27.108051657s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:45.988724037 (714h20m28.901061668s) [upload=685519]: Next returned nil segment
Apr 5 05:25:45.988736951 (714h20m28.901074582s) [upload=685519]: Waiting for error group managing segments...
Apr 5 05:25:46.278334571 (714h20m29.190672202s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.281965999 (714h20m29.19430363s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.292588509 (714h20m29.204926151s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.302631332 (714h20m29.214968973s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.305482154 (714h20m29.217819785s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:26:13.304943733 (714h20m56.217281364s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:58840->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.304978810 (714h20m56.217316442s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: rpc: dial tcp 145.79.227.42:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305052803 (714h20m56.217390434s) [noise=true , address=145.79.227.28:4729 , piece_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: read tcp 188.166.225.170:46538->145.79.227.28:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305085256 (714h20m56.217422877s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: read tcp 188.166.225.170:58958->145.79.227.39:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305144721 (714h20m56.217482352s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: read tcp 188.166.225.170:44166->145.79.227.3:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305173626 (714h20m56.217511257s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: read tcp 188.166.225.170:45470->145.79.227.29:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305241718 (714h20m56.217579349s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:127L5Mr53vCqKr4gy7N4rAaVCC9FvU8XdYKFrxWrJL728dt3LmR, address:145.79.227.32:4833): protocol: read tcp 188.166.225.170:42334->145.79.227.32:4833: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305260534 (714h20m56.217598165s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1sDggu72xPTG9o2K8sbPHbYfwXarrvqP98YySo7ZSEZ2ddYFEz, address:145.79.227.7:4298): protocol: read tcp 188.166.225.170:38480->145.79.227.7:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305329106 (714h20m56.217666737s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:46896->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305435973 (714h20m56.217773604s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1zDoLfA5bQWM7Uf479gfFVjG3yYvxG5EZS5yZSFGTvoFoPYEAo, address:145.79.227.46:4607): protocol: read tcp 188.166.225.170:49286->145.79.227.46:4607: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305531467 (714h20m56.217869098s) [noise=true , address=145.79.227.18:4010 , piece_id=XC2T7S72SFQ2Z45N7VLGY7CAMBLCZ4MOWU3BALSHACYGOPQPTEWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp, address:145.79.227.18:4010): protocol: read tcp 188.166.225.170:44936->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305549902 (714h20m56.217887534s) [seg_pos={0 0}, upload=685519]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Apr 5 05:26:13.536348486 (714h20m56.448686117s) [upload=685519]: Done uploading segments...
Apr 5 05:26:13.536382360 (714h20m56.448719991s) [upload=685519]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.686842287s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.....................X..............................X.......] duration:
4.709865325s trace:
https://jaeger.dev.storj.io/trace/77d099d715fa4f39
FAIL (2026-04-05 06:00:09 UTC)
orbiter warning: context deadline exceeded
Duration: 4.709892759s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
5.781320822s trace:
https://jaeger.dev.storj.io/trace/5a5341770822a3cc
FAIL (2026-04-05 05:59:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
ce_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.192807417 (714h20m27.105145058s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193033373 (714h20m27.105371014s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193198932 (714h20m27.105536564s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193549759 (714h20m27.10588739s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193572203 (714h20m27.105909834s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194022512 (714h20m27.106360143s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194104560 (714h20m27.106442191s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194389039 (714h20m27.10672667s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194717774 (714h20m27.107055415s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194730568 (714h20m27.107068199s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195059643 (714h20m27.107397274s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195234200 (714h20m27.107571831s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195436932 (714h20m27.107774563s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195714016 (714h20m27.108051657s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:45.988724037 (714h20m28.901061668s) [upload=685519]: Next returned nil segment
Apr 5 05:25:45.988736951 (714h20m28.901074582s) [upload=685519]: Waiting for error group managing segments...
Apr 5 05:25:46.278334571 (714h20m29.190672202s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.281965999 (714h20m29.19430363s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.292588509 (714h20m29.204926151s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.302631332 (714h20m29.214968973s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.305482154 (714h20m29.217819785s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:26:13.304943733 (714h20m56.217281364s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:58840->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.304978810 (714h20m56.217316442s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: rpc: dial tcp 145.79.227.42:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305052803 (714h20m56.217390434s) [noise=true , address=145.79.227.28:4729 , piece_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: read tcp 188.166.225.170:46538->145.79.227.28:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305085256 (714h20m56.217422877s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: read tcp 188.166.225.170:58958->145.79.227.39:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305144721 (714h20m56.217482352s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: read tcp 188.166.225.170:44166->145.79.227.3:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305173626 (714h20m56.217511257s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: read tcp 188.166.225.170:45470->145.79.227.29:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305241718 (714h20m56.217579349s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:127L5Mr53vCqKr4gy7N4rAaVCC9FvU8XdYKFrxWrJL728dt3LmR, address:145.79.227.32:4833): protocol: read tcp 188.166.225.170:42334->145.79.227.32:4833: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305260534 (714h20m56.217598165s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1sDggu72xPTG9o2K8sbPHbYfwXarrvqP98YySo7ZSEZ2ddYFEz, address:145.79.227.7:4298): protocol: read tcp 188.166.225.170:38480->145.79.227.7:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305329106 (714h20m56.217666737s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:46896->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305435973 (714h20m56.217773604s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1zDoLfA5bQWM7Uf479gfFVjG3yYvxG5EZS5yZSFGTvoFoPYEAo, address:145.79.227.46:4607): protocol: read tcp 188.166.225.170:49286->145.79.227.46:4607: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305531467 (714h20m56.217869098s) [noise=true , address=145.79.227.18:4010 , piece_id=XC2T7S72SFQ2Z45N7VLGY7CAMBLCZ4MOWU3BALSHACYGOPQPTEWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp, address:145.79.227.18:4010): protocol: read tcp 188.166.225.170:44936->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305549902 (714h20m56.217887534s) [seg_pos={0 0}, upload=685519]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Apr 5 05:26:13.536348486 (714h20m56.448686117s) [upload=685519]: Done uploading segments...
Apr 5 05:26:13.536382360 (714h20m56.448719991s) [upload=685519]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.78134472s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [......................X..............................X......] duration:
3.788124561s trace:
https://jaeger.dev.storj.io/trace/77d099d715fa4f39
FAIL (2026-04-05 05:59:03 UTC)
orbiter warning: context deadline exceeded
Duration: 3.788153547s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
5.731819547s trace:
https://jaeger.dev.storj.io/trace/5a5341770822a3cc
FAIL (2026-04-05 05:58:28 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
ce_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.192807417 (714h20m27.105145058s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193033373 (714h20m27.105371014s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193198932 (714h20m27.105536564s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193549759 (714h20m27.10588739s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193572203 (714h20m27.105909834s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194022512 (714h20m27.106360143s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194104560 (714h20m27.106442191s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194389039 (714h20m27.10672667s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194717774 (714h20m27.107055415s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194730568 (714h20m27.107068199s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195059643 (714h20m27.107397274s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195234200 (714h20m27.107571831s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195436932 (714h20m27.107774563s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195714016 (714h20m27.108051657s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:45.988724037 (714h20m28.901061668s) [upload=685519]: Next returned nil segment
Apr 5 05:25:45.988736951 (714h20m28.901074582s) [upload=685519]: Waiting for error group managing segments...
Apr 5 05:25:46.278334571 (714h20m29.190672202s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.281965999 (714h20m29.19430363s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.292588509 (714h20m29.204926151s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.302631332 (714h20m29.214968973s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.305482154 (714h20m29.217819785s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:26:13.304943733 (714h20m56.217281364s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:58840->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.304978810 (714h20m56.217316442s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: rpc: dial tcp 145.79.227.42:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305052803 (714h20m56.217390434s) [noise=true , address=145.79.227.28:4729 , piece_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: read tcp 188.166.225.170:46538->145.79.227.28:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305085256 (714h20m56.217422877s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: read tcp 188.166.225.170:58958->145.79.227.39:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305144721 (714h20m56.217482352s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: read tcp 188.166.225.170:44166->145.79.227.3:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305173626 (714h20m56.217511257s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: read tcp 188.166.225.170:45470->145.79.227.29:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305241718 (714h20m56.217579349s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:127L5Mr53vCqKr4gy7N4rAaVCC9FvU8XdYKFrxWrJL728dt3LmR, address:145.79.227.32:4833): protocol: read tcp 188.166.225.170:42334->145.79.227.32:4833: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305260534 (714h20m56.217598165s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1sDggu72xPTG9o2K8sbPHbYfwXarrvqP98YySo7ZSEZ2ddYFEz, address:145.79.227.7:4298): protocol: read tcp 188.166.225.170:38480->145.79.227.7:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305329106 (714h20m56.217666737s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:46896->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305435973 (714h20m56.217773604s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1zDoLfA5bQWM7Uf479gfFVjG3yYvxG5EZS5yZSFGTvoFoPYEAo, address:145.79.227.46:4607): protocol: read tcp 188.166.225.170:49286->145.79.227.46:4607: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305531467 (714h20m56.217869098s) [noise=true , address=145.79.227.18:4010 , piece_id=XC2T7S72SFQ2Z45N7VLGY7CAMBLCZ4MOWU3BALSHACYGOPQPTEWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp, address:145.79.227.18:4010): protocol: read tcp 188.166.225.170:44936->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305549902 (714h20m56.217887534s) [seg_pos={0 0}, upload=685519]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Apr 5 05:26:13.536348486 (714h20m56.448686117s) [upload=685519]: Done uploading segments...
Apr 5 05:26:13.536382360 (714h20m56.448719991s) [upload=685519]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.73184718s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.......................X..............................X.....] duration:
3.659968401s trace:
https://jaeger.dev.storj.io/trace/77d099d715fa4f39
FAIL (2026-04-05 05:58:07 UTC)
orbiter warning: context deadline exceeded
Duration: 3.659995233s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [........................X..............................X....] duration:
3.777437103s trace:
https://jaeger.dev.storj.io/trace/77d099d715fa4f39
FAIL (2026-04-05 05:57:52 UTC)
orbiter warning: context deadline exceeded
Duration: 3.777463885s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
5.793011129s trace:
https://jaeger.dev.storj.io/trace/5a5341770822a3cc
FAIL (2026-04-05 05:57:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
ce_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.192807417 (714h20m27.105145058s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193033373 (714h20m27.105371014s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193198932 (714h20m27.105536564s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193549759 (714h20m27.10588739s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193572203 (714h20m27.105909834s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194022512 (714h20m27.106360143s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194104560 (714h20m27.106442191s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194389039 (714h20m27.10672667s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194717774 (714h20m27.107055415s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194730568 (714h20m27.107068199s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195059643 (714h20m27.107397274s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195234200 (714h20m27.107571831s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195436932 (714h20m27.107774563s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195714016 (714h20m27.108051657s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:45.988724037 (714h20m28.901061668s) [upload=685519]: Next returned nil segment
Apr 5 05:25:45.988736951 (714h20m28.901074582s) [upload=685519]: Waiting for error group managing segments...
Apr 5 05:25:46.278334571 (714h20m29.190672202s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.281965999 (714h20m29.19430363s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.292588509 (714h20m29.204926151s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.302631332 (714h20m29.214968973s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.305482154 (714h20m29.217819785s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:26:13.304943733 (714h20m56.217281364s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:58840->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.304978810 (714h20m56.217316442s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: rpc: dial tcp 145.79.227.42:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305052803 (714h20m56.217390434s) [noise=true , address=145.79.227.28:4729 , piece_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: read tcp 188.166.225.170:46538->145.79.227.28:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305085256 (714h20m56.217422877s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: read tcp 188.166.225.170:58958->145.79.227.39:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305144721 (714h20m56.217482352s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: read tcp 188.166.225.170:44166->145.79.227.3:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305173626 (714h20m56.217511257s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: read tcp 188.166.225.170:45470->145.79.227.29:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305241718 (714h20m56.217579349s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:127L5Mr53vCqKr4gy7N4rAaVCC9FvU8XdYKFrxWrJL728dt3LmR, address:145.79.227.32:4833): protocol: read tcp 188.166.225.170:42334->145.79.227.32:4833: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305260534 (714h20m56.217598165s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1sDggu72xPTG9o2K8sbPHbYfwXarrvqP98YySo7ZSEZ2ddYFEz, address:145.79.227.7:4298): protocol: read tcp 188.166.225.170:38480->145.79.227.7:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305329106 (714h20m56.217666737s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:46896->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305435973 (714h20m56.217773604s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1zDoLfA5bQWM7Uf479gfFVjG3yYvxG5EZS5yZSFGTvoFoPYEAo, address:145.79.227.46:4607): protocol: read tcp 188.166.225.170:49286->145.79.227.46:4607: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305531467 (714h20m56.217869098s) [noise=true , address=145.79.227.18:4010 , piece_id=XC2T7S72SFQ2Z45N7VLGY7CAMBLCZ4MOWU3BALSHACYGOPQPTEWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp, address:145.79.227.18:4010): protocol: read tcp 188.166.225.170:44936->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305549902 (714h20m56.217887534s) [seg_pos={0 0}, upload=685519]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Apr 5 05:26:13.536348486 (714h20m56.448686117s) [upload=685519]: Done uploading segments...
Apr 5 05:26:13.536382360 (714h20m56.448719991s) [upload=685519]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.793039605s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
5.771091609s trace:
https://jaeger.dev.storj.io/trace/5a5341770822a3cc
FAIL (2026-04-05 05:57:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
ce_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.192807417 (714h20m27.105145058s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193033373 (714h20m27.105371014s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193198932 (714h20m27.105536564s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193549759 (714h20m27.10588739s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193572203 (714h20m27.105909834s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194022512 (714h20m27.106360143s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194104560 (714h20m27.106442191s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194389039 (714h20m27.10672667s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194717774 (714h20m27.107055415s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194730568 (714h20m27.107068199s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195059643 (714h20m27.107397274s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195234200 (714h20m27.107571831s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195436932 (714h20m27.107774563s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195714016 (714h20m27.108051657s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:45.988724037 (714h20m28.901061668s) [upload=685519]: Next returned nil segment
Apr 5 05:25:45.988736951 (714h20m28.901074582s) [upload=685519]: Waiting for error group managing segments...
Apr 5 05:25:46.278334571 (714h20m29.190672202s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.281965999 (714h20m29.19430363s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.292588509 (714h20m29.204926151s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.302631332 (714h20m29.214968973s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.305482154 (714h20m29.217819785s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:26:13.304943733 (714h20m56.217281364s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:58840->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.304978810 (714h20m56.217316442s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: rpc: dial tcp 145.79.227.42:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305052803 (714h20m56.217390434s) [noise=true , address=145.79.227.28:4729 , piece_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: read tcp 188.166.225.170:46538->145.79.227.28:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305085256 (714h20m56.217422877s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: read tcp 188.166.225.170:58958->145.79.227.39:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305144721 (714h20m56.217482352s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: read tcp 188.166.225.170:44166->145.79.227.3:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305173626 (714h20m56.217511257s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: read tcp 188.166.225.170:45470->145.79.227.29:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305241718 (714h20m56.217579349s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:127L5Mr53vCqKr4gy7N4rAaVCC9FvU8XdYKFrxWrJL728dt3LmR, address:145.79.227.32:4833): protocol: read tcp 188.166.225.170:42334->145.79.227.32:4833: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305260534 (714h20m56.217598165s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1sDggu72xPTG9o2K8sbPHbYfwXarrvqP98YySo7ZSEZ2ddYFEz, address:145.79.227.7:4298): protocol: read tcp 188.166.225.170:38480->145.79.227.7:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305329106 (714h20m56.217666737s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:46896->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305435973 (714h20m56.217773604s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1zDoLfA5bQWM7Uf479gfFVjG3yYvxG5EZS5yZSFGTvoFoPYEAo, address:145.79.227.46:4607): protocol: read tcp 188.166.225.170:49286->145.79.227.46:4607: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305531467 (714h20m56.217869098s) [noise=true , address=145.79.227.18:4010 , piece_id=XC2T7S72SFQ2Z45N7VLGY7CAMBLCZ4MOWU3BALSHACYGOPQPTEWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp, address:145.79.227.18:4010): protocol: read tcp 188.166.225.170:44936->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305549902 (714h20m56.217887534s) [seg_pos={0 0}, upload=685519]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Apr 5 05:26:13.536348486 (714h20m56.448686117s) [upload=685519]: Done uploading segments...
Apr 5 05:26:13.536382360 (714h20m56.448719991s) [upload=685519]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.771111217s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.........................X..............................X...] duration:
3.712155307s trace:
https://jaeger.dev.storj.io/trace/77d099d715fa4f39
FAIL (2026-04-05 05:56:51 UTC)
orbiter warning: context deadline exceeded
Duration: 3.712185926s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
5.751122399s trace:
https://jaeger.dev.storj.io/trace/5a5341770822a3cc
FAIL (2026-04-05 05:56:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
ce_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.192807417 (714h20m27.105145058s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193033373 (714h20m27.105371014s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193198932 (714h20m27.105536564s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193549759 (714h20m27.10588739s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193572203 (714h20m27.105909834s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194022512 (714h20m27.106360143s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194104560 (714h20m27.106442191s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194389039 (714h20m27.10672667s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194717774 (714h20m27.107055415s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194730568 (714h20m27.107068199s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195059643 (714h20m27.107397274s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195234200 (714h20m27.107571831s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195436932 (714h20m27.107774563s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195714016 (714h20m27.108051657s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:45.988724037 (714h20m28.901061668s) [upload=685519]: Next returned nil segment
Apr 5 05:25:45.988736951 (714h20m28.901074582s) [upload=685519]: Waiting for error group managing segments...
Apr 5 05:25:46.278334571 (714h20m29.190672202s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.281965999 (714h20m29.19430363s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.292588509 (714h20m29.204926151s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.302631332 (714h20m29.214968973s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.305482154 (714h20m29.217819785s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:26:13.304943733 (714h20m56.217281364s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:58840->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.304978810 (714h20m56.217316442s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: rpc: dial tcp 145.79.227.42:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305052803 (714h20m56.217390434s) [noise=true , address=145.79.227.28:4729 , piece_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: read tcp 188.166.225.170:46538->145.79.227.28:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305085256 (714h20m56.217422877s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: read tcp 188.166.225.170:58958->145.79.227.39:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305144721 (714h20m56.217482352s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: read tcp 188.166.225.170:44166->145.79.227.3:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305173626 (714h20m56.217511257s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: read tcp 188.166.225.170:45470->145.79.227.29:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305241718 (714h20m56.217579349s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:127L5Mr53vCqKr4gy7N4rAaVCC9FvU8XdYKFrxWrJL728dt3LmR, address:145.79.227.32:4833): protocol: read tcp 188.166.225.170:42334->145.79.227.32:4833: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305260534 (714h20m56.217598165s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1sDggu72xPTG9o2K8sbPHbYfwXarrvqP98YySo7ZSEZ2ddYFEz, address:145.79.227.7:4298): protocol: read tcp 188.166.225.170:38480->145.79.227.7:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305329106 (714h20m56.217666737s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:46896->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305435973 (714h20m56.217773604s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1zDoLfA5bQWM7Uf479gfFVjG3yYvxG5EZS5yZSFGTvoFoPYEAo, address:145.79.227.46:4607): protocol: read tcp 188.166.225.170:49286->145.79.227.46:4607: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305531467 (714h20m56.217869098s) [noise=true , address=145.79.227.18:4010 , piece_id=XC2T7S72SFQ2Z45N7VLGY7CAMBLCZ4MOWU3BALSHACYGOPQPTEWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp, address:145.79.227.18:4010): protocol: read tcp 188.166.225.170:44936->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305549902 (714h20m56.217887534s) [seg_pos={0 0}, upload=685519]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Apr 5 05:26:13.536348486 (714h20m56.448686117s) [upload=685519]: Done uploading segments...
Apr 5 05:26:13.536382360 (714h20m56.448719991s) [upload=685519]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.751148179s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..........................X..............................X..] duration:
3.412052505s trace:
https://jaeger.dev.storj.io/trace/77d099d715fa4f39
FAIL (2026-04-05 05:56:10 UTC)
orbiter warning: context deadline exceeded
Duration: 3.412081831s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
5.684574809s trace:
https://jaeger.dev.storj.io/trace/5a5341770822a3cc
FAIL (2026-04-05 05:55:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
ce_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.192807417 (714h20m27.105145058s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193033373 (714h20m27.105371014s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193198932 (714h20m27.105536564s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193549759 (714h20m27.10588739s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193572203 (714h20m27.105909834s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194022512 (714h20m27.106360143s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194104560 (714h20m27.106442191s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194389039 (714h20m27.10672667s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194717774 (714h20m27.107055415s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194730568 (714h20m27.107068199s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195059643 (714h20m27.107397274s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195234200 (714h20m27.107571831s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195436932 (714h20m27.107774563s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195714016 (714h20m27.108051657s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:45.988724037 (714h20m28.901061668s) [upload=685519]: Next returned nil segment
Apr 5 05:25:45.988736951 (714h20m28.901074582s) [upload=685519]: Waiting for error group managing segments...
Apr 5 05:25:46.278334571 (714h20m29.190672202s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.281965999 (714h20m29.19430363s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.292588509 (714h20m29.204926151s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.302631332 (714h20m29.214968973s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.305482154 (714h20m29.217819785s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:26:13.304943733 (714h20m56.217281364s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:58840->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.304978810 (714h20m56.217316442s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: rpc: dial tcp 145.79.227.42:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305052803 (714h20m56.217390434s) [noise=true , address=145.79.227.28:4729 , piece_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: read tcp 188.166.225.170:46538->145.79.227.28:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305085256 (714h20m56.217422877s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: read tcp 188.166.225.170:58958->145.79.227.39:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305144721 (714h20m56.217482352s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: read tcp 188.166.225.170:44166->145.79.227.3:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305173626 (714h20m56.217511257s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: read tcp 188.166.225.170:45470->145.79.227.29:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305241718 (714h20m56.217579349s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:127L5Mr53vCqKr4gy7N4rAaVCC9FvU8XdYKFrxWrJL728dt3LmR, address:145.79.227.32:4833): protocol: read tcp 188.166.225.170:42334->145.79.227.32:4833: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305260534 (714h20m56.217598165s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1sDggu72xPTG9o2K8sbPHbYfwXarrvqP98YySo7ZSEZ2ddYFEz, address:145.79.227.7:4298): protocol: read tcp 188.166.225.170:38480->145.79.227.7:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305329106 (714h20m56.217666737s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:46896->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305435973 (714h20m56.217773604s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1zDoLfA5bQWM7Uf479gfFVjG3yYvxG5EZS5yZSFGTvoFoPYEAo, address:145.79.227.46:4607): protocol: read tcp 188.166.225.170:49286->145.79.227.46:4607: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305531467 (714h20m56.217869098s) [noise=true , address=145.79.227.18:4010 , piece_id=XC2T7S72SFQ2Z45N7VLGY7CAMBLCZ4MOWU3BALSHACYGOPQPTEWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp, address:145.79.227.18:4010): protocol: read tcp 188.166.225.170:44936->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305549902 (714h20m56.217887534s) [seg_pos={0 0}, upload=685519]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Apr 5 05:26:13.536348486 (714h20m56.448686117s) [upload=685519]: Done uploading segments...
Apr 5 05:26:13.536382360 (714h20m56.448719991s) [upload=685519]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.684604558s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...........................X..............................X.] duration:
3.473543385s trace:
https://jaeger.dev.storj.io/trace/77d099d715fa4f39
FAIL (2026-04-05 05:55:25 UTC)
orbiter warning: context deadline exceeded
Duration: 3.473574255s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
5.743466676s trace:
https://jaeger.dev.storj.io/trace/5a5341770822a3cc
FAIL (2026-04-05 05:54:55 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
ce_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.192807417 (714h20m27.105145058s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193033373 (714h20m27.105371014s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193198932 (714h20m27.105536564s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193549759 (714h20m27.10588739s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193572203 (714h20m27.105909834s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194022512 (714h20m27.106360143s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194104560 (714h20m27.106442191s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194389039 (714h20m27.10672667s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194717774 (714h20m27.107055415s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194730568 (714h20m27.107068199s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195059643 (714h20m27.107397274s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195234200 (714h20m27.107571831s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195436932 (714h20m27.107774563s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195714016 (714h20m27.108051657s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:45.988724037 (714h20m28.901061668s) [upload=685519]: Next returned nil segment
Apr 5 05:25:45.988736951 (714h20m28.901074582s) [upload=685519]: Waiting for error group managing segments...
Apr 5 05:25:46.278334571 (714h20m29.190672202s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.281965999 (714h20m29.19430363s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.292588509 (714h20m29.204926151s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.302631332 (714h20m29.214968973s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.305482154 (714h20m29.217819785s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:26:13.304943733 (714h20m56.217281364s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:58840->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.304978810 (714h20m56.217316442s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: rpc: dial tcp 145.79.227.42:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305052803 (714h20m56.217390434s) [noise=true , address=145.79.227.28:4729 , piece_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: read tcp 188.166.225.170:46538->145.79.227.28:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305085256 (714h20m56.217422877s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: read tcp 188.166.225.170:58958->145.79.227.39:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305144721 (714h20m56.217482352s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: read tcp 188.166.225.170:44166->145.79.227.3:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305173626 (714h20m56.217511257s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: read tcp 188.166.225.170:45470->145.79.227.29:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305241718 (714h20m56.217579349s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:127L5Mr53vCqKr4gy7N4rAaVCC9FvU8XdYKFrxWrJL728dt3LmR, address:145.79.227.32:4833): protocol: read tcp 188.166.225.170:42334->145.79.227.32:4833: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305260534 (714h20m56.217598165s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1sDggu72xPTG9o2K8sbPHbYfwXarrvqP98YySo7ZSEZ2ddYFEz, address:145.79.227.7:4298): protocol: read tcp 188.166.225.170:38480->145.79.227.7:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305329106 (714h20m56.217666737s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:46896->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305435973 (714h20m56.217773604s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1zDoLfA5bQWM7Uf479gfFVjG3yYvxG5EZS5yZSFGTvoFoPYEAo, address:145.79.227.46:4607): protocol: read tcp 188.166.225.170:49286->145.79.227.46:4607: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305531467 (714h20m56.217869098s) [noise=true , address=145.79.227.18:4010 , piece_id=XC2T7S72SFQ2Z45N7VLGY7CAMBLCZ4MOWU3BALSHACYGOPQPTEWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp, address:145.79.227.18:4010): protocol: read tcp 188.166.225.170:44936->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305549902 (714h20m56.217887534s) [seg_pos={0 0}, upload=685519]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Apr 5 05:26:13.536348486 (714h20m56.448686117s) [upload=685519]: Done uploading segments...
Apr 5 05:26:13.536382360 (714h20m56.448719991s) [upload=685519]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.743492666s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
5.764918598s trace:
https://jaeger.dev.storj.io/trace/5a5341770822a3cc
FAIL (2026-04-05 05:54:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
ce_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.192807417 (714h20m27.105145058s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193033373 (714h20m27.105371014s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193198932 (714h20m27.105536564s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193549759 (714h20m27.10588739s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193572203 (714h20m27.105909834s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194022512 (714h20m27.106360143s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194104560 (714h20m27.106442191s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194389039 (714h20m27.10672667s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194717774 (714h20m27.107055415s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194730568 (714h20m27.107068199s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195059643 (714h20m27.107397274s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195234200 (714h20m27.107571831s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195436932 (714h20m27.107774563s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195714016 (714h20m27.108051657s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:45.988724037 (714h20m28.901061668s) [upload=685519]: Next returned nil segment
Apr 5 05:25:45.988736951 (714h20m28.901074582s) [upload=685519]: Waiting for error group managing segments...
Apr 5 05:25:46.278334571 (714h20m29.190672202s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.281965999 (714h20m29.19430363s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.292588509 (714h20m29.204926151s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.302631332 (714h20m29.214968973s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.305482154 (714h20m29.217819785s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:26:13.304943733 (714h20m56.217281364s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:58840->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.304978810 (714h20m56.217316442s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: rpc: dial tcp 145.79.227.42:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305052803 (714h20m56.217390434s) [noise=true , address=145.79.227.28:4729 , piece_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: read tcp 188.166.225.170:46538->145.79.227.28:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305085256 (714h20m56.217422877s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: read tcp 188.166.225.170:58958->145.79.227.39:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305144721 (714h20m56.217482352s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: read tcp 188.166.225.170:44166->145.79.227.3:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305173626 (714h20m56.217511257s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: read tcp 188.166.225.170:45470->145.79.227.29:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305241718 (714h20m56.217579349s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:127L5Mr53vCqKr4gy7N4rAaVCC9FvU8XdYKFrxWrJL728dt3LmR, address:145.79.227.32:4833): protocol: read tcp 188.166.225.170:42334->145.79.227.32:4833: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305260534 (714h20m56.217598165s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1sDggu72xPTG9o2K8sbPHbYfwXarrvqP98YySo7ZSEZ2ddYFEz, address:145.79.227.7:4298): protocol: read tcp 188.166.225.170:38480->145.79.227.7:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305329106 (714h20m56.217666737s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:46896->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305435973 (714h20m56.217773604s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1zDoLfA5bQWM7Uf479gfFVjG3yYvxG5EZS5yZSFGTvoFoPYEAo, address:145.79.227.46:4607): protocol: read tcp 188.166.225.170:49286->145.79.227.46:4607: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305531467 (714h20m56.217869098s) [noise=true , address=145.79.227.18:4010 , piece_id=XC2T7S72SFQ2Z45N7VLGY7CAMBLCZ4MOWU3BALSHACYGOPQPTEWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp, address:145.79.227.18:4010): protocol: read tcp 188.166.225.170:44936->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305549902 (714h20m56.217887534s) [seg_pos={0 0}, upload=685519]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Apr 5 05:26:13.536348486 (714h20m56.448686117s) [upload=685519]: Done uploading segments...
Apr 5 05:26:13.536382360 (714h20m56.448719991s) [upload=685519]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.764939217s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [............................X..............................X] duration:
5.001047579s trace:
https://jaeger.dev.storj.io/trace/77d099d715fa4f39
FAIL (2026-04-05 05:54:15 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001067137s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
5.711125076s trace:
https://jaeger.dev.storj.io/trace/5a5341770822a3cc
FAIL (2026-04-05 05:53:40 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
ce_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.192807417 (714h20m27.105145058s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193033373 (714h20m27.105371014s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193198932 (714h20m27.105536564s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193549759 (714h20m27.10588739s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193572203 (714h20m27.105909834s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194022512 (714h20m27.106360143s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194104560 (714h20m27.106442191s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194389039 (714h20m27.10672667s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194717774 (714h20m27.107055415s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194730568 (714h20m27.107068199s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195059643 (714h20m27.107397274s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195234200 (714h20m27.107571831s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195436932 (714h20m27.107774563s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195714016 (714h20m27.108051657s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:45.988724037 (714h20m28.901061668s) [upload=685519]: Next returned nil segment
Apr 5 05:25:45.988736951 (714h20m28.901074582s) [upload=685519]: Waiting for error group managing segments...
Apr 5 05:25:46.278334571 (714h20m29.190672202s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.281965999 (714h20m29.19430363s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.292588509 (714h20m29.204926151s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.302631332 (714h20m29.214968973s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.305482154 (714h20m29.217819785s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:26:13.304943733 (714h20m56.217281364s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:58840->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.304978810 (714h20m56.217316442s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: rpc: dial tcp 145.79.227.42:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305052803 (714h20m56.217390434s) [noise=true , address=145.79.227.28:4729 , piece_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: read tcp 188.166.225.170:46538->145.79.227.28:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305085256 (714h20m56.217422877s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: read tcp 188.166.225.170:58958->145.79.227.39:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305144721 (714h20m56.217482352s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: read tcp 188.166.225.170:44166->145.79.227.3:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305173626 (714h20m56.217511257s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: read tcp 188.166.225.170:45470->145.79.227.29:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305241718 (714h20m56.217579349s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:127L5Mr53vCqKr4gy7N4rAaVCC9FvU8XdYKFrxWrJL728dt3LmR, address:145.79.227.32:4833): protocol: read tcp 188.166.225.170:42334->145.79.227.32:4833: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305260534 (714h20m56.217598165s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1sDggu72xPTG9o2K8sbPHbYfwXarrvqP98YySo7ZSEZ2ddYFEz, address:145.79.227.7:4298): protocol: read tcp 188.166.225.170:38480->145.79.227.7:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305329106 (714h20m56.217666737s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:46896->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305435973 (714h20m56.217773604s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1zDoLfA5bQWM7Uf479gfFVjG3yYvxG5EZS5yZSFGTvoFoPYEAo, address:145.79.227.46:4607): protocol: read tcp 188.166.225.170:49286->145.79.227.46:4607: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305531467 (714h20m56.217869098s) [noise=true , address=145.79.227.18:4010 , piece_id=XC2T7S72SFQ2Z45N7VLGY7CAMBLCZ4MOWU3BALSHACYGOPQPTEWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp, address:145.79.227.18:4010): protocol: read tcp 188.166.225.170:44936->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305549902 (714h20m56.217887534s) [seg_pos={0 0}, upload=685519]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Apr 5 05:26:13.536348486 (714h20m56.448686117s) [upload=685519]: Done uploading segments...
Apr 5 05:26:13.536382360 (714h20m56.448719991s) [upload=685519]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.711141447s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.............................X..............................] duration:
3.39006149s trace:
https://jaeger.dev.storj.io/trace/6879597f5e6bc99f
FAIL (2026-04-05 05:53:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.390084655s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..............................X.............................] duration:
3.430349902s trace:
https://jaeger.dev.storj.io/trace/6879597f5e6bc99f
FAIL (2026-04-05 05:52:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.430374049s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
5.733800466s trace:
https://jaeger.dev.storj.io/trace/5a5341770822a3cc
FAIL (2026-04-05 05:52:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
ce_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.192807417 (714h20m27.105145058s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193033373 (714h20m27.105371014s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193198932 (714h20m27.105536564s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193549759 (714h20m27.10588739s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.193572203 (714h20m27.105909834s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194022512 (714h20m27.106360143s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194104560 (714h20m27.106442191s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194389039 (714h20m27.10672667s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194717774 (714h20m27.107055415s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.194730568 (714h20m27.107068199s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195059643 (714h20m27.107397274s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195234200 (714h20m27.107571831s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195436932 (714h20m27.107774563s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:44.195714016 (714h20m27.108051657s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Uploading piece...
Apr 5 05:25:45.988724037 (714h20m28.901061668s) [upload=685519]: Next returned nil segment
Apr 5 05:25:45.988736951 (714h20m28.901074582s) [upload=685519]: Waiting for error group managing segments...
Apr 5 05:25:46.278334571 (714h20m29.190672202s) [noise=true , address=77.90.31.12:4589 , piece_id=5EIZZQVG4OL56UVOWYVJFMCKN57KGCLVPPNVLX7A55CWMO3MG7HQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.281965999 (714h20m29.19430363s) [noise=true , address=45.91.116.7:4012 , piece_id=K3P4CE47CFI7WDCF2IQSMGI4QYO3OAD22IR4L7NM7DDUCUUAFI5A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.292588509 (714h20m29.204926151s) [noise=true , address=45.91.116.14:4196 , piece_id=BYNHXQ7TDLJTVRCQYQEKDUBYKN246C5U3RKYKUUPPYCKFRFWNT6A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.302631332 (714h20m29.214968973s) [noise=true , address=45.91.116.8:4048 , piece_id=GSZIBQCJMZLV67ZMOBKYKDNZL47QOPUYVCWDVWETHFQYSLNVQUHQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:25:46.305482154 (714h20m29.217819785s) [noise=true , address=77.90.31.14:4665 , piece_id=T3IBLI4ZIWVVES66B2R7WRRZP6D5HTHPCAK5BVPYFB5KMUJV2KDA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: <nil>
Apr 5 05:26:13.304943733 (714h20m56.217281364s) [noise=true , address=145.79.227.22:4511 , piece_id=FWYY2WKKRPGVKZUCMBVYOWP6DWBRE6QHSWKLYNRS73OWLMG5Q72A, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12AbLS96MKBYqYykQQJbAFHhfcmturggwPPCsNZ4oGzyMNugVfG, address:145.79.227.22:4511): protocol: read tcp 188.166.225.170:58840->145.79.227.22:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.304978810 (714h20m56.217316442s) [noise=true , address=145.79.227.42:4121 , piece_id=DYXFXHC6IEEUAVY7AQLGV6JFFZFVFRA376KBQVYDHEKWYKQ4GEBA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: rpc: dial tcp 145.79.227.42:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305052803 (714h20m56.217390434s) [noise=true , address=145.79.227.28:4729 , piece_id=TE3NYZGTDXCUMJ74EZ3K5BCOIDN4S4UHDQOWND2CZDREAXIS7RFQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: read tcp 188.166.225.170:46538->145.79.227.28:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305085256 (714h20m56.217422877s) [noise=true , address=145.79.227.39:4014 , piece_id=EODV6OPZIWAO63WX4FHD3FZVEIFWRS6ZRTAAO7OEHIASYV3X73EQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: read tcp 188.166.225.170:58958->145.79.227.39:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305144721 (714h20m56.217482352s) [noise=true , address=145.79.227.3:4149 , piece_id=CLGAYBSQB26AWGE2XAH4YD25XLZF7I2VOD7BZVTS7MHZG4JIHAWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:145.79.227.3:4149): protocol: read tcp 188.166.225.170:44166->145.79.227.3:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305173626 (714h20m56.217511257s) [noise=true , address=145.79.227.29:4943 , piece_id=57Z3LYZRMAHOHJMZLX65R37RVJ7O2WDVRNBAVV7EMQYWAS7IOLCQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: read tcp 188.166.225.170:45470->145.79.227.29:4943: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305241718 (714h20m56.217579349s) [noise=true , address=145.79.227.32:4833 , piece_id=H2MW3GELCOWYZZPV5D4YWWIGCXVZMCJVFDQHS2ETHQR6HQSQUUIA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:127L5Mr53vCqKr4gy7N4rAaVCC9FvU8XdYKFrxWrJL728dt3LmR, address:145.79.227.32:4833): protocol: read tcp 188.166.225.170:42334->145.79.227.32:4833: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305260534 (714h20m56.217598165s) [noise=true , address=145.79.227.7:4298 , piece_id=6GXVZOGMMH56AFQ6U7CEBEBT3632GD754NAN2SBN76ZYKOXPOBWQ, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1sDggu72xPTG9o2K8sbPHbYfwXarrvqP98YySo7ZSEZ2ddYFEz, address:145.79.227.7:4298): protocol: read tcp 188.166.225.170:38480->145.79.227.7:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305329106 (714h20m56.217666737s) [noise=true , address=145.79.227.23:4545 , piece_id=OS3XJOHYHNBEADQKIAY5EQSNSJEUGELU3OBSQ77QPVG6KT6XHAFA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: read tcp 188.166.225.170:46896->145.79.227.23:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305435973 (714h20m56.217773604s) [noise=true , address=145.79.227.46:4607 , piece_id=GUKCXM3G4EXJ5P2NRQNXHD5QL6RGFUQ6HMN2GNUMIFHLC46U2USA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:1zDoLfA5bQWM7Uf479gfFVjG3yYvxG5EZS5yZSFGTvoFoPYEAo, address:145.79.227.46:4607): protocol: read tcp 188.166.225.170:49286->145.79.227.46:4607: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305531467 (714h20m56.217869098s) [noise=true , address=145.79.227.18:4010 , piece_id=XC2T7S72SFQ2Z45N7VLGY7CAMBLCZ4MOWU3BALSHACYGOPQPTEWA, seg_pos={0 0}, upload=685519]: Done uploading piece. err: ecclient: upload failed (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp, address:145.79.227.18:4010): protocol: read tcp 188.166.225.170:44936->145.79.227.18:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 5 05:26:13.305549902 (714h20m56.217887534s) [seg_pos={0 0}, upload=685519]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Apr 5 05:26:13.536348486 (714h20m56.448686117s) [upload=685519]: Done uploading segments...
Apr 5 05:26:13.536382360 (714h20m56.448719991s) [upload=685519]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.733826397s
core
uplink
us-select-1
|