Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.........................................................X..] duration:
3.46081649s trace:
https://jaeger.dev.storj.io/trace/10f04c3c775dba0
FAIL (2026-04-11 01:38:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.460841749s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..........................................................X.] duration:
3.428248262s trace:
https://jaeger.dev.storj.io/trace/10f04c3c775dba0
FAIL (2026-04-11 01:38:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.42826726s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...........................................................X] duration:
5.00154645s trace:
https://jaeger.dev.storj.io/trace/10f04c3c775dba0
FAIL (2026-04-11 01:37:43 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001568453s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [X...........................................................] duration:
3.954591013s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 01:24:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.95476084s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.X..........................................................] duration:
3.611782546s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 01:23:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.611804818s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [X.X.........................................................] duration:
3.521381774s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 01:22:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.521402283s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
5.973714134s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 01:22:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.973741527s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.X.X........................................................] duration:
3.625894934s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 01:22:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.625930124s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
5.743994555s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 01:21:58 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.74403816s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [XX..........................................................] duration:
6.179387645s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 01:21:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 6.179412985s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [X.X.X.......................................................] duration:
3.535108041s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 01:21:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.53513332s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.XX.........................................................] duration:
7.185238367s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 01:21:16 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 7.185264348s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
5.646642794s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 01:21:13 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.64666657s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [XX.X.X......................................................] duration:
3.423110345s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 01:20:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.423135755s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
6.6034563s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 01:20:16 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.603480979s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..XX........................................................] duration:
5.811264771s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 01:20:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.811289067s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.XX.X.X.....................................................] duration:
3.523781513s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 01:19:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.52380527s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
5.883128138s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 01:19:24 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.883384574s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...XX.......................................................] duration:
5.674911759s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 01:19:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.674924505s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..XX.X.X....................................................] duration:
3.50938071s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 01:18:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.509405508s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
5.816897442s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 01:18:40 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.816919104s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....XX......................................................] duration:
5.815554276s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 01:18:20 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.81557756s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....XX.....................................................] duration:
5.78195552s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 01:17:56 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.781973404s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...XX.X.X...................................................] duration:
3.436453683s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 01:17:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.436479102s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
6.577171118s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 01:17:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.577195276s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......XX....................................................] duration:
5.767369944s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 01:17:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.767394672s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
6.74177319s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 01:17:27 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.741794712s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [....XX.X.X..................................................] duration:
3.631477152s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 01:17:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.631502661s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
5.771964411s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 01:16:36 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.771983378s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......XX...................................................] duration:
5.679840889s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 01:16:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.679869405s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
6.629120779s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 01:16:25 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.629143422s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
5.710460531s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 01:16:09 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.710483415s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
6.671965526s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 01:16:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.671981137s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.....XX.X.X.................................................] duration:
3.590204049s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 01:15:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.590229747s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
6.101276449s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 01:15:50 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.101317229s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........XX..................................................] duration:
5.663011489s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 01:15:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.663037669s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
6.178636643s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 01:15:18 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 6.178659787s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [......XX.X.X................................................] duration:
3.522679531s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 01:14:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.522716102s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
5.445299172s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 01:14:48 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.445324332s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
5.820275989s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 01:14:47 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.820289985s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........XX.................................................] duration:
5.733423623s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 01:14:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.733448522s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.......XX.X.X...............................................] duration:
3.602425849s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 01:14:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.602453432s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........XX................................................] duration:
5.766754261s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 01:14:11 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.76677443s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
5.62834946s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 01:14:10 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.628374109s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [........XX.X.X..............................................] duration:
3.472212214s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 01:13:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.472227454s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
5.663265007s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 01:13:49 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.663288172s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
5.504084286s trace:
https://jaeger.dev.storj.io/trace/7bf7d700005c5ffc
FAIL (2026-04-11 01:13:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 12)
======LOGS======
1 00:35:02.102980102 (80h48m48.922216136s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Apr 11 00:35:02.102988498 (80h48m48.922224532s) [upload=77643]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Apr 11 00:35:02.347014884 (80h48m49.166250918s) [seg_pos={0 0}, upload=77643]: Begin upload segment...
Apr 11 00:35:02.347096521 (80h48m49.166332555s) [seg_pos={0 0}, upload=77643]: Done begin upload segment.
Apr 11 00:35:02.347134595 (80h48m49.166370629s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347219359 (80h48m49.166455393s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347397994 (80h48m49.166634028s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.347951241 (80h48m49.167187295s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.348403333 (80h48m49.167639377s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.349580416 (80h48m49.16881644s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350222825 (80h48m49.169458859s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.350716198 (80h48m49.169952232s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351232323 (80h48m49.170468358s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.351725215 (80h48m49.170961259s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352236692 (80h48m49.171472726s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.352725695 (80h48m49.171961729s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.353282871 (80h48m49.172518905s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.355150295 (80h48m49.174386329s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356256440 (80h48m49.175492474s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.356714043 (80h48m49.175950077s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.358291207 (80h48m49.177527241s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:02.360085730 (80h48m49.179321764s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Uploading piece...
Apr 11 00:35:03.644395589 (80h48m50.463631623s) [upload=77643]: Next returned nil segment
Apr 11 00:35:03.644419985 (80h48m50.463656019s) [upload=77643]: Waiting for error group managing segments...
Apr 11 00:35:04.076514972 (80h48m50.895751006s) [noise=true , address=193.149.164.71:4296 , piece_id=ICYYOFCZH7TADQFI2TGSPCEJXX4WZELENPWM62K2ZMF2K2YO566Q, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087396093 (80h48m50.906632127s) [noise=true , address=193.149.164.70:4132 , piece_id=CGIAUTQFH6QFLO5NQBTRFUKEJP4V35ERVC5YMWIM7TFF7U7CWOSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.087802908 (80h48m50.907038942s) [noise=true , address=64.49.13.26:4948 , piece_id=6YCV7WXCWU47K7SWDZPRMRMQQGDPEDUJSPGE2N2JATBLHRHJ42UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.101188451 (80h48m50.920424475s) [noise=true , address=64.49.13.137:4503 , piece_id=52NNEHEWZWCQWFIFDX4DFYA4MWMRRASOLYLHDPXXT3GVRQT5S4UA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.129414164 (80h48m50.948650208s) [noise=true , address=193.149.164.67:4007 , piece_id=XJSSKWPSBIUORDJ2OC64EOK6JOBLIXSP7MQYKSGGJQNLW7NSAH3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.130587098 (80h48m50.949823132s) [noise=true , address=193.149.164.68:4055 , piece_id=7HP27JRNWZEBXVPI6I4BV5YBBUH2WM5CH2MEFAMWY5GQ6FQSRBOQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.145662963 (80h48m50.964898997s) [noise=true , address=64.49.13.170:4719 , piece_id=RSUNMVCLMQH7DIECFJCHZXMM4NOK3RJXDMK66B7GX2KTZ2W6SGXQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.148310473 (80h48m50.967546507s) [noise=true , address=193.149.164.69:4069 , piece_id=FUIABW2DEM66WHOM3FDNFXOVQUOVFMEBXRXGZFGYDFMHY2SCJ2KQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.253409518 (80h48m51.072645562s) [noise=true , address=45.91.116.13:4186 , piece_id=UJJE5F7N74ME5E7OBBXTA5KEPMFNY4WZ4AHSOX4HRNWYFNBPYBSA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.320968782 (80h48m51.140204816s) [noise=true , address=77.90.31.15:4723 , piece_id=Z4EC4PUVABSBPPK56JR3IB4ULAAYOXIPLHTWGNJEMBLQRX4KQRTA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:04.385096245 (80h48m51.204332289s) [noise=true , address=45.91.116.7:4028 , piece_id=A3JVIG2WKBOAVQJXWET3YCUOD5WGIGCYRQYHJDZOMABMUA6RXWHQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:35:05.729858372 (80h48m52.549094406s) [noise=true , address=77.90.31.17:4787 , piece_id=BM62FJFUV3OICUAGDRZL4SVDWJBVRFXXBMSCWVXGK2MZEOFJ5GIQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: <nil>
Apr 11 00:36:01.437011024 (80h49m48.256247048s) [noise=true , address=145.79.227.37:4584 , piece_id=PJHMI7QYRI3A7WZAEUM2ONTS4IHN62QBRL6MAFWWYGJHP3DBNN3A, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12uS1kJzcCiNR4GwBxZqkGoSoGYJCDnwUPMpJLn6KrPE59Hq2S, address:145.79.227.37:4584): protocol: rpc: dial tcp 145.79.227.37:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437062985 (80h49m48.256299009s) [noise=true , address=145.79.227.33:4875 , piece_id=Q6WMCYRBOLPPTLTRWLLRNFDHGTY2NE3J3KEYGI6U4YEMXB2XKCFQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:145.79.227.33:4875): protocol: rpc: dial tcp 145.79.227.33:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437204669 (80h49m48.256440693s) [noise=true , address=145.79.227.46:4625 , piece_id=CDFOLHPDRI4DVXHEHB46WWUSR3VRLYGVPQIQO7N2FMONQ2QNPELQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1qZcuemXhaKWsit4JtJ9MoYjwECsHviNrdAzTbWrPQADn4RbGP, address:145.79.227.46:4625): protocol: rpc: dial tcp 145.79.227.46:4625: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437311625 (80h49m48.256547649s) [noise=true , address=145.79.227.13:4824 , piece_id=2FRAED3YTIE65VE72MLIUFDU72JBTHL27QDRE7SK6RIO7PBKYWBA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:128RwAx77X1bmYyNBtGGMbUUKtvpZjtSBRK2M7AKtebEP4TY4wm, address:145.79.227.13:4824): protocol: rpc: dial tcp 145.79.227.13:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437529275 (80h49m48.256765299s) [noise=true , address=145.79.227.43:4169 , piece_id=U3B2KD66A5Z4SUJAKBYZWUQEJBCGIVGDDAKT74XBCXPVDN3DMQHA, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:145.79.227.43:4169): protocol: rpc: dial tcp 145.79.227.43:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437574683 (80h49m48.256810717s) [noise=true , address=145.79.227.35:4490 , piece_id=QWHI2B4W3HDXOI2JHHQSQ2BD2CMRNDCASZNSXUQNTOUKI4VZB6VQ, seg_pos={0 0}, upload=77643]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:145.79.227.35:4490): protocol: rpc: dial tcp 145.79.227.35:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:36:01.437616122 (80h49m48.256852156s) [seg_pos={0 0}, upload=77643]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil>
Apr 11 00:36:01.715269026 (80h49m48.53450508s) [upload=77643]: Done uploading segments...
Apr 11 00:36:01.715313421 (80h49m48.534549455s) [upload=77643]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12)
Duration: 5.504110838s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
6.775523894s trace:
https://jaeger.dev.storj.io/trace/2ead608de6f962d2
FAIL (2026-04-11 01:13:04 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.775546107s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........XX...............................................] duration:
5.748981223s trace:
https://jaeger.dev.storj.io/trace/276b2e32db11ea61
FAIL (2026-04-11 01:13:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 1)
======LOGS======
BBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446530549 (80h48m57.265766573s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.446959266 (80h48m57.26619529s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447310023 (80h48m57.266546047s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447634449 (80h48m57.266870473s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.447903609 (80h48m57.267139633s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448200061 (80h48m57.267436085s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.448524397 (80h48m57.267760431s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.449987020 (80h48m57.269223044s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.450280857 (80h48m57.269516882s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.451699786 (80h48m57.27093581s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.452991929 (80h48m57.272227953s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.454275917 (80h48m57.273511941s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.455479750 (80h48m57.274715775s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:10.456609872 (80h48m57.275845906s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Uploading piece...
Apr 11 00:35:12.280983620 (80h48m59.100219644s) [upload=77646]: Next returned nil segment
Apr 11 00:35:12.280999701 (80h48m59.100235735s) [upload=77646]: Waiting for error group managing segments...
Apr 11 00:35:14.470144773 (80h49m1.289380797s) [noise=true , address=77.90.31.17:4769 , piece_id=KSZRPYQ4245Q5HHYIGREJGRQFFIX7II3G25VQL3QUKGMQACUX4PQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: <nil>
Apr 11 00:35:39.550189016 (80h49m26.36942505s) [noise=true , address=145.79.227.41:4087 , piece_id=UGSA2KGDKKDUKH56ZRMGWBFIE3GVJSVSXJM7J5UOXTVD3Z7BXLWQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:145.79.227.41:4087): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550223232 (80h49m26.369459256s) [noise=true , address=145.79.227.27:4692 , piece_id=NRPL7ABNY5VOBM3CCV564NOJUYBVEVS6RP7UKSOYCJ3E2QJ55IUA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1n3SDzXGgXbvYBGtLdND9oy93AHNFokRnF6Yv3EvoMC9UgN2H9, address:145.79.227.27:4692): protocol: rpc: dial tcp 145.79.227.27:4692: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550359355 (80h49m26.369595379s) [noise=true , address=145.79.227.13:4804 , piece_id=GUJNCXOBUHKIFXVKUIGLOWKBTNEHSRPT2W4BXJDENUOH5TA6JFNA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:145.79.227.13:4804): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550423339 (80h49m26.369659363s) [noise=true , address=145.79.227.10:4407 , piece_id=R6QCBNR6SDAL3Z6OV5BW5IDH3WPVEVNRITQYOP27QAYZFPEKXDDQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1kJKjEfzwg1wuBUP2Bw4oaSoy6mQZdvDED6PpkbrqQwbkh3bUW, address:145.79.227.10:4407): protocol: rpc: dial tcp 145.79.227.10:4407: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550507481 (80h49m26.369743505s) [noise=true , address=145.79.227.26:4653 , piece_id=UXV2X7X5NCDJCF2KWMFSZYJ4OL3WPZTK2SBXIAIGXPCKNE4SMJ3Q, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ, address:145.79.227.26:4653): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550602244 (80h49m26.369838268s) [noise=true , address=145.79.227.4:4189 , piece_id=X6FDGNOFONSJ2NX6Y5BQ66M2YFPB4G626AHD3IWIYVMLH33ZO3UA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:145.79.227.4:4189): protocol: rpc: dial tcp 145.79.227.4:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550702908 (80h49m26.369938932s) [noise=true , address=145.79.227.8:4332 , piece_id=F2OTPJR2GQW3DGMPS4MILL7GJ7TXJ7YFUFBBWVUOB2MYAYVTHDAA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1khNvxTmr2av1PFmxCv4cEcMv25F9y41bndYPQKuP2LCx7WoMQ, address:145.79.227.8:4332): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.550756361 (80h49m26.369992385s) [noise=true , address=145.79.227.39:4015 , piece_id=MGIMNB3YUJLI6OI5EBZOD655WNRLUZC3YY52ZTA6SB2DAFSZZ2TQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: rpc: dial tcp 145.79.227.39:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550854590 (80h49m26.370090614s) [noise=true , address=145.79.227.28:4729 , piece_id=YSYFJAPUJUMBHWNJN4EAITQ4HYQ2BUGGHM6UMY6X34GQT3JWC6QQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12ruoUvv6Q9UoRJwugnxbUREY1rjuCipHFtteYbwg5HYCHDXQ3L, address:145.79.227.28:4729): protocol: rpc: dial tcp 145.79.227.28:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550897464 (80h49m26.370133498s) [noise=true , address=145.79.227.52:4445 , piece_id=UESFHZZHC4APJUVTEQNFS6W7SUAUMIWOM5IARKF6UGDGVXQZ3EYA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12vXrkhLqCofhR4Fmfqb98gxnqWQF6vMV4q3ATbRvPfcqi82WSo, address:145.79.227.52:4445): protocol: rpc: dial tcp 145.79.227.52:4445: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550956588 (80h49m26.370192612s) [noise=true , address=145.79.227.12:4770 , piece_id=SCPQOYZQV2NAYYYTUW64J36FYH5SLCVBS4ZME6OLFCPM72UALPBQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.550985494 (80h49m26.370221518s) [noise=true , address=145.79.227.15:4873 , piece_id=XW5BM3OMNITH5HL2S6EUADYD2ZIW4AHZGROMBZCIX6TF7IVFOK6A, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:145.79.227.15:4873): protocol: rpc: dial tcp 145.79.227.15:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Apr 11 00:35:39.551000572 (80h49m26.370236596s) [noise=true , address=145.79.227.14:4839 , piece_id=QBQZJPH3ND4CFJXFOPLBI2W3OJ75DRNCWYBIF6N3Z757BEEBJJEA, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551084434 (80h49m26.370320469s) [noise=true , address=145.79.227.22:4513 , piece_id=ADDMI4UFZULT4KYF26R7R37LBG5IELKP57UY5LLB7QTL4DSSKFLQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12BykEXT7spgn4jkPucFmPvjvgKPKi5fZgRAeMTbfecZ7jCNori, address:145.79.227.22:4513): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551180850 (80h49m26.370416874s) [noise=true , address=145.79.227.20:4441 , piece_id=VWOXNGMUDFI5XKTTLBNMXOXXIDD4PGABJOXUIMG2QAX4ISNJP2CQ, seg_pos={0 0}, upload=77646]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:145.79.227.20:4441): protocol: context deadline exceeded; context deadline exceeded
Apr 11 00:35:39.551213854 (80h49m26.370449878s) [seg_pos={0 0}, upload=77646]: Done waiting for segment. successful: 1 optimal: 13 errs: <nil>
Apr 11 00:35:39.781642337 (80h49m26.600878371s) [upload=77646]: Done uploading segments...
Apr 11 00:35:39.781678407 (80h49m26.600914441s) [upload=77646]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 1)
Duration: 5.749011421s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.........XX.X.X.............................................] duration:
3.735712029s trace:
https://jaeger.dev.storj.io/trace/5366ea4092e184b8
FAIL (2026-04-11 01:12:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.73573825s
core
uplink
us-select-1
us1-placement-20
|