Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
11 out of 60 is failed (threshold=3) [.....X.............X..X..X...X..........X.X.X.....X.......XX] duration:
1m0.000977939s trace:
https://jaeger.dev.storj.io/trace/6552c1277e4ef807
FAIL (2026-01-16 14:13:57 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.000990032s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
2 out of 60 is failed (threshold=3) [......................................X....................X] duration:
30.000862547s trace:
https://jaeger.dev.storj.io/trace/373f63b93bd9d9fd
FAIL (2026-01-16 14:13:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000880731s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XXXXXX] duration:
5.001420917s trace:
https://jaeger.dev.storj.io/trace/52e467d4425ee59f
FAIL (2026-01-16 14:13:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00143796s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
57 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXX.XXXX] duration:
30.001366885s trace:
https://jaeger.dev.storj.io/trace/1aa91f254fdc9955
FAIL (2026-01-16 14:13:34 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001379919s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
6 out of 60 is failed (threshold=3) [........X........X..X.XX.....X..............................] duration:
4.409426772s trace:
https://jaeger.dev.storj.io/trace/18ea158581539e05
FAIL (2026-01-16 14:13:15 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 25)
======LOGS======
: Uploading piece...
Jan 16 13:47:34.817121271 (21h51m31.71082988s) [noise=true , address=81.171.14.70:4030 , piece_id=SLIXLC6YJ5PLS4MCXJIACYV5MIFOQSBARATYC3WAEZEZSVWICMCQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817237642 (21h51m31.710946251s) [noise=true , address=81.171.14.70:4040 , piece_id=4ZRIS4UHZC6SMLPEIXS6KU6UAKYVIGI6ZMJFRFJ25PY4RB32WVEQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817351119 (21h51m31.711059728s) [noise=true , address=89.149.201.147:4011 , piece_id=TFRZQWAVN2NUSEWM4L33PKZKUYWQYCV3KCRHTKVTABLDZ7GSVVSQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817710643 (21h51m31.711419282s) [noise=true , address=185.167.204.205:4018 , piece_id=3J3LKBXHO5E2IYUVRO7NG7BSYXYBQEFRHV23OQHXHFBYBIYOBQ2A, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817769405 (21h51m31.711478014s) [noise=true , address=185.167.204.206:4067 , piece_id=BSRJBOCNWZIYTEP6HY3NOGZORQTPCWKN43E6HPVC37LY5I5CJYNA, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818062553 (21h51m31.711771162s) [noise=true , address=185.167.204.206:4062 , piece_id=3Z6VWEXXIA3ISY73C5JV5S5ROLCBR6DSQWZVI6Q3QSSYLRBNM6LQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818137756 (21h51m31.711846365s) [noise=true , address=89.149.201.147:4012 , piece_id=SPAZ2DY75JXHC2PWPRI3QG7HYCGHM4GVXN5DY5ZWYDTM6637EV7Q, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818415194 (21h51m31.712123793s) [noise=true , address=185.167.204.205:4006 , piece_id=XCZ2NHQZKDA7AADK3JP26Y3WOZMXQ5IJV5THD5V2P5TX3F7WAFOQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818739812 (21h51m31.712448421s) [noise=true , address=89.149.201.147:4023 , piece_id=LQXOUV6UVIHCQTXPVMC7NYINBLEZWF4H7W6BSOWGK6BFG5ARQ3VQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819043791 (21h51m31.7127524s) [noise=true , address=185.167.204.205:4026 , piece_id=SVVM4Q6OCLSV7TWVURRBYN765J44OTG7AFOQS3RZX7GKFMDJ6ZJQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819337630 (21h51m31.713046239s) [noise=true , address=185.167.204.206:4050 , piece_id=E5FLLMTDTPGGIAKZDIIWPBOFWQL7YXZSUA7Z5YSN2KWJOASDJF6Q, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819543422 (21h51m31.713252031s) [noise=true , address=81.171.14.70:4042 , piece_id=PFKTRW3DI6O7EQ4LSDBQM5YQDJ6JQLXCNZTBCNMTYD3SKQWVGGLQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819810982 (21h51m31.713519591s) [noise=true , address=81.171.14.70:4027 , piece_id=7YO3ZAQFFZOSWSG2JH3KOHWDEFQJLOCCQ5FUGSIT7UUREHROOGYA, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:35.646384760 (21h51m32.540093369s) [upload=19196]: Next returned nil segment
Jan 16 13:47:35.646399467 (21h51m32.540108076s) [upload=19196]: Waiting for error group managing segments...
Jan 16 13:47:36.044357389 (21h51m32.938065998s) [noise=true , address=81.171.14.70:4045 , piece_id=GC5TS2METLE6BBQQGYRMWPTMVH3RMRKQJA2RXKQFCCTCJB2JXMBA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.049970457 (21h51m32.943679076s) [noise=true , address=81.171.14.70:4042 , piece_id=PFKTRW3DI6O7EQ4LSDBQM5YQDJ6JQLXCNZTBCNMTYD3SKQWVGGLQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.057550509 (21h51m32.951259118s) [noise=true , address=81.171.14.70:4040 , piece_id=4ZRIS4UHZC6SMLPEIXS6KU6UAKYVIGI6ZMJFRFJ25PY4RB32WVEQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.093832752 (21h51m32.987541361s) [noise=true , address=81.171.14.70:4026 , piece_id=YZUBTL2WRUNDUPW376BG676VAWOU2GMIMQITAKTWIKJ3HKZXNLZA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.370239906 (21h51m33.263948515s) [noise=true , address=185.167.204.206:4067 , piece_id=BSRJBOCNWZIYTEP6HY3NOGZORQTPCWKN43E6HPVC37LY5I5CJYNA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.384125184 (21h51m33.277833793s) [noise=true , address=185.167.204.206:4062 , piece_id=3Z6VWEXXIA3ISY73C5JV5S5ROLCBR6DSQWZVI6Q3QSSYLRBNM6LQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.391293162 (21h51m33.285001771s) [noise=true , address=185.167.204.206:4064 , piece_id=2PDTBJHQW5AHYD2SOM6Q5D6BYGQJCWSWQNLRXCNRPWYBHKBS6B3Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.391398192 (21h51m33.285106801s) [noise=true , address=185.167.204.206:4040 , piece_id=XCGSNFY225UO7MYFSYWQQLVMP73RIUVII7ZYTTGWZ3TQU2DPMOTA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.392101912 (21h51m33.285810521s) [noise=true , address=185.167.204.206:4061 , piece_id=EN3JISM5WLYJWU3UG3BFTNCBOP7MTIFJBRYU4KTKDFVSWCEJRVJA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.400384883 (21h51m33.294093492s) [noise=true , address=185.167.204.206:4039 , piece_id=BJWZY2S2EZSQO6CJM25WV4RIYTMS3P4CG5RHC44JXDMUGA3Z3JEQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.405248333 (21h51m33.298956942s) [noise=true , address=185.167.204.206:4050 , piece_id=E5FLLMTDTPGGIAKZDIIWPBOFWQL7YXZSUA7Z5YSN2KWJOASDJF6Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.414198665 (21h51m33.307907274s) [noise=true , address=185.167.204.205:4006 , piece_id=XCZ2NHQZKDA7AADK3JP26Y3WOZMXQ5IJV5THD5V2P5TX3F7WAFOQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.423293562 (21h51m33.317002171s) [noise=true , address=185.167.204.206:4045 , piece_id=FHIBGSVU2OZBBQETRZNJK6QW7DTCV7ITCEMUGEYQAN6ZYMQQYTOQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.423504314 (21h51m33.317212923s) [noise=true , address=185.167.204.205:4018 , piece_id=3J3LKBXHO5E2IYUVRO7NG7BSYXYBQEFRHV23OQHXHFBYBIYOBQ2A, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.435923832 (21h51m33.329632461s) [noise=true , address=185.167.204.205:4019 , piece_id=NGOWYQ3TZHPLUJ73WVIC3YYL65EYJVBKOPUWCI4DLEGRPQIECBBA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.436584970 (21h51m33.330293579s) [noise=true , address=185.167.204.205:4026 , piece_id=SVVM4Q6OCLSV7TWVURRBYN765J44OTG7AFOQS3RZX7GKFMDJ6ZJQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.491250847 (21h51m33.384959456s) [noise=true , address=81.171.14.70:4031 , piece_id=EF67IQMYFBPUP6UZXDGSQLHR6X4G6OSCKEH4DYWCKI4UIHO4UVDA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.518740798 (21h51m33.412449407s) [noise=true , address=81.171.14.70:4030 , piece_id=SLIXLC6YJ5PLS4MCXJIACYV5MIFOQSBARATYC3WAEZEZSVWICMCQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.526006844 (21h51m33.419715453s) [noise=true , address=81.171.14.70:4025 , piece_id=DCILR24FF45JKLU36SHTMAJLY3NTFNGT4UWNFJXATADZJY2W3TBA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.564445552 (21h51m33.458154161s) [noise=true , address=81.171.14.70:4027 , piece_id=7YO3ZAQFFZOSWSG2JH3KOHWDEFQJLOCCQ5FUGSIT7UUREHROOGYA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.587122821 (21h51m33.48083146s) [noise=true , address=81.171.14.70:4038 , piece_id=VAUCLB55KGVJVL4BSWY4ZY2747BDYFXI2RK3JMPDO4ZOCVOT43EA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.679729699 (21h51m33.573438309s) [noise=true , address=89.149.201.147:4012 , piece_id=SPAZ2DY75JXHC2PWPRI3QG7HYCGHM4GVXN5DY5ZWYDTM6637EV7Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.775377568 (21h51m33.669086177s) [noise=true , address=89.149.201.147:4010 , piece_id=WBP6VM7X2ZISV4K4SE6LL3ZSPYHJQK24IOGILVH6S2ZLWRRJVN7Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:37.160795473 (21h51m34.054504082s) [noise=true , address=89.149.201.147:4023 , piece_id=LQXOUV6UVIHCQTXPVMC7NYINBLEZWF4H7W6BSOWGK6BFG5ARQ3VQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:37.251044932 (21h51m34.144753541s) [noise=true , address=89.149.201.147:4007 , piece_id=J4W7LHO6YZCI3KPZ3F3DN6P4EAV5RIMRQ2X7ZOKLAAQVNGD7HVMA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:48:34.164143586 (21h52m31.057852195s) [noise=true , address=89.149.201.147:4011 , piece_id=TFRZQWAVN2NUSEWM4L33PKZKUYWQYCV3KCRHTKVTABLDZ7GSVVSQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: ecclient: upload failed (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg, address:89.149.201.147:4011): protocol: read tcp 188.166.225.170:48732->89.149.201.147:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 16 13:48:34.164198381 (21h52m31.05790699s) [noise=true , address=89.149.201.147:4020 , piece_id=CQAAHSLJFZE4GHOKB5UO7A2ANJ6N4HHFZDQ33WV25JF76L7JMMDQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: expected piece hash; context deadline exceeded; EOF
Jan 16 13:48:34.164358165 (21h52m31.058066774s) [noise=true , address=89.149.201.147:4014 , piece_id=BNGVISOX55CNKPIHIJJCXFNIMGW5FCE6GWEY4QKLIAWMVYMRAYFA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: context deadline exceeded; context deadline exceeded
Jan 16 13:48:34.164382872 (21h52m31.058091481s) [seg_pos={0 0}, upload=19196]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil>
Jan 16 13:48:34.330174768 (21h52m31.223883387s) [upload=19196]: Done uploading segments...
Jan 16 13:48:34.330206869 (21h52m31.223915478s) [upload=19196]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25)
Duration: 4.409458491s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XXXXX] duration:
5.001668279s trace:
https://jaeger.dev.storj.io/trace/1c3834f9ab804ed5
FAIL (2026-01-16 14:13:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001692956s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
10 out of 60 is failed (threshold=3) [......X.............X..X..X...X..........X.X.X.....X.......X] duration:
1m0.001191876s trace:
https://jaeger.dev.storj.io/trace/1dd4b677808db3ea
FAIL (2026-01-16 14:12:56 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.001215011s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XXXX] duration:
5.000945361s trace:
https://jaeger.dev.storj.io/trace/663f96d652cb6613
FAIL (2026-01-16 14:12:46 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000957724s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
6 out of 60 is failed (threshold=3) [.........X........X..X.XX.....X.............................] duration:
5.246183567s trace:
https://jaeger.dev.storj.io/trace/18ea158581539e05
FAIL (2026-01-16 14:12:22 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 25)
======LOGS======
: Uploading piece...
Jan 16 13:47:34.817121271 (21h51m31.71082988s) [noise=true , address=81.171.14.70:4030 , piece_id=SLIXLC6YJ5PLS4MCXJIACYV5MIFOQSBARATYC3WAEZEZSVWICMCQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817237642 (21h51m31.710946251s) [noise=true , address=81.171.14.70:4040 , piece_id=4ZRIS4UHZC6SMLPEIXS6KU6UAKYVIGI6ZMJFRFJ25PY4RB32WVEQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817351119 (21h51m31.711059728s) [noise=true , address=89.149.201.147:4011 , piece_id=TFRZQWAVN2NUSEWM4L33PKZKUYWQYCV3KCRHTKVTABLDZ7GSVVSQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817710643 (21h51m31.711419282s) [noise=true , address=185.167.204.205:4018 , piece_id=3J3LKBXHO5E2IYUVRO7NG7BSYXYBQEFRHV23OQHXHFBYBIYOBQ2A, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817769405 (21h51m31.711478014s) [noise=true , address=185.167.204.206:4067 , piece_id=BSRJBOCNWZIYTEP6HY3NOGZORQTPCWKN43E6HPVC37LY5I5CJYNA, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818062553 (21h51m31.711771162s) [noise=true , address=185.167.204.206:4062 , piece_id=3Z6VWEXXIA3ISY73C5JV5S5ROLCBR6DSQWZVI6Q3QSSYLRBNM6LQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818137756 (21h51m31.711846365s) [noise=true , address=89.149.201.147:4012 , piece_id=SPAZ2DY75JXHC2PWPRI3QG7HYCGHM4GVXN5DY5ZWYDTM6637EV7Q, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818415194 (21h51m31.712123793s) [noise=true , address=185.167.204.205:4006 , piece_id=XCZ2NHQZKDA7AADK3JP26Y3WOZMXQ5IJV5THD5V2P5TX3F7WAFOQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818739812 (21h51m31.712448421s) [noise=true , address=89.149.201.147:4023 , piece_id=LQXOUV6UVIHCQTXPVMC7NYINBLEZWF4H7W6BSOWGK6BFG5ARQ3VQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819043791 (21h51m31.7127524s) [noise=true , address=185.167.204.205:4026 , piece_id=SVVM4Q6OCLSV7TWVURRBYN765J44OTG7AFOQS3RZX7GKFMDJ6ZJQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819337630 (21h51m31.713046239s) [noise=true , address=185.167.204.206:4050 , piece_id=E5FLLMTDTPGGIAKZDIIWPBOFWQL7YXZSUA7Z5YSN2KWJOASDJF6Q, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819543422 (21h51m31.713252031s) [noise=true , address=81.171.14.70:4042 , piece_id=PFKTRW3DI6O7EQ4LSDBQM5YQDJ6JQLXCNZTBCNMTYD3SKQWVGGLQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819810982 (21h51m31.713519591s) [noise=true , address=81.171.14.70:4027 , piece_id=7YO3ZAQFFZOSWSG2JH3KOHWDEFQJLOCCQ5FUGSIT7UUREHROOGYA, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:35.646384760 (21h51m32.540093369s) [upload=19196]: Next returned nil segment
Jan 16 13:47:35.646399467 (21h51m32.540108076s) [upload=19196]: Waiting for error group managing segments...
Jan 16 13:47:36.044357389 (21h51m32.938065998s) [noise=true , address=81.171.14.70:4045 , piece_id=GC5TS2METLE6BBQQGYRMWPTMVH3RMRKQJA2RXKQFCCTCJB2JXMBA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.049970457 (21h51m32.943679076s) [noise=true , address=81.171.14.70:4042 , piece_id=PFKTRW3DI6O7EQ4LSDBQM5YQDJ6JQLXCNZTBCNMTYD3SKQWVGGLQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.057550509 (21h51m32.951259118s) [noise=true , address=81.171.14.70:4040 , piece_id=4ZRIS4UHZC6SMLPEIXS6KU6UAKYVIGI6ZMJFRFJ25PY4RB32WVEQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.093832752 (21h51m32.987541361s) [noise=true , address=81.171.14.70:4026 , piece_id=YZUBTL2WRUNDUPW376BG676VAWOU2GMIMQITAKTWIKJ3HKZXNLZA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.370239906 (21h51m33.263948515s) [noise=true , address=185.167.204.206:4067 , piece_id=BSRJBOCNWZIYTEP6HY3NOGZORQTPCWKN43E6HPVC37LY5I5CJYNA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.384125184 (21h51m33.277833793s) [noise=true , address=185.167.204.206:4062 , piece_id=3Z6VWEXXIA3ISY73C5JV5S5ROLCBR6DSQWZVI6Q3QSSYLRBNM6LQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.391293162 (21h51m33.285001771s) [noise=true , address=185.167.204.206:4064 , piece_id=2PDTBJHQW5AHYD2SOM6Q5D6BYGQJCWSWQNLRXCNRPWYBHKBS6B3Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.391398192 (21h51m33.285106801s) [noise=true , address=185.167.204.206:4040 , piece_id=XCGSNFY225UO7MYFSYWQQLVMP73RIUVII7ZYTTGWZ3TQU2DPMOTA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.392101912 (21h51m33.285810521s) [noise=true , address=185.167.204.206:4061 , piece_id=EN3JISM5WLYJWU3UG3BFTNCBOP7MTIFJBRYU4KTKDFVSWCEJRVJA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.400384883 (21h51m33.294093492s) [noise=true , address=185.167.204.206:4039 , piece_id=BJWZY2S2EZSQO6CJM25WV4RIYTMS3P4CG5RHC44JXDMUGA3Z3JEQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.405248333 (21h51m33.298956942s) [noise=true , address=185.167.204.206:4050 , piece_id=E5FLLMTDTPGGIAKZDIIWPBOFWQL7YXZSUA7Z5YSN2KWJOASDJF6Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.414198665 (21h51m33.307907274s) [noise=true , address=185.167.204.205:4006 , piece_id=XCZ2NHQZKDA7AADK3JP26Y3WOZMXQ5IJV5THD5V2P5TX3F7WAFOQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.423293562 (21h51m33.317002171s) [noise=true , address=185.167.204.206:4045 , piece_id=FHIBGSVU2OZBBQETRZNJK6QW7DTCV7ITCEMUGEYQAN6ZYMQQYTOQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.423504314 (21h51m33.317212923s) [noise=true , address=185.167.204.205:4018 , piece_id=3J3LKBXHO5E2IYUVRO7NG7BSYXYBQEFRHV23OQHXHFBYBIYOBQ2A, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.435923832 (21h51m33.329632461s) [noise=true , address=185.167.204.205:4019 , piece_id=NGOWYQ3TZHPLUJ73WVIC3YYL65EYJVBKOPUWCI4DLEGRPQIECBBA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.436584970 (21h51m33.330293579s) [noise=true , address=185.167.204.205:4026 , piece_id=SVVM4Q6OCLSV7TWVURRBYN765J44OTG7AFOQS3RZX7GKFMDJ6ZJQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.491250847 (21h51m33.384959456s) [noise=true , address=81.171.14.70:4031 , piece_id=EF67IQMYFBPUP6UZXDGSQLHR6X4G6OSCKEH4DYWCKI4UIHO4UVDA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.518740798 (21h51m33.412449407s) [noise=true , address=81.171.14.70:4030 , piece_id=SLIXLC6YJ5PLS4MCXJIACYV5MIFOQSBARATYC3WAEZEZSVWICMCQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.526006844 (21h51m33.419715453s) [noise=true , address=81.171.14.70:4025 , piece_id=DCILR24FF45JKLU36SHTMAJLY3NTFNGT4UWNFJXATADZJY2W3TBA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.564445552 (21h51m33.458154161s) [noise=true , address=81.171.14.70:4027 , piece_id=7YO3ZAQFFZOSWSG2JH3KOHWDEFQJLOCCQ5FUGSIT7UUREHROOGYA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.587122821 (21h51m33.48083146s) [noise=true , address=81.171.14.70:4038 , piece_id=VAUCLB55KGVJVL4BSWY4ZY2747BDYFXI2RK3JMPDO4ZOCVOT43EA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.679729699 (21h51m33.573438309s) [noise=true , address=89.149.201.147:4012 , piece_id=SPAZ2DY75JXHC2PWPRI3QG7HYCGHM4GVXN5DY5ZWYDTM6637EV7Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.775377568 (21h51m33.669086177s) [noise=true , address=89.149.201.147:4010 , piece_id=WBP6VM7X2ZISV4K4SE6LL3ZSPYHJQK24IOGILVH6S2ZLWRRJVN7Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:37.160795473 (21h51m34.054504082s) [noise=true , address=89.149.201.147:4023 , piece_id=LQXOUV6UVIHCQTXPVMC7NYINBLEZWF4H7W6BSOWGK6BFG5ARQ3VQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:37.251044932 (21h51m34.144753541s) [noise=true , address=89.149.201.147:4007 , piece_id=J4W7LHO6YZCI3KPZ3F3DN6P4EAV5RIMRQ2X7ZOKLAAQVNGD7HVMA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:48:34.164143586 (21h52m31.057852195s) [noise=true , address=89.149.201.147:4011 , piece_id=TFRZQWAVN2NUSEWM4L33PKZKUYWQYCV3KCRHTKVTABLDZ7GSVVSQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: ecclient: upload failed (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg, address:89.149.201.147:4011): protocol: read tcp 188.166.225.170:48732->89.149.201.147:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 16 13:48:34.164198381 (21h52m31.05790699s) [noise=true , address=89.149.201.147:4020 , piece_id=CQAAHSLJFZE4GHOKB5UO7A2ANJ6N4HHFZDQ33WV25JF76L7JMMDQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: expected piece hash; context deadline exceeded; EOF
Jan 16 13:48:34.164358165 (21h52m31.058066774s) [noise=true , address=89.149.201.147:4014 , piece_id=BNGVISOX55CNKPIHIJJCXFNIMGW5FCE6GWEY4QKLIAWMVYMRAYFA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: context deadline exceeded; context deadline exceeded
Jan 16 13:48:34.164382872 (21h52m31.058091481s) [seg_pos={0 0}, upload=19196]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil>
Jan 16 13:48:34.330174768 (21h52m31.223883387s) [upload=19196]: Done uploading segments...
Jan 16 13:48:34.330206869 (21h52m31.223915478s) [upload=19196]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25)
Duration: 5.24621093s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
9.602399263s trace:
https://jaeger.dev.storj.io/trace/a3cda5286a23a90
FAIL (2026-01-16 14:12:22 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 9.602427699s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
56 out of 60 is failed (threshold=3) [..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXX.XXX] duration:
30.000720646s trace:
https://jaeger.dev.storj.io/trace/f26f43dc982ae9e
FAIL (2026-01-16 14:12:20 UTC)
orbiter warning: context deadline exceeded
Duration: 30.000739691s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XXX] duration:
5.002002445s trace:
https://jaeger.dev.storj.io/trace/719b0466e71171f7
FAIL (2026-01-16 14:12:03 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002027012s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
56 out of 60 is failed (threshold=3) [X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXX.XX] duration:
10.876626314s trace:
https://jaeger.dev.storj.io/trace/b1c497ccaab8ae3
FAIL (2026-01-16 14:11:36 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 10.876652665s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
9 out of 60 is failed (threshold=3) [.......X.............X..X..X...X..........X.X.X.....X.......] duration:
21.795100798s trace:
https://jaeger.dev.storj.io/trace/346659e7312f1ccc
FAIL (2026-01-16 14:11:32 UTC)
orbiter warning: context deadline exceeded
Duration: 21.79512803s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
7.058525877s trace:
https://jaeger.dev.storj.io/trace/a3cda5286a23a90
FAIL (2026-01-16 14:11:28 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 7.058567716s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
6 out of 60 is failed (threshold=3) [..........X........X..X.XX.....X............................] duration:
4.733792801s trace:
https://jaeger.dev.storj.io/trace/18ea158581539e05
FAIL (2026-01-16 14:11:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 25)
======LOGS======
: Uploading piece...
Jan 16 13:47:34.817121271 (21h51m31.71082988s) [noise=true , address=81.171.14.70:4030 , piece_id=SLIXLC6YJ5PLS4MCXJIACYV5MIFOQSBARATYC3WAEZEZSVWICMCQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817237642 (21h51m31.710946251s) [noise=true , address=81.171.14.70:4040 , piece_id=4ZRIS4UHZC6SMLPEIXS6KU6UAKYVIGI6ZMJFRFJ25PY4RB32WVEQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817351119 (21h51m31.711059728s) [noise=true , address=89.149.201.147:4011 , piece_id=TFRZQWAVN2NUSEWM4L33PKZKUYWQYCV3KCRHTKVTABLDZ7GSVVSQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817710643 (21h51m31.711419282s) [noise=true , address=185.167.204.205:4018 , piece_id=3J3LKBXHO5E2IYUVRO7NG7BSYXYBQEFRHV23OQHXHFBYBIYOBQ2A, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817769405 (21h51m31.711478014s) [noise=true , address=185.167.204.206:4067 , piece_id=BSRJBOCNWZIYTEP6HY3NOGZORQTPCWKN43E6HPVC37LY5I5CJYNA, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818062553 (21h51m31.711771162s) [noise=true , address=185.167.204.206:4062 , piece_id=3Z6VWEXXIA3ISY73C5JV5S5ROLCBR6DSQWZVI6Q3QSSYLRBNM6LQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818137756 (21h51m31.711846365s) [noise=true , address=89.149.201.147:4012 , piece_id=SPAZ2DY75JXHC2PWPRI3QG7HYCGHM4GVXN5DY5ZWYDTM6637EV7Q, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818415194 (21h51m31.712123793s) [noise=true , address=185.167.204.205:4006 , piece_id=XCZ2NHQZKDA7AADK3JP26Y3WOZMXQ5IJV5THD5V2P5TX3F7WAFOQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818739812 (21h51m31.712448421s) [noise=true , address=89.149.201.147:4023 , piece_id=LQXOUV6UVIHCQTXPVMC7NYINBLEZWF4H7W6BSOWGK6BFG5ARQ3VQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819043791 (21h51m31.7127524s) [noise=true , address=185.167.204.205:4026 , piece_id=SVVM4Q6OCLSV7TWVURRBYN765J44OTG7AFOQS3RZX7GKFMDJ6ZJQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819337630 (21h51m31.713046239s) [noise=true , address=185.167.204.206:4050 , piece_id=E5FLLMTDTPGGIAKZDIIWPBOFWQL7YXZSUA7Z5YSN2KWJOASDJF6Q, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819543422 (21h51m31.713252031s) [noise=true , address=81.171.14.70:4042 , piece_id=PFKTRW3DI6O7EQ4LSDBQM5YQDJ6JQLXCNZTBCNMTYD3SKQWVGGLQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819810982 (21h51m31.713519591s) [noise=true , address=81.171.14.70:4027 , piece_id=7YO3ZAQFFZOSWSG2JH3KOHWDEFQJLOCCQ5FUGSIT7UUREHROOGYA, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:35.646384760 (21h51m32.540093369s) [upload=19196]: Next returned nil segment
Jan 16 13:47:35.646399467 (21h51m32.540108076s) [upload=19196]: Waiting for error group managing segments...
Jan 16 13:47:36.044357389 (21h51m32.938065998s) [noise=true , address=81.171.14.70:4045 , piece_id=GC5TS2METLE6BBQQGYRMWPTMVH3RMRKQJA2RXKQFCCTCJB2JXMBA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.049970457 (21h51m32.943679076s) [noise=true , address=81.171.14.70:4042 , piece_id=PFKTRW3DI6O7EQ4LSDBQM5YQDJ6JQLXCNZTBCNMTYD3SKQWVGGLQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.057550509 (21h51m32.951259118s) [noise=true , address=81.171.14.70:4040 , piece_id=4ZRIS4UHZC6SMLPEIXS6KU6UAKYVIGI6ZMJFRFJ25PY4RB32WVEQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.093832752 (21h51m32.987541361s) [noise=true , address=81.171.14.70:4026 , piece_id=YZUBTL2WRUNDUPW376BG676VAWOU2GMIMQITAKTWIKJ3HKZXNLZA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.370239906 (21h51m33.263948515s) [noise=true , address=185.167.204.206:4067 , piece_id=BSRJBOCNWZIYTEP6HY3NOGZORQTPCWKN43E6HPVC37LY5I5CJYNA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.384125184 (21h51m33.277833793s) [noise=true , address=185.167.204.206:4062 , piece_id=3Z6VWEXXIA3ISY73C5JV5S5ROLCBR6DSQWZVI6Q3QSSYLRBNM6LQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.391293162 (21h51m33.285001771s) [noise=true , address=185.167.204.206:4064 , piece_id=2PDTBJHQW5AHYD2SOM6Q5D6BYGQJCWSWQNLRXCNRPWYBHKBS6B3Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.391398192 (21h51m33.285106801s) [noise=true , address=185.167.204.206:4040 , piece_id=XCGSNFY225UO7MYFSYWQQLVMP73RIUVII7ZYTTGWZ3TQU2DPMOTA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.392101912 (21h51m33.285810521s) [noise=true , address=185.167.204.206:4061 , piece_id=EN3JISM5WLYJWU3UG3BFTNCBOP7MTIFJBRYU4KTKDFVSWCEJRVJA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.400384883 (21h51m33.294093492s) [noise=true , address=185.167.204.206:4039 , piece_id=BJWZY2S2EZSQO6CJM25WV4RIYTMS3P4CG5RHC44JXDMUGA3Z3JEQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.405248333 (21h51m33.298956942s) [noise=true , address=185.167.204.206:4050 , piece_id=E5FLLMTDTPGGIAKZDIIWPBOFWQL7YXZSUA7Z5YSN2KWJOASDJF6Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.414198665 (21h51m33.307907274s) [noise=true , address=185.167.204.205:4006 , piece_id=XCZ2NHQZKDA7AADK3JP26Y3WOZMXQ5IJV5THD5V2P5TX3F7WAFOQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.423293562 (21h51m33.317002171s) [noise=true , address=185.167.204.206:4045 , piece_id=FHIBGSVU2OZBBQETRZNJK6QW7DTCV7ITCEMUGEYQAN6ZYMQQYTOQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.423504314 (21h51m33.317212923s) [noise=true , address=185.167.204.205:4018 , piece_id=3J3LKBXHO5E2IYUVRO7NG7BSYXYBQEFRHV23OQHXHFBYBIYOBQ2A, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.435923832 (21h51m33.329632461s) [noise=true , address=185.167.204.205:4019 , piece_id=NGOWYQ3TZHPLUJ73WVIC3YYL65EYJVBKOPUWCI4DLEGRPQIECBBA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.436584970 (21h51m33.330293579s) [noise=true , address=185.167.204.205:4026 , piece_id=SVVM4Q6OCLSV7TWVURRBYN765J44OTG7AFOQS3RZX7GKFMDJ6ZJQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.491250847 (21h51m33.384959456s) [noise=true , address=81.171.14.70:4031 , piece_id=EF67IQMYFBPUP6UZXDGSQLHR6X4G6OSCKEH4DYWCKI4UIHO4UVDA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.518740798 (21h51m33.412449407s) [noise=true , address=81.171.14.70:4030 , piece_id=SLIXLC6YJ5PLS4MCXJIACYV5MIFOQSBARATYC3WAEZEZSVWICMCQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.526006844 (21h51m33.419715453s) [noise=true , address=81.171.14.70:4025 , piece_id=DCILR24FF45JKLU36SHTMAJLY3NTFNGT4UWNFJXATADZJY2W3TBA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.564445552 (21h51m33.458154161s) [noise=true , address=81.171.14.70:4027 , piece_id=7YO3ZAQFFZOSWSG2JH3KOHWDEFQJLOCCQ5FUGSIT7UUREHROOGYA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.587122821 (21h51m33.48083146s) [noise=true , address=81.171.14.70:4038 , piece_id=VAUCLB55KGVJVL4BSWY4ZY2747BDYFXI2RK3JMPDO4ZOCVOT43EA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.679729699 (21h51m33.573438309s) [noise=true , address=89.149.201.147:4012 , piece_id=SPAZ2DY75JXHC2PWPRI3QG7HYCGHM4GVXN5DY5ZWYDTM6637EV7Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.775377568 (21h51m33.669086177s) [noise=true , address=89.149.201.147:4010 , piece_id=WBP6VM7X2ZISV4K4SE6LL3ZSPYHJQK24IOGILVH6S2ZLWRRJVN7Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:37.160795473 (21h51m34.054504082s) [noise=true , address=89.149.201.147:4023 , piece_id=LQXOUV6UVIHCQTXPVMC7NYINBLEZWF4H7W6BSOWGK6BFG5ARQ3VQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:37.251044932 (21h51m34.144753541s) [noise=true , address=89.149.201.147:4007 , piece_id=J4W7LHO6YZCI3KPZ3F3DN6P4EAV5RIMRQ2X7ZOKLAAQVNGD7HVMA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:48:34.164143586 (21h52m31.057852195s) [noise=true , address=89.149.201.147:4011 , piece_id=TFRZQWAVN2NUSEWM4L33PKZKUYWQYCV3KCRHTKVTABLDZ7GSVVSQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: ecclient: upload failed (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg, address:89.149.201.147:4011): protocol: read tcp 188.166.225.170:48732->89.149.201.147:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 16 13:48:34.164198381 (21h52m31.05790699s) [noise=true , address=89.149.201.147:4020 , piece_id=CQAAHSLJFZE4GHOKB5UO7A2ANJ6N4HHFZDQ33WV25JF76L7JMMDQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: expected piece hash; context deadline exceeded; EOF
Jan 16 13:48:34.164358165 (21h52m31.058066774s) [noise=true , address=89.149.201.147:4014 , piece_id=BNGVISOX55CNKPIHIJJCXFNIMGW5FCE6GWEY4QKLIAWMVYMRAYFA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: context deadline exceeded; context deadline exceeded
Jan 16 13:48:34.164382872 (21h52m31.058091481s) [seg_pos={0 0}, upload=19196]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil>
Jan 16 13:48:34.330174768 (21h52m31.223883387s) [upload=19196]: Done uploading segments...
Jan 16 13:48:34.330206869 (21h52m31.223915478s) [upload=19196]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25)
Duration: 4.733816035s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX] duration:
5.00085995s trace:
https://jaeger.dev.storj.io/trace/d0309ed6021613f
FAIL (2026-01-16 14:11:23 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000884005s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
55 out of 60 is failed (threshold=3) [.X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXX.X] duration:
30.00151195s trace:
https://jaeger.dev.storj.io/trace/b92df9f7b601c92
FAIL (2026-01-16 14:11:13 UTC)
orbiter warning: context deadline exceeded
Duration: 30.00160102s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..X] duration:
5.000857326s trace:
https://jaeger.dev.storj.io/trace/375ed121861f5ac9
FAIL (2026-01-16 14:11:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000874217s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
6 out of 60 is failed (threshold=3) [...........X........X..X.XX.....X...........................] duration:
4.756466253s trace:
https://jaeger.dev.storj.io/trace/18ea158581539e05
FAIL (2026-01-16 14:11:00 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 25)
======LOGS======
: Uploading piece...
Jan 16 13:47:34.817121271 (21h51m31.71082988s) [noise=true , address=81.171.14.70:4030 , piece_id=SLIXLC6YJ5PLS4MCXJIACYV5MIFOQSBARATYC3WAEZEZSVWICMCQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817237642 (21h51m31.710946251s) [noise=true , address=81.171.14.70:4040 , piece_id=4ZRIS4UHZC6SMLPEIXS6KU6UAKYVIGI6ZMJFRFJ25PY4RB32WVEQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817351119 (21h51m31.711059728s) [noise=true , address=89.149.201.147:4011 , piece_id=TFRZQWAVN2NUSEWM4L33PKZKUYWQYCV3KCRHTKVTABLDZ7GSVVSQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817710643 (21h51m31.711419282s) [noise=true , address=185.167.204.205:4018 , piece_id=3J3LKBXHO5E2IYUVRO7NG7BSYXYBQEFRHV23OQHXHFBYBIYOBQ2A, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817769405 (21h51m31.711478014s) [noise=true , address=185.167.204.206:4067 , piece_id=BSRJBOCNWZIYTEP6HY3NOGZORQTPCWKN43E6HPVC37LY5I5CJYNA, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818062553 (21h51m31.711771162s) [noise=true , address=185.167.204.206:4062 , piece_id=3Z6VWEXXIA3ISY73C5JV5S5ROLCBR6DSQWZVI6Q3QSSYLRBNM6LQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818137756 (21h51m31.711846365s) [noise=true , address=89.149.201.147:4012 , piece_id=SPAZ2DY75JXHC2PWPRI3QG7HYCGHM4GVXN5DY5ZWYDTM6637EV7Q, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818415194 (21h51m31.712123793s) [noise=true , address=185.167.204.205:4006 , piece_id=XCZ2NHQZKDA7AADK3JP26Y3WOZMXQ5IJV5THD5V2P5TX3F7WAFOQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818739812 (21h51m31.712448421s) [noise=true , address=89.149.201.147:4023 , piece_id=LQXOUV6UVIHCQTXPVMC7NYINBLEZWF4H7W6BSOWGK6BFG5ARQ3VQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819043791 (21h51m31.7127524s) [noise=true , address=185.167.204.205:4026 , piece_id=SVVM4Q6OCLSV7TWVURRBYN765J44OTG7AFOQS3RZX7GKFMDJ6ZJQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819337630 (21h51m31.713046239s) [noise=true , address=185.167.204.206:4050 , piece_id=E5FLLMTDTPGGIAKZDIIWPBOFWQL7YXZSUA7Z5YSN2KWJOASDJF6Q, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819543422 (21h51m31.713252031s) [noise=true , address=81.171.14.70:4042 , piece_id=PFKTRW3DI6O7EQ4LSDBQM5YQDJ6JQLXCNZTBCNMTYD3SKQWVGGLQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819810982 (21h51m31.713519591s) [noise=true , address=81.171.14.70:4027 , piece_id=7YO3ZAQFFZOSWSG2JH3KOHWDEFQJLOCCQ5FUGSIT7UUREHROOGYA, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:35.646384760 (21h51m32.540093369s) [upload=19196]: Next returned nil segment
Jan 16 13:47:35.646399467 (21h51m32.540108076s) [upload=19196]: Waiting for error group managing segments...
Jan 16 13:47:36.044357389 (21h51m32.938065998s) [noise=true , address=81.171.14.70:4045 , piece_id=GC5TS2METLE6BBQQGYRMWPTMVH3RMRKQJA2RXKQFCCTCJB2JXMBA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.049970457 (21h51m32.943679076s) [noise=true , address=81.171.14.70:4042 , piece_id=PFKTRW3DI6O7EQ4LSDBQM5YQDJ6JQLXCNZTBCNMTYD3SKQWVGGLQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.057550509 (21h51m32.951259118s) [noise=true , address=81.171.14.70:4040 , piece_id=4ZRIS4UHZC6SMLPEIXS6KU6UAKYVIGI6ZMJFRFJ25PY4RB32WVEQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.093832752 (21h51m32.987541361s) [noise=true , address=81.171.14.70:4026 , piece_id=YZUBTL2WRUNDUPW376BG676VAWOU2GMIMQITAKTWIKJ3HKZXNLZA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.370239906 (21h51m33.263948515s) [noise=true , address=185.167.204.206:4067 , piece_id=BSRJBOCNWZIYTEP6HY3NOGZORQTPCWKN43E6HPVC37LY5I5CJYNA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.384125184 (21h51m33.277833793s) [noise=true , address=185.167.204.206:4062 , piece_id=3Z6VWEXXIA3ISY73C5JV5S5ROLCBR6DSQWZVI6Q3QSSYLRBNM6LQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.391293162 (21h51m33.285001771s) [noise=true , address=185.167.204.206:4064 , piece_id=2PDTBJHQW5AHYD2SOM6Q5D6BYGQJCWSWQNLRXCNRPWYBHKBS6B3Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.391398192 (21h51m33.285106801s) [noise=true , address=185.167.204.206:4040 , piece_id=XCGSNFY225UO7MYFSYWQQLVMP73RIUVII7ZYTTGWZ3TQU2DPMOTA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.392101912 (21h51m33.285810521s) [noise=true , address=185.167.204.206:4061 , piece_id=EN3JISM5WLYJWU3UG3BFTNCBOP7MTIFJBRYU4KTKDFVSWCEJRVJA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.400384883 (21h51m33.294093492s) [noise=true , address=185.167.204.206:4039 , piece_id=BJWZY2S2EZSQO6CJM25WV4RIYTMS3P4CG5RHC44JXDMUGA3Z3JEQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.405248333 (21h51m33.298956942s) [noise=true , address=185.167.204.206:4050 , piece_id=E5FLLMTDTPGGIAKZDIIWPBOFWQL7YXZSUA7Z5YSN2KWJOASDJF6Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.414198665 (21h51m33.307907274s) [noise=true , address=185.167.204.205:4006 , piece_id=XCZ2NHQZKDA7AADK3JP26Y3WOZMXQ5IJV5THD5V2P5TX3F7WAFOQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.423293562 (21h51m33.317002171s) [noise=true , address=185.167.204.206:4045 , piece_id=FHIBGSVU2OZBBQETRZNJK6QW7DTCV7ITCEMUGEYQAN6ZYMQQYTOQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.423504314 (21h51m33.317212923s) [noise=true , address=185.167.204.205:4018 , piece_id=3J3LKBXHO5E2IYUVRO7NG7BSYXYBQEFRHV23OQHXHFBYBIYOBQ2A, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.435923832 (21h51m33.329632461s) [noise=true , address=185.167.204.205:4019 , piece_id=NGOWYQ3TZHPLUJ73WVIC3YYL65EYJVBKOPUWCI4DLEGRPQIECBBA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.436584970 (21h51m33.330293579s) [noise=true , address=185.167.204.205:4026 , piece_id=SVVM4Q6OCLSV7TWVURRBYN765J44OTG7AFOQS3RZX7GKFMDJ6ZJQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.491250847 (21h51m33.384959456s) [noise=true , address=81.171.14.70:4031 , piece_id=EF67IQMYFBPUP6UZXDGSQLHR6X4G6OSCKEH4DYWCKI4UIHO4UVDA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.518740798 (21h51m33.412449407s) [noise=true , address=81.171.14.70:4030 , piece_id=SLIXLC6YJ5PLS4MCXJIACYV5MIFOQSBARATYC3WAEZEZSVWICMCQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.526006844 (21h51m33.419715453s) [noise=true , address=81.171.14.70:4025 , piece_id=DCILR24FF45JKLU36SHTMAJLY3NTFNGT4UWNFJXATADZJY2W3TBA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.564445552 (21h51m33.458154161s) [noise=true , address=81.171.14.70:4027 , piece_id=7YO3ZAQFFZOSWSG2JH3KOHWDEFQJLOCCQ5FUGSIT7UUREHROOGYA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.587122821 (21h51m33.48083146s) [noise=true , address=81.171.14.70:4038 , piece_id=VAUCLB55KGVJVL4BSWY4ZY2747BDYFXI2RK3JMPDO4ZOCVOT43EA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.679729699 (21h51m33.573438309s) [noise=true , address=89.149.201.147:4012 , piece_id=SPAZ2DY75JXHC2PWPRI3QG7HYCGHM4GVXN5DY5ZWYDTM6637EV7Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.775377568 (21h51m33.669086177s) [noise=true , address=89.149.201.147:4010 , piece_id=WBP6VM7X2ZISV4K4SE6LL3ZSPYHJQK24IOGILVH6S2ZLWRRJVN7Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:37.160795473 (21h51m34.054504082s) [noise=true , address=89.149.201.147:4023 , piece_id=LQXOUV6UVIHCQTXPVMC7NYINBLEZWF4H7W6BSOWGK6BFG5ARQ3VQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:37.251044932 (21h51m34.144753541s) [noise=true , address=89.149.201.147:4007 , piece_id=J4W7LHO6YZCI3KPZ3F3DN6P4EAV5RIMRQ2X7ZOKLAAQVNGD7HVMA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:48:34.164143586 (21h52m31.057852195s) [noise=true , address=89.149.201.147:4011 , piece_id=TFRZQWAVN2NUSEWM4L33PKZKUYWQYCV3KCRHTKVTABLDZ7GSVVSQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: ecclient: upload failed (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg, address:89.149.201.147:4011): protocol: read tcp 188.166.225.170:48732->89.149.201.147:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 16 13:48:34.164198381 (21h52m31.05790699s) [noise=true , address=89.149.201.147:4020 , piece_id=CQAAHSLJFZE4GHOKB5UO7A2ANJ6N4HHFZDQ33WV25JF76L7JMMDQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: expected piece hash; context deadline exceeded; EOF
Jan 16 13:48:34.164358165 (21h52m31.058066774s) [noise=true , address=89.149.201.147:4014 , piece_id=BNGVISOX55CNKPIHIJJCXFNIMGW5FCE6GWEY4QKLIAWMVYMRAYFA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: context deadline exceeded; context deadline exceeded
Jan 16 13:48:34.164382872 (21h52m31.058091481s) [seg_pos={0 0}, upload=19196]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil>
Jan 16 13:48:34.330174768 (21h52m31.223883387s) [upload=19196]: Done uploading segments...
Jan 16 13:48:34.330206869 (21h52m31.223915478s) [upload=19196]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25)
Duration: 4.756492061s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
22.38909573s trace:
https://jaeger.dev.storj.io/trace/a3cda5286a23a90
FAIL (2026-01-16 14:10:48 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 22.389122852s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
6 out of 60 is failed (threshold=3) [............X........X..X.XX.....X..........................] duration:
4.715933056s trace:
https://jaeger.dev.storj.io/trace/18ea158581539e05
FAIL (2026-01-16 14:10:42 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 25)
======LOGS======
: Uploading piece...
Jan 16 13:47:34.817121271 (21h51m31.71082988s) [noise=true , address=81.171.14.70:4030 , piece_id=SLIXLC6YJ5PLS4MCXJIACYV5MIFOQSBARATYC3WAEZEZSVWICMCQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817237642 (21h51m31.710946251s) [noise=true , address=81.171.14.70:4040 , piece_id=4ZRIS4UHZC6SMLPEIXS6KU6UAKYVIGI6ZMJFRFJ25PY4RB32WVEQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817351119 (21h51m31.711059728s) [noise=true , address=89.149.201.147:4011 , piece_id=TFRZQWAVN2NUSEWM4L33PKZKUYWQYCV3KCRHTKVTABLDZ7GSVVSQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817710643 (21h51m31.711419282s) [noise=true , address=185.167.204.205:4018 , piece_id=3J3LKBXHO5E2IYUVRO7NG7BSYXYBQEFRHV23OQHXHFBYBIYOBQ2A, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817769405 (21h51m31.711478014s) [noise=true , address=185.167.204.206:4067 , piece_id=BSRJBOCNWZIYTEP6HY3NOGZORQTPCWKN43E6HPVC37LY5I5CJYNA, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818062553 (21h51m31.711771162s) [noise=true , address=185.167.204.206:4062 , piece_id=3Z6VWEXXIA3ISY73C5JV5S5ROLCBR6DSQWZVI6Q3QSSYLRBNM6LQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818137756 (21h51m31.711846365s) [noise=true , address=89.149.201.147:4012 , piece_id=SPAZ2DY75JXHC2PWPRI3QG7HYCGHM4GVXN5DY5ZWYDTM6637EV7Q, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818415194 (21h51m31.712123793s) [noise=true , address=185.167.204.205:4006 , piece_id=XCZ2NHQZKDA7AADK3JP26Y3WOZMXQ5IJV5THD5V2P5TX3F7WAFOQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818739812 (21h51m31.712448421s) [noise=true , address=89.149.201.147:4023 , piece_id=LQXOUV6UVIHCQTXPVMC7NYINBLEZWF4H7W6BSOWGK6BFG5ARQ3VQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819043791 (21h51m31.7127524s) [noise=true , address=185.167.204.205:4026 , piece_id=SVVM4Q6OCLSV7TWVURRBYN765J44OTG7AFOQS3RZX7GKFMDJ6ZJQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819337630 (21h51m31.713046239s) [noise=true , address=185.167.204.206:4050 , piece_id=E5FLLMTDTPGGIAKZDIIWPBOFWQL7YXZSUA7Z5YSN2KWJOASDJF6Q, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819543422 (21h51m31.713252031s) [noise=true , address=81.171.14.70:4042 , piece_id=PFKTRW3DI6O7EQ4LSDBQM5YQDJ6JQLXCNZTBCNMTYD3SKQWVGGLQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819810982 (21h51m31.713519591s) [noise=true , address=81.171.14.70:4027 , piece_id=7YO3ZAQFFZOSWSG2JH3KOHWDEFQJLOCCQ5FUGSIT7UUREHROOGYA, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:35.646384760 (21h51m32.540093369s) [upload=19196]: Next returned nil segment
Jan 16 13:47:35.646399467 (21h51m32.540108076s) [upload=19196]: Waiting for error group managing segments...
Jan 16 13:47:36.044357389 (21h51m32.938065998s) [noise=true , address=81.171.14.70:4045 , piece_id=GC5TS2METLE6BBQQGYRMWPTMVH3RMRKQJA2RXKQFCCTCJB2JXMBA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.049970457 (21h51m32.943679076s) [noise=true , address=81.171.14.70:4042 , piece_id=PFKTRW3DI6O7EQ4LSDBQM5YQDJ6JQLXCNZTBCNMTYD3SKQWVGGLQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.057550509 (21h51m32.951259118s) [noise=true , address=81.171.14.70:4040 , piece_id=4ZRIS4UHZC6SMLPEIXS6KU6UAKYVIGI6ZMJFRFJ25PY4RB32WVEQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.093832752 (21h51m32.987541361s) [noise=true , address=81.171.14.70:4026 , piece_id=YZUBTL2WRUNDUPW376BG676VAWOU2GMIMQITAKTWIKJ3HKZXNLZA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.370239906 (21h51m33.263948515s) [noise=true , address=185.167.204.206:4067 , piece_id=BSRJBOCNWZIYTEP6HY3NOGZORQTPCWKN43E6HPVC37LY5I5CJYNA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.384125184 (21h51m33.277833793s) [noise=true , address=185.167.204.206:4062 , piece_id=3Z6VWEXXIA3ISY73C5JV5S5ROLCBR6DSQWZVI6Q3QSSYLRBNM6LQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.391293162 (21h51m33.285001771s) [noise=true , address=185.167.204.206:4064 , piece_id=2PDTBJHQW5AHYD2SOM6Q5D6BYGQJCWSWQNLRXCNRPWYBHKBS6B3Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.391398192 (21h51m33.285106801s) [noise=true , address=185.167.204.206:4040 , piece_id=XCGSNFY225UO7MYFSYWQQLVMP73RIUVII7ZYTTGWZ3TQU2DPMOTA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.392101912 (21h51m33.285810521s) [noise=true , address=185.167.204.206:4061 , piece_id=EN3JISM5WLYJWU3UG3BFTNCBOP7MTIFJBRYU4KTKDFVSWCEJRVJA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.400384883 (21h51m33.294093492s) [noise=true , address=185.167.204.206:4039 , piece_id=BJWZY2S2EZSQO6CJM25WV4RIYTMS3P4CG5RHC44JXDMUGA3Z3JEQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.405248333 (21h51m33.298956942s) [noise=true , address=185.167.204.206:4050 , piece_id=E5FLLMTDTPGGIAKZDIIWPBOFWQL7YXZSUA7Z5YSN2KWJOASDJF6Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.414198665 (21h51m33.307907274s) [noise=true , address=185.167.204.205:4006 , piece_id=XCZ2NHQZKDA7AADK3JP26Y3WOZMXQ5IJV5THD5V2P5TX3F7WAFOQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.423293562 (21h51m33.317002171s) [noise=true , address=185.167.204.206:4045 , piece_id=FHIBGSVU2OZBBQETRZNJK6QW7DTCV7ITCEMUGEYQAN6ZYMQQYTOQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.423504314 (21h51m33.317212923s) [noise=true , address=185.167.204.205:4018 , piece_id=3J3LKBXHO5E2IYUVRO7NG7BSYXYBQEFRHV23OQHXHFBYBIYOBQ2A, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.435923832 (21h51m33.329632461s) [noise=true , address=185.167.204.205:4019 , piece_id=NGOWYQ3TZHPLUJ73WVIC3YYL65EYJVBKOPUWCI4DLEGRPQIECBBA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.436584970 (21h51m33.330293579s) [noise=true , address=185.167.204.205:4026 , piece_id=SVVM4Q6OCLSV7TWVURRBYN765J44OTG7AFOQS3RZX7GKFMDJ6ZJQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.491250847 (21h51m33.384959456s) [noise=true , address=81.171.14.70:4031 , piece_id=EF67IQMYFBPUP6UZXDGSQLHR6X4G6OSCKEH4DYWCKI4UIHO4UVDA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.518740798 (21h51m33.412449407s) [noise=true , address=81.171.14.70:4030 , piece_id=SLIXLC6YJ5PLS4MCXJIACYV5MIFOQSBARATYC3WAEZEZSVWICMCQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.526006844 (21h51m33.419715453s) [noise=true , address=81.171.14.70:4025 , piece_id=DCILR24FF45JKLU36SHTMAJLY3NTFNGT4UWNFJXATADZJY2W3TBA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.564445552 (21h51m33.458154161s) [noise=true , address=81.171.14.70:4027 , piece_id=7YO3ZAQFFZOSWSG2JH3KOHWDEFQJLOCCQ5FUGSIT7UUREHROOGYA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.587122821 (21h51m33.48083146s) [noise=true , address=81.171.14.70:4038 , piece_id=VAUCLB55KGVJVL4BSWY4ZY2747BDYFXI2RK3JMPDO4ZOCVOT43EA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.679729699 (21h51m33.573438309s) [noise=true , address=89.149.201.147:4012 , piece_id=SPAZ2DY75JXHC2PWPRI3QG7HYCGHM4GVXN5DY5ZWYDTM6637EV7Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.775377568 (21h51m33.669086177s) [noise=true , address=89.149.201.147:4010 , piece_id=WBP6VM7X2ZISV4K4SE6LL3ZSPYHJQK24IOGILVH6S2ZLWRRJVN7Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:37.160795473 (21h51m34.054504082s) [noise=true , address=89.149.201.147:4023 , piece_id=LQXOUV6UVIHCQTXPVMC7NYINBLEZWF4H7W6BSOWGK6BFG5ARQ3VQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:37.251044932 (21h51m34.144753541s) [noise=true , address=89.149.201.147:4007 , piece_id=J4W7LHO6YZCI3KPZ3F3DN6P4EAV5RIMRQ2X7ZOKLAAQVNGD7HVMA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:48:34.164143586 (21h52m31.057852195s) [noise=true , address=89.149.201.147:4011 , piece_id=TFRZQWAVN2NUSEWM4L33PKZKUYWQYCV3KCRHTKVTABLDZ7GSVVSQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: ecclient: upload failed (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg, address:89.149.201.147:4011): protocol: read tcp 188.166.225.170:48732->89.149.201.147:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 16 13:48:34.164198381 (21h52m31.05790699s) [noise=true , address=89.149.201.147:4020 , piece_id=CQAAHSLJFZE4GHOKB5UO7A2ANJ6N4HHFZDQ33WV25JF76L7JMMDQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: expected piece hash; context deadline exceeded; EOF
Jan 16 13:48:34.164358165 (21h52m31.058066774s) [noise=true , address=89.149.201.147:4014 , piece_id=BNGVISOX55CNKPIHIJJCXFNIMGW5FCE6GWEY4QKLIAWMVYMRAYFA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: context deadline exceeded; context deadline exceeded
Jan 16 13:48:34.164382872 (21h52m31.058091481s) [seg_pos={0 0}, upload=19196]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil>
Jan 16 13:48:34.330174768 (21h52m31.223883387s) [upload=19196]: Done uploading segments...
Jan 16 13:48:34.330206869 (21h52m31.223915478s) [upload=19196]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25)
Duration: 4.715950317s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
58 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..] duration:
4.902900762s trace:
https://jaeger.dev.storj.io/trace/4c07ac9953bead69
FAIL (2026-01-16 14:10:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.902929307s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
55 out of 60 is failed (threshold=3) [X.X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXX.] duration:
30.001764791s trace:
https://jaeger.dev.storj.io/trace/4bdf63507f1525b
FAIL (2026-01-16 14:10:21 UTC)
orbiter warning: context deadline exceeded
Duration: 30.00178986s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
9 out of 60 is failed (threshold=3) [........X.............X..X..X...X..........X.X.X.....X......] duration:
12.353128746s trace:
https://jaeger.dev.storj.io/trace/346659e7312f1ccc
FAIL (2026-01-16 14:10:15 UTC)
orbiter warning: context deadline exceeded
Duration: 12.353158914s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
15.368829271s trace:
https://jaeger.dev.storj.io/trace/a3cda5286a23a90
FAIL (2026-01-16 14:09:54 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 15.368842507s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
6 out of 60 is failed (threshold=3) [.............X........X..X.XX.....X.........................] duration:
4.847783144s trace:
https://jaeger.dev.storj.io/trace/18ea158581539e05
FAIL (2026-01-16 14:09:35 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 25)
======LOGS======
: Uploading piece...
Jan 16 13:47:34.817121271 (21h51m31.71082988s) [noise=true , address=81.171.14.70:4030 , piece_id=SLIXLC6YJ5PLS4MCXJIACYV5MIFOQSBARATYC3WAEZEZSVWICMCQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817237642 (21h51m31.710946251s) [noise=true , address=81.171.14.70:4040 , piece_id=4ZRIS4UHZC6SMLPEIXS6KU6UAKYVIGI6ZMJFRFJ25PY4RB32WVEQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817351119 (21h51m31.711059728s) [noise=true , address=89.149.201.147:4011 , piece_id=TFRZQWAVN2NUSEWM4L33PKZKUYWQYCV3KCRHTKVTABLDZ7GSVVSQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817710643 (21h51m31.711419282s) [noise=true , address=185.167.204.205:4018 , piece_id=3J3LKBXHO5E2IYUVRO7NG7BSYXYBQEFRHV23OQHXHFBYBIYOBQ2A, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817769405 (21h51m31.711478014s) [noise=true , address=185.167.204.206:4067 , piece_id=BSRJBOCNWZIYTEP6HY3NOGZORQTPCWKN43E6HPVC37LY5I5CJYNA, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818062553 (21h51m31.711771162s) [noise=true , address=185.167.204.206:4062 , piece_id=3Z6VWEXXIA3ISY73C5JV5S5ROLCBR6DSQWZVI6Q3QSSYLRBNM6LQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818137756 (21h51m31.711846365s) [noise=true , address=89.149.201.147:4012 , piece_id=SPAZ2DY75JXHC2PWPRI3QG7HYCGHM4GVXN5DY5ZWYDTM6637EV7Q, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818415194 (21h51m31.712123793s) [noise=true , address=185.167.204.205:4006 , piece_id=XCZ2NHQZKDA7AADK3JP26Y3WOZMXQ5IJV5THD5V2P5TX3F7WAFOQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818739812 (21h51m31.712448421s) [noise=true , address=89.149.201.147:4023 , piece_id=LQXOUV6UVIHCQTXPVMC7NYINBLEZWF4H7W6BSOWGK6BFG5ARQ3VQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819043791 (21h51m31.7127524s) [noise=true , address=185.167.204.205:4026 , piece_id=SVVM4Q6OCLSV7TWVURRBYN765J44OTG7AFOQS3RZX7GKFMDJ6ZJQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819337630 (21h51m31.713046239s) [noise=true , address=185.167.204.206:4050 , piece_id=E5FLLMTDTPGGIAKZDIIWPBOFWQL7YXZSUA7Z5YSN2KWJOASDJF6Q, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819543422 (21h51m31.713252031s) [noise=true , address=81.171.14.70:4042 , piece_id=PFKTRW3DI6O7EQ4LSDBQM5YQDJ6JQLXCNZTBCNMTYD3SKQWVGGLQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819810982 (21h51m31.713519591s) [noise=true , address=81.171.14.70:4027 , piece_id=7YO3ZAQFFZOSWSG2JH3KOHWDEFQJLOCCQ5FUGSIT7UUREHROOGYA, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:35.646384760 (21h51m32.540093369s) [upload=19196]: Next returned nil segment
Jan 16 13:47:35.646399467 (21h51m32.540108076s) [upload=19196]: Waiting for error group managing segments...
Jan 16 13:47:36.044357389 (21h51m32.938065998s) [noise=true , address=81.171.14.70:4045 , piece_id=GC5TS2METLE6BBQQGYRMWPTMVH3RMRKQJA2RXKQFCCTCJB2JXMBA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.049970457 (21h51m32.943679076s) [noise=true , address=81.171.14.70:4042 , piece_id=PFKTRW3DI6O7EQ4LSDBQM5YQDJ6JQLXCNZTBCNMTYD3SKQWVGGLQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.057550509 (21h51m32.951259118s) [noise=true , address=81.171.14.70:4040 , piece_id=4ZRIS4UHZC6SMLPEIXS6KU6UAKYVIGI6ZMJFRFJ25PY4RB32WVEQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.093832752 (21h51m32.987541361s) [noise=true , address=81.171.14.70:4026 , piece_id=YZUBTL2WRUNDUPW376BG676VAWOU2GMIMQITAKTWIKJ3HKZXNLZA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.370239906 (21h51m33.263948515s) [noise=true , address=185.167.204.206:4067 , piece_id=BSRJBOCNWZIYTEP6HY3NOGZORQTPCWKN43E6HPVC37LY5I5CJYNA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.384125184 (21h51m33.277833793s) [noise=true , address=185.167.204.206:4062 , piece_id=3Z6VWEXXIA3ISY73C5JV5S5ROLCBR6DSQWZVI6Q3QSSYLRBNM6LQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.391293162 (21h51m33.285001771s) [noise=true , address=185.167.204.206:4064 , piece_id=2PDTBJHQW5AHYD2SOM6Q5D6BYGQJCWSWQNLRXCNRPWYBHKBS6B3Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.391398192 (21h51m33.285106801s) [noise=true , address=185.167.204.206:4040 , piece_id=XCGSNFY225UO7MYFSYWQQLVMP73RIUVII7ZYTTGWZ3TQU2DPMOTA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.392101912 (21h51m33.285810521s) [noise=true , address=185.167.204.206:4061 , piece_id=EN3JISM5WLYJWU3UG3BFTNCBOP7MTIFJBRYU4KTKDFVSWCEJRVJA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.400384883 (21h51m33.294093492s) [noise=true , address=185.167.204.206:4039 , piece_id=BJWZY2S2EZSQO6CJM25WV4RIYTMS3P4CG5RHC44JXDMUGA3Z3JEQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.405248333 (21h51m33.298956942s) [noise=true , address=185.167.204.206:4050 , piece_id=E5FLLMTDTPGGIAKZDIIWPBOFWQL7YXZSUA7Z5YSN2KWJOASDJF6Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.414198665 (21h51m33.307907274s) [noise=true , address=185.167.204.205:4006 , piece_id=XCZ2NHQZKDA7AADK3JP26Y3WOZMXQ5IJV5THD5V2P5TX3F7WAFOQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.423293562 (21h51m33.317002171s) [noise=true , address=185.167.204.206:4045 , piece_id=FHIBGSVU2OZBBQETRZNJK6QW7DTCV7ITCEMUGEYQAN6ZYMQQYTOQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.423504314 (21h51m33.317212923s) [noise=true , address=185.167.204.205:4018 , piece_id=3J3LKBXHO5E2IYUVRO7NG7BSYXYBQEFRHV23OQHXHFBYBIYOBQ2A, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.435923832 (21h51m33.329632461s) [noise=true , address=185.167.204.205:4019 , piece_id=NGOWYQ3TZHPLUJ73WVIC3YYL65EYJVBKOPUWCI4DLEGRPQIECBBA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.436584970 (21h51m33.330293579s) [noise=true , address=185.167.204.205:4026 , piece_id=SVVM4Q6OCLSV7TWVURRBYN765J44OTG7AFOQS3RZX7GKFMDJ6ZJQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.491250847 (21h51m33.384959456s) [noise=true , address=81.171.14.70:4031 , piece_id=EF67IQMYFBPUP6UZXDGSQLHR6X4G6OSCKEH4DYWCKI4UIHO4UVDA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.518740798 (21h51m33.412449407s) [noise=true , address=81.171.14.70:4030 , piece_id=SLIXLC6YJ5PLS4MCXJIACYV5MIFOQSBARATYC3WAEZEZSVWICMCQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.526006844 (21h51m33.419715453s) [noise=true , address=81.171.14.70:4025 , piece_id=DCILR24FF45JKLU36SHTMAJLY3NTFNGT4UWNFJXATADZJY2W3TBA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.564445552 (21h51m33.458154161s) [noise=true , address=81.171.14.70:4027 , piece_id=7YO3ZAQFFZOSWSG2JH3KOHWDEFQJLOCCQ5FUGSIT7UUREHROOGYA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.587122821 (21h51m33.48083146s) [noise=true , address=81.171.14.70:4038 , piece_id=VAUCLB55KGVJVL4BSWY4ZY2747BDYFXI2RK3JMPDO4ZOCVOT43EA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.679729699 (21h51m33.573438309s) [noise=true , address=89.149.201.147:4012 , piece_id=SPAZ2DY75JXHC2PWPRI3QG7HYCGHM4GVXN5DY5ZWYDTM6637EV7Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.775377568 (21h51m33.669086177s) [noise=true , address=89.149.201.147:4010 , piece_id=WBP6VM7X2ZISV4K4SE6LL3ZSPYHJQK24IOGILVH6S2ZLWRRJVN7Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:37.160795473 (21h51m34.054504082s) [noise=true , address=89.149.201.147:4023 , piece_id=LQXOUV6UVIHCQTXPVMC7NYINBLEZWF4H7W6BSOWGK6BFG5ARQ3VQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:37.251044932 (21h51m34.144753541s) [noise=true , address=89.149.201.147:4007 , piece_id=J4W7LHO6YZCI3KPZ3F3DN6P4EAV5RIMRQ2X7ZOKLAAQVNGD7HVMA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:48:34.164143586 (21h52m31.057852195s) [noise=true , address=89.149.201.147:4011 , piece_id=TFRZQWAVN2NUSEWM4L33PKZKUYWQYCV3KCRHTKVTABLDZ7GSVVSQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: ecclient: upload failed (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg, address:89.149.201.147:4011): protocol: read tcp 188.166.225.170:48732->89.149.201.147:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 16 13:48:34.164198381 (21h52m31.05790699s) [noise=true , address=89.149.201.147:4020 , piece_id=CQAAHSLJFZE4GHOKB5UO7A2ANJ6N4HHFZDQ33WV25JF76L7JMMDQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: expected piece hash; context deadline exceeded; EOF
Jan 16 13:48:34.164358165 (21h52m31.058066774s) [noise=true , address=89.149.201.147:4014 , piece_id=BNGVISOX55CNKPIHIJJCXFNIMGW5FCE6GWEY4QKLIAWMVYMRAYFA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: context deadline exceeded; context deadline exceeded
Jan 16 13:48:34.164382872 (21h52m31.058091481s) [seg_pos={0 0}, upload=19196]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil>
Jan 16 13:48:34.330174768 (21h52m31.223883387s) [upload=19196]: Done uploading segments...
Jan 16 13:48:34.330206869 (21h52m31.223915478s) [upload=19196]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25)
Duration: 4.847810377s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration:
4.691695054s trace:
https://jaeger.dev.storj.io/trace/4c07ac9953bead69
FAIL (2026-01-16 14:09:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.691740711s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
9 out of 60 is failed (threshold=3) [.........X.............X..X..X...X..........X.X.X.....X.....] duration:
22.922770328s trace:
https://jaeger.dev.storj.io/trace/346659e7312f1ccc
FAIL (2026-01-16 14:09:22 UTC)
orbiter warning: context deadline exceeded
Duration: 22.922794564s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
56 out of 60 is failed (threshold=3) [XX.X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXX] duration:
30.001705092s trace:
https://jaeger.dev.storj.io/trace/4bdf63507f1525b
FAIL (2026-01-16 14:09:21 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001728376s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001313804s trace:
https://jaeger.dev.storj.io/trace/4c07ac9953bead69
FAIL (2026-01-16 14:09:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001335054s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
7 out of 60 is failed (threshold=3) [X.............X........X..X.XX.....X........................] duration:
4.500035486s trace:
https://jaeger.dev.storj.io/trace/18ea158581539e05
FAIL (2026-01-16 14:09:03 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 25)
======LOGS======
: Uploading piece...
Jan 16 13:47:34.817121271 (21h51m31.71082988s) [noise=true , address=81.171.14.70:4030 , piece_id=SLIXLC6YJ5PLS4MCXJIACYV5MIFOQSBARATYC3WAEZEZSVWICMCQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817237642 (21h51m31.710946251s) [noise=true , address=81.171.14.70:4040 , piece_id=4ZRIS4UHZC6SMLPEIXS6KU6UAKYVIGI6ZMJFRFJ25PY4RB32WVEQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817351119 (21h51m31.711059728s) [noise=true , address=89.149.201.147:4011 , piece_id=TFRZQWAVN2NUSEWM4L33PKZKUYWQYCV3KCRHTKVTABLDZ7GSVVSQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817710643 (21h51m31.711419282s) [noise=true , address=185.167.204.205:4018 , piece_id=3J3LKBXHO5E2IYUVRO7NG7BSYXYBQEFRHV23OQHXHFBYBIYOBQ2A, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817769405 (21h51m31.711478014s) [noise=true , address=185.167.204.206:4067 , piece_id=BSRJBOCNWZIYTEP6HY3NOGZORQTPCWKN43E6HPVC37LY5I5CJYNA, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818062553 (21h51m31.711771162s) [noise=true , address=185.167.204.206:4062 , piece_id=3Z6VWEXXIA3ISY73C5JV5S5ROLCBR6DSQWZVI6Q3QSSYLRBNM6LQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818137756 (21h51m31.711846365s) [noise=true , address=89.149.201.147:4012 , piece_id=SPAZ2DY75JXHC2PWPRI3QG7HYCGHM4GVXN5DY5ZWYDTM6637EV7Q, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818415194 (21h51m31.712123793s) [noise=true , address=185.167.204.205:4006 , piece_id=XCZ2NHQZKDA7AADK3JP26Y3WOZMXQ5IJV5THD5V2P5TX3F7WAFOQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818739812 (21h51m31.712448421s) [noise=true , address=89.149.201.147:4023 , piece_id=LQXOUV6UVIHCQTXPVMC7NYINBLEZWF4H7W6BSOWGK6BFG5ARQ3VQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819043791 (21h51m31.7127524s) [noise=true , address=185.167.204.205:4026 , piece_id=SVVM4Q6OCLSV7TWVURRBYN765J44OTG7AFOQS3RZX7GKFMDJ6ZJQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819337630 (21h51m31.713046239s) [noise=true , address=185.167.204.206:4050 , piece_id=E5FLLMTDTPGGIAKZDIIWPBOFWQL7YXZSUA7Z5YSN2KWJOASDJF6Q, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819543422 (21h51m31.713252031s) [noise=true , address=81.171.14.70:4042 , piece_id=PFKTRW3DI6O7EQ4LSDBQM5YQDJ6JQLXCNZTBCNMTYD3SKQWVGGLQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819810982 (21h51m31.713519591s) [noise=true , address=81.171.14.70:4027 , piece_id=7YO3ZAQFFZOSWSG2JH3KOHWDEFQJLOCCQ5FUGSIT7UUREHROOGYA, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:35.646384760 (21h51m32.540093369s) [upload=19196]: Next returned nil segment
Jan 16 13:47:35.646399467 (21h51m32.540108076s) [upload=19196]: Waiting for error group managing segments...
Jan 16 13:47:36.044357389 (21h51m32.938065998s) [noise=true , address=81.171.14.70:4045 , piece_id=GC5TS2METLE6BBQQGYRMWPTMVH3RMRKQJA2RXKQFCCTCJB2JXMBA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.049970457 (21h51m32.943679076s) [noise=true , address=81.171.14.70:4042 , piece_id=PFKTRW3DI6O7EQ4LSDBQM5YQDJ6JQLXCNZTBCNMTYD3SKQWVGGLQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.057550509 (21h51m32.951259118s) [noise=true , address=81.171.14.70:4040 , piece_id=4ZRIS4UHZC6SMLPEIXS6KU6UAKYVIGI6ZMJFRFJ25PY4RB32WVEQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.093832752 (21h51m32.987541361s) [noise=true , address=81.171.14.70:4026 , piece_id=YZUBTL2WRUNDUPW376BG676VAWOU2GMIMQITAKTWIKJ3HKZXNLZA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.370239906 (21h51m33.263948515s) [noise=true , address=185.167.204.206:4067 , piece_id=BSRJBOCNWZIYTEP6HY3NOGZORQTPCWKN43E6HPVC37LY5I5CJYNA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.384125184 (21h51m33.277833793s) [noise=true , address=185.167.204.206:4062 , piece_id=3Z6VWEXXIA3ISY73C5JV5S5ROLCBR6DSQWZVI6Q3QSSYLRBNM6LQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.391293162 (21h51m33.285001771s) [noise=true , address=185.167.204.206:4064 , piece_id=2PDTBJHQW5AHYD2SOM6Q5D6BYGQJCWSWQNLRXCNRPWYBHKBS6B3Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.391398192 (21h51m33.285106801s) [noise=true , address=185.167.204.206:4040 , piece_id=XCGSNFY225UO7MYFSYWQQLVMP73RIUVII7ZYTTGWZ3TQU2DPMOTA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.392101912 (21h51m33.285810521s) [noise=true , address=185.167.204.206:4061 , piece_id=EN3JISM5WLYJWU3UG3BFTNCBOP7MTIFJBRYU4KTKDFVSWCEJRVJA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.400384883 (21h51m33.294093492s) [noise=true , address=185.167.204.206:4039 , piece_id=BJWZY2S2EZSQO6CJM25WV4RIYTMS3P4CG5RHC44JXDMUGA3Z3JEQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.405248333 (21h51m33.298956942s) [noise=true , address=185.167.204.206:4050 , piece_id=E5FLLMTDTPGGIAKZDIIWPBOFWQL7YXZSUA7Z5YSN2KWJOASDJF6Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.414198665 (21h51m33.307907274s) [noise=true , address=185.167.204.205:4006 , piece_id=XCZ2NHQZKDA7AADK3JP26Y3WOZMXQ5IJV5THD5V2P5TX3F7WAFOQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.423293562 (21h51m33.317002171s) [noise=true , address=185.167.204.206:4045 , piece_id=FHIBGSVU2OZBBQETRZNJK6QW7DTCV7ITCEMUGEYQAN6ZYMQQYTOQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.423504314 (21h51m33.317212923s) [noise=true , address=185.167.204.205:4018 , piece_id=3J3LKBXHO5E2IYUVRO7NG7BSYXYBQEFRHV23OQHXHFBYBIYOBQ2A, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.435923832 (21h51m33.329632461s) [noise=true , address=185.167.204.205:4019 , piece_id=NGOWYQ3TZHPLUJ73WVIC3YYL65EYJVBKOPUWCI4DLEGRPQIECBBA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.436584970 (21h51m33.330293579s) [noise=true , address=185.167.204.205:4026 , piece_id=SVVM4Q6OCLSV7TWVURRBYN765J44OTG7AFOQS3RZX7GKFMDJ6ZJQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.491250847 (21h51m33.384959456s) [noise=true , address=81.171.14.70:4031 , piece_id=EF67IQMYFBPUP6UZXDGSQLHR6X4G6OSCKEH4DYWCKI4UIHO4UVDA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.518740798 (21h51m33.412449407s) [noise=true , address=81.171.14.70:4030 , piece_id=SLIXLC6YJ5PLS4MCXJIACYV5MIFOQSBARATYC3WAEZEZSVWICMCQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.526006844 (21h51m33.419715453s) [noise=true , address=81.171.14.70:4025 , piece_id=DCILR24FF45JKLU36SHTMAJLY3NTFNGT4UWNFJXATADZJY2W3TBA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.564445552 (21h51m33.458154161s) [noise=true , address=81.171.14.70:4027 , piece_id=7YO3ZAQFFZOSWSG2JH3KOHWDEFQJLOCCQ5FUGSIT7UUREHROOGYA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.587122821 (21h51m33.48083146s) [noise=true , address=81.171.14.70:4038 , piece_id=VAUCLB55KGVJVL4BSWY4ZY2747BDYFXI2RK3JMPDO4ZOCVOT43EA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.679729699 (21h51m33.573438309s) [noise=true , address=89.149.201.147:4012 , piece_id=SPAZ2DY75JXHC2PWPRI3QG7HYCGHM4GVXN5DY5ZWYDTM6637EV7Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.775377568 (21h51m33.669086177s) [noise=true , address=89.149.201.147:4010 , piece_id=WBP6VM7X2ZISV4K4SE6LL3ZSPYHJQK24IOGILVH6S2ZLWRRJVN7Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:37.160795473 (21h51m34.054504082s) [noise=true , address=89.149.201.147:4023 , piece_id=LQXOUV6UVIHCQTXPVMC7NYINBLEZWF4H7W6BSOWGK6BFG5ARQ3VQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:37.251044932 (21h51m34.144753541s) [noise=true , address=89.149.201.147:4007 , piece_id=J4W7LHO6YZCI3KPZ3F3DN6P4EAV5RIMRQ2X7ZOKLAAQVNGD7HVMA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:48:34.164143586 (21h52m31.057852195s) [noise=true , address=89.149.201.147:4011 , piece_id=TFRZQWAVN2NUSEWM4L33PKZKUYWQYCV3KCRHTKVTABLDZ7GSVVSQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: ecclient: upload failed (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg, address:89.149.201.147:4011): protocol: read tcp 188.166.225.170:48732->89.149.201.147:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 16 13:48:34.164198381 (21h52m31.05790699s) [noise=true , address=89.149.201.147:4020 , piece_id=CQAAHSLJFZE4GHOKB5UO7A2ANJ6N4HHFZDQ33WV25JF76L7JMMDQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: expected piece hash; context deadline exceeded; EOF
Jan 16 13:48:34.164358165 (21h52m31.058066774s) [noise=true , address=89.149.201.147:4014 , piece_id=BNGVISOX55CNKPIHIJJCXFNIMGW5FCE6GWEY4QKLIAWMVYMRAYFA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: context deadline exceeded; context deadline exceeded
Jan 16 13:48:34.164382872 (21h52m31.058091481s) [seg_pos={0 0}, upload=19196]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil>
Jan 16 13:48:34.330174768 (21h52m31.223883387s) [upload=19196]: Done uploading segments...
Jan 16 13:48:34.330206869 (21h52m31.223915478s) [upload=19196]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25)
Duration: 4.500063129s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
13.427291161s trace:
https://jaeger.dev.storj.io/trace/a3cda5286a23a90
FAIL (2026-01-16 14:08:50 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 13.427305768s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
21.266006133s trace:
https://jaeger.dev.storj.io/trace/a3cda5286a23a90
FAIL (2026-01-16 14:08:36 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 21.266022905s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
9 out of 60 is failed (threshold=3) [..........X.............X..X..X...X..........X.X.X.....X....] duration:
25.428544799s trace:
https://jaeger.dev.storj.io/trace/346659e7312f1ccc
FAIL (2026-01-16 14:08:19 UTC)
orbiter warning: context deadline exceeded
Duration: 25.428569596s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
56 out of 60 is failed (threshold=3) [XXX.X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX] duration:
30.001955582s trace:
https://jaeger.dev.storj.io/trace/75facb3615d4d4d2
FAIL (2026-01-16 14:08:12 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001967495s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001233571s trace:
https://jaeger.dev.storj.io/trace/3cd379df5dc2d4f
FAIL (2026-01-16 14:08:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001256174s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
7 out of 60 is failed (threshold=3) [.X.............X........X..X.XX.....X.......................] duration:
4.515120028s trace:
https://jaeger.dev.storj.io/trace/18ea158581539e05
FAIL (2026-01-16 14:07:56 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 25)
======LOGS======
: Uploading piece...
Jan 16 13:47:34.817121271 (21h51m31.71082988s) [noise=true , address=81.171.14.70:4030 , piece_id=SLIXLC6YJ5PLS4MCXJIACYV5MIFOQSBARATYC3WAEZEZSVWICMCQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817237642 (21h51m31.710946251s) [noise=true , address=81.171.14.70:4040 , piece_id=4ZRIS4UHZC6SMLPEIXS6KU6UAKYVIGI6ZMJFRFJ25PY4RB32WVEQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817351119 (21h51m31.711059728s) [noise=true , address=89.149.201.147:4011 , piece_id=TFRZQWAVN2NUSEWM4L33PKZKUYWQYCV3KCRHTKVTABLDZ7GSVVSQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817710643 (21h51m31.711419282s) [noise=true , address=185.167.204.205:4018 , piece_id=3J3LKBXHO5E2IYUVRO7NG7BSYXYBQEFRHV23OQHXHFBYBIYOBQ2A, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817769405 (21h51m31.711478014s) [noise=true , address=185.167.204.206:4067 , piece_id=BSRJBOCNWZIYTEP6HY3NOGZORQTPCWKN43E6HPVC37LY5I5CJYNA, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818062553 (21h51m31.711771162s) [noise=true , address=185.167.204.206:4062 , piece_id=3Z6VWEXXIA3ISY73C5JV5S5ROLCBR6DSQWZVI6Q3QSSYLRBNM6LQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818137756 (21h51m31.711846365s) [noise=true , address=89.149.201.147:4012 , piece_id=SPAZ2DY75JXHC2PWPRI3QG7HYCGHM4GVXN5DY5ZWYDTM6637EV7Q, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818415194 (21h51m31.712123793s) [noise=true , address=185.167.204.205:4006 , piece_id=XCZ2NHQZKDA7AADK3JP26Y3WOZMXQ5IJV5THD5V2P5TX3F7WAFOQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818739812 (21h51m31.712448421s) [noise=true , address=89.149.201.147:4023 , piece_id=LQXOUV6UVIHCQTXPVMC7NYINBLEZWF4H7W6BSOWGK6BFG5ARQ3VQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819043791 (21h51m31.7127524s) [noise=true , address=185.167.204.205:4026 , piece_id=SVVM4Q6OCLSV7TWVURRBYN765J44OTG7AFOQS3RZX7GKFMDJ6ZJQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819337630 (21h51m31.713046239s) [noise=true , address=185.167.204.206:4050 , piece_id=E5FLLMTDTPGGIAKZDIIWPBOFWQL7YXZSUA7Z5YSN2KWJOASDJF6Q, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819543422 (21h51m31.713252031s) [noise=true , address=81.171.14.70:4042 , piece_id=PFKTRW3DI6O7EQ4LSDBQM5YQDJ6JQLXCNZTBCNMTYD3SKQWVGGLQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819810982 (21h51m31.713519591s) [noise=true , address=81.171.14.70:4027 , piece_id=7YO3ZAQFFZOSWSG2JH3KOHWDEFQJLOCCQ5FUGSIT7UUREHROOGYA, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:35.646384760 (21h51m32.540093369s) [upload=19196]: Next returned nil segment
Jan 16 13:47:35.646399467 (21h51m32.540108076s) [upload=19196]: Waiting for error group managing segments...
Jan 16 13:47:36.044357389 (21h51m32.938065998s) [noise=true , address=81.171.14.70:4045 , piece_id=GC5TS2METLE6BBQQGYRMWPTMVH3RMRKQJA2RXKQFCCTCJB2JXMBA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.049970457 (21h51m32.943679076s) [noise=true , address=81.171.14.70:4042 , piece_id=PFKTRW3DI6O7EQ4LSDBQM5YQDJ6JQLXCNZTBCNMTYD3SKQWVGGLQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.057550509 (21h51m32.951259118s) [noise=true , address=81.171.14.70:4040 , piece_id=4ZRIS4UHZC6SMLPEIXS6KU6UAKYVIGI6ZMJFRFJ25PY4RB32WVEQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.093832752 (21h51m32.987541361s) [noise=true , address=81.171.14.70:4026 , piece_id=YZUBTL2WRUNDUPW376BG676VAWOU2GMIMQITAKTWIKJ3HKZXNLZA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.370239906 (21h51m33.263948515s) [noise=true , address=185.167.204.206:4067 , piece_id=BSRJBOCNWZIYTEP6HY3NOGZORQTPCWKN43E6HPVC37LY5I5CJYNA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.384125184 (21h51m33.277833793s) [noise=true , address=185.167.204.206:4062 , piece_id=3Z6VWEXXIA3ISY73C5JV5S5ROLCBR6DSQWZVI6Q3QSSYLRBNM6LQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.391293162 (21h51m33.285001771s) [noise=true , address=185.167.204.206:4064 , piece_id=2PDTBJHQW5AHYD2SOM6Q5D6BYGQJCWSWQNLRXCNRPWYBHKBS6B3Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.391398192 (21h51m33.285106801s) [noise=true , address=185.167.204.206:4040 , piece_id=XCGSNFY225UO7MYFSYWQQLVMP73RIUVII7ZYTTGWZ3TQU2DPMOTA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.392101912 (21h51m33.285810521s) [noise=true , address=185.167.204.206:4061 , piece_id=EN3JISM5WLYJWU3UG3BFTNCBOP7MTIFJBRYU4KTKDFVSWCEJRVJA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.400384883 (21h51m33.294093492s) [noise=true , address=185.167.204.206:4039 , piece_id=BJWZY2S2EZSQO6CJM25WV4RIYTMS3P4CG5RHC44JXDMUGA3Z3JEQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.405248333 (21h51m33.298956942s) [noise=true , address=185.167.204.206:4050 , piece_id=E5FLLMTDTPGGIAKZDIIWPBOFWQL7YXZSUA7Z5YSN2KWJOASDJF6Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.414198665 (21h51m33.307907274s) [noise=true , address=185.167.204.205:4006 , piece_id=XCZ2NHQZKDA7AADK3JP26Y3WOZMXQ5IJV5THD5V2P5TX3F7WAFOQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.423293562 (21h51m33.317002171s) [noise=true , address=185.167.204.206:4045 , piece_id=FHIBGSVU2OZBBQETRZNJK6QW7DTCV7ITCEMUGEYQAN6ZYMQQYTOQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.423504314 (21h51m33.317212923s) [noise=true , address=185.167.204.205:4018 , piece_id=3J3LKBXHO5E2IYUVRO7NG7BSYXYBQEFRHV23OQHXHFBYBIYOBQ2A, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.435923832 (21h51m33.329632461s) [noise=true , address=185.167.204.205:4019 , piece_id=NGOWYQ3TZHPLUJ73WVIC3YYL65EYJVBKOPUWCI4DLEGRPQIECBBA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.436584970 (21h51m33.330293579s) [noise=true , address=185.167.204.205:4026 , piece_id=SVVM4Q6OCLSV7TWVURRBYN765J44OTG7AFOQS3RZX7GKFMDJ6ZJQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.491250847 (21h51m33.384959456s) [noise=true , address=81.171.14.70:4031 , piece_id=EF67IQMYFBPUP6UZXDGSQLHR6X4G6OSCKEH4DYWCKI4UIHO4UVDA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.518740798 (21h51m33.412449407s) [noise=true , address=81.171.14.70:4030 , piece_id=SLIXLC6YJ5PLS4MCXJIACYV5MIFOQSBARATYC3WAEZEZSVWICMCQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.526006844 (21h51m33.419715453s) [noise=true , address=81.171.14.70:4025 , piece_id=DCILR24FF45JKLU36SHTMAJLY3NTFNGT4UWNFJXATADZJY2W3TBA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.564445552 (21h51m33.458154161s) [noise=true , address=81.171.14.70:4027 , piece_id=7YO3ZAQFFZOSWSG2JH3KOHWDEFQJLOCCQ5FUGSIT7UUREHROOGYA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.587122821 (21h51m33.48083146s) [noise=true , address=81.171.14.70:4038 , piece_id=VAUCLB55KGVJVL4BSWY4ZY2747BDYFXI2RK3JMPDO4ZOCVOT43EA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.679729699 (21h51m33.573438309s) [noise=true , address=89.149.201.147:4012 , piece_id=SPAZ2DY75JXHC2PWPRI3QG7HYCGHM4GVXN5DY5ZWYDTM6637EV7Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.775377568 (21h51m33.669086177s) [noise=true , address=89.149.201.147:4010 , piece_id=WBP6VM7X2ZISV4K4SE6LL3ZSPYHJQK24IOGILVH6S2ZLWRRJVN7Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:37.160795473 (21h51m34.054504082s) [noise=true , address=89.149.201.147:4023 , piece_id=LQXOUV6UVIHCQTXPVMC7NYINBLEZWF4H7W6BSOWGK6BFG5ARQ3VQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:37.251044932 (21h51m34.144753541s) [noise=true , address=89.149.201.147:4007 , piece_id=J4W7LHO6YZCI3KPZ3F3DN6P4EAV5RIMRQ2X7ZOKLAAQVNGD7HVMA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:48:34.164143586 (21h52m31.057852195s) [noise=true , address=89.149.201.147:4011 , piece_id=TFRZQWAVN2NUSEWM4L33PKZKUYWQYCV3KCRHTKVTABLDZ7GSVVSQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: ecclient: upload failed (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg, address:89.149.201.147:4011): protocol: read tcp 188.166.225.170:48732->89.149.201.147:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 16 13:48:34.164198381 (21h52m31.05790699s) [noise=true , address=89.149.201.147:4020 , piece_id=CQAAHSLJFZE4GHOKB5UO7A2ANJ6N4HHFZDQ33WV25JF76L7JMMDQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: expected piece hash; context deadline exceeded; EOF
Jan 16 13:48:34.164358165 (21h52m31.058066774s) [noise=true , address=89.149.201.147:4014 , piece_id=BNGVISOX55CNKPIHIJJCXFNIMGW5FCE6GWEY4QKLIAWMVYMRAYFA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: context deadline exceeded; context deadline exceeded
Jan 16 13:48:34.164382872 (21h52m31.058091481s) [seg_pos={0 0}, upload=19196]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil>
Jan 16 13:48:34.330174768 (21h52m31.223883387s) [upload=19196]: Done uploading segments...
Jan 16 13:48:34.330206869 (21h52m31.223915478s) [upload=19196]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25)
Duration: 4.515150346s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001488066s trace:
https://jaeger.dev.storj.io/trace/6119b28a58a8e222
FAIL (2026-01-16 14:07:44 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001509527s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
55 out of 60 is failed (threshold=3) [.XXX.X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX] duration:
30.001390852s trace:
https://jaeger.dev.storj.io/trace/452e83b015ca76e1
FAIL (2026-01-16 14:07:42 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001413916s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
8.617897515s trace:
https://jaeger.dev.storj.io/trace/a3cda5286a23a90
FAIL (2026-01-16 14:07:40 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 8.617926921s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
7 out of 60 is failed (threshold=3) [..X.............X........X..X.XX.....X......................] duration:
5.273530808s trace:
https://jaeger.dev.storj.io/trace/18ea158581539e05
FAIL (2026-01-16 14:07:21 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 25)
======LOGS======
: Uploading piece...
Jan 16 13:47:34.817121271 (21h51m31.71082988s) [noise=true , address=81.171.14.70:4030 , piece_id=SLIXLC6YJ5PLS4MCXJIACYV5MIFOQSBARATYC3WAEZEZSVWICMCQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817237642 (21h51m31.710946251s) [noise=true , address=81.171.14.70:4040 , piece_id=4ZRIS4UHZC6SMLPEIXS6KU6UAKYVIGI6ZMJFRFJ25PY4RB32WVEQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817351119 (21h51m31.711059728s) [noise=true , address=89.149.201.147:4011 , piece_id=TFRZQWAVN2NUSEWM4L33PKZKUYWQYCV3KCRHTKVTABLDZ7GSVVSQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817710643 (21h51m31.711419282s) [noise=true , address=185.167.204.205:4018 , piece_id=3J3LKBXHO5E2IYUVRO7NG7BSYXYBQEFRHV23OQHXHFBYBIYOBQ2A, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817769405 (21h51m31.711478014s) [noise=true , address=185.167.204.206:4067 , piece_id=BSRJBOCNWZIYTEP6HY3NOGZORQTPCWKN43E6HPVC37LY5I5CJYNA, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818062553 (21h51m31.711771162s) [noise=true , address=185.167.204.206:4062 , piece_id=3Z6VWEXXIA3ISY73C5JV5S5ROLCBR6DSQWZVI6Q3QSSYLRBNM6LQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818137756 (21h51m31.711846365s) [noise=true , address=89.149.201.147:4012 , piece_id=SPAZ2DY75JXHC2PWPRI3QG7HYCGHM4GVXN5DY5ZWYDTM6637EV7Q, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818415194 (21h51m31.712123793s) [noise=true , address=185.167.204.205:4006 , piece_id=XCZ2NHQZKDA7AADK3JP26Y3WOZMXQ5IJV5THD5V2P5TX3F7WAFOQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818739812 (21h51m31.712448421s) [noise=true , address=89.149.201.147:4023 , piece_id=LQXOUV6UVIHCQTXPVMC7NYINBLEZWF4H7W6BSOWGK6BFG5ARQ3VQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819043791 (21h51m31.7127524s) [noise=true , address=185.167.204.205:4026 , piece_id=SVVM4Q6OCLSV7TWVURRBYN765J44OTG7AFOQS3RZX7GKFMDJ6ZJQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819337630 (21h51m31.713046239s) [noise=true , address=185.167.204.206:4050 , piece_id=E5FLLMTDTPGGIAKZDIIWPBOFWQL7YXZSUA7Z5YSN2KWJOASDJF6Q, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819543422 (21h51m31.713252031s) [noise=true , address=81.171.14.70:4042 , piece_id=PFKTRW3DI6O7EQ4LSDBQM5YQDJ6JQLXCNZTBCNMTYD3SKQWVGGLQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819810982 (21h51m31.713519591s) [noise=true , address=81.171.14.70:4027 , piece_id=7YO3ZAQFFZOSWSG2JH3KOHWDEFQJLOCCQ5FUGSIT7UUREHROOGYA, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:35.646384760 (21h51m32.540093369s) [upload=19196]: Next returned nil segment
Jan 16 13:47:35.646399467 (21h51m32.540108076s) [upload=19196]: Waiting for error group managing segments...
Jan 16 13:47:36.044357389 (21h51m32.938065998s) [noise=true , address=81.171.14.70:4045 , piece_id=GC5TS2METLE6BBQQGYRMWPTMVH3RMRKQJA2RXKQFCCTCJB2JXMBA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.049970457 (21h51m32.943679076s) [noise=true , address=81.171.14.70:4042 , piece_id=PFKTRW3DI6O7EQ4LSDBQM5YQDJ6JQLXCNZTBCNMTYD3SKQWVGGLQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.057550509 (21h51m32.951259118s) [noise=true , address=81.171.14.70:4040 , piece_id=4ZRIS4UHZC6SMLPEIXS6KU6UAKYVIGI6ZMJFRFJ25PY4RB32WVEQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.093832752 (21h51m32.987541361s) [noise=true , address=81.171.14.70:4026 , piece_id=YZUBTL2WRUNDUPW376BG676VAWOU2GMIMQITAKTWIKJ3HKZXNLZA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.370239906 (21h51m33.263948515s) [noise=true , address=185.167.204.206:4067 , piece_id=BSRJBOCNWZIYTEP6HY3NOGZORQTPCWKN43E6HPVC37LY5I5CJYNA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.384125184 (21h51m33.277833793s) [noise=true , address=185.167.204.206:4062 , piece_id=3Z6VWEXXIA3ISY73C5JV5S5ROLCBR6DSQWZVI6Q3QSSYLRBNM6LQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.391293162 (21h51m33.285001771s) [noise=true , address=185.167.204.206:4064 , piece_id=2PDTBJHQW5AHYD2SOM6Q5D6BYGQJCWSWQNLRXCNRPWYBHKBS6B3Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.391398192 (21h51m33.285106801s) [noise=true , address=185.167.204.206:4040 , piece_id=XCGSNFY225UO7MYFSYWQQLVMP73RIUVII7ZYTTGWZ3TQU2DPMOTA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.392101912 (21h51m33.285810521s) [noise=true , address=185.167.204.206:4061 , piece_id=EN3JISM5WLYJWU3UG3BFTNCBOP7MTIFJBRYU4KTKDFVSWCEJRVJA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.400384883 (21h51m33.294093492s) [noise=true , address=185.167.204.206:4039 , piece_id=BJWZY2S2EZSQO6CJM25WV4RIYTMS3P4CG5RHC44JXDMUGA3Z3JEQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.405248333 (21h51m33.298956942s) [noise=true , address=185.167.204.206:4050 , piece_id=E5FLLMTDTPGGIAKZDIIWPBOFWQL7YXZSUA7Z5YSN2KWJOASDJF6Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.414198665 (21h51m33.307907274s) [noise=true , address=185.167.204.205:4006 , piece_id=XCZ2NHQZKDA7AADK3JP26Y3WOZMXQ5IJV5THD5V2P5TX3F7WAFOQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.423293562 (21h51m33.317002171s) [noise=true , address=185.167.204.206:4045 , piece_id=FHIBGSVU2OZBBQETRZNJK6QW7DTCV7ITCEMUGEYQAN6ZYMQQYTOQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.423504314 (21h51m33.317212923s) [noise=true , address=185.167.204.205:4018 , piece_id=3J3LKBXHO5E2IYUVRO7NG7BSYXYBQEFRHV23OQHXHFBYBIYOBQ2A, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.435923832 (21h51m33.329632461s) [noise=true , address=185.167.204.205:4019 , piece_id=NGOWYQ3TZHPLUJ73WVIC3YYL65EYJVBKOPUWCI4DLEGRPQIECBBA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.436584970 (21h51m33.330293579s) [noise=true , address=185.167.204.205:4026 , piece_id=SVVM4Q6OCLSV7TWVURRBYN765J44OTG7AFOQS3RZX7GKFMDJ6ZJQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.491250847 (21h51m33.384959456s) [noise=true , address=81.171.14.70:4031 , piece_id=EF67IQMYFBPUP6UZXDGSQLHR6X4G6OSCKEH4DYWCKI4UIHO4UVDA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.518740798 (21h51m33.412449407s) [noise=true , address=81.171.14.70:4030 , piece_id=SLIXLC6YJ5PLS4MCXJIACYV5MIFOQSBARATYC3WAEZEZSVWICMCQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.526006844 (21h51m33.419715453s) [noise=true , address=81.171.14.70:4025 , piece_id=DCILR24FF45JKLU36SHTMAJLY3NTFNGT4UWNFJXATADZJY2W3TBA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.564445552 (21h51m33.458154161s) [noise=true , address=81.171.14.70:4027 , piece_id=7YO3ZAQFFZOSWSG2JH3KOHWDEFQJLOCCQ5FUGSIT7UUREHROOGYA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.587122821 (21h51m33.48083146s) [noise=true , address=81.171.14.70:4038 , piece_id=VAUCLB55KGVJVL4BSWY4ZY2747BDYFXI2RK3JMPDO4ZOCVOT43EA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.679729699 (21h51m33.573438309s) [noise=true , address=89.149.201.147:4012 , piece_id=SPAZ2DY75JXHC2PWPRI3QG7HYCGHM4GVXN5DY5ZWYDTM6637EV7Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.775377568 (21h51m33.669086177s) [noise=true , address=89.149.201.147:4010 , piece_id=WBP6VM7X2ZISV4K4SE6LL3ZSPYHJQK24IOGILVH6S2ZLWRRJVN7Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:37.160795473 (21h51m34.054504082s) [noise=true , address=89.149.201.147:4023 , piece_id=LQXOUV6UVIHCQTXPVMC7NYINBLEZWF4H7W6BSOWGK6BFG5ARQ3VQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:37.251044932 (21h51m34.144753541s) [noise=true , address=89.149.201.147:4007 , piece_id=J4W7LHO6YZCI3KPZ3F3DN6P4EAV5RIMRQ2X7ZOKLAAQVNGD7HVMA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:48:34.164143586 (21h52m31.057852195s) [noise=true , address=89.149.201.147:4011 , piece_id=TFRZQWAVN2NUSEWM4L33PKZKUYWQYCV3KCRHTKVTABLDZ7GSVVSQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: ecclient: upload failed (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg, address:89.149.201.147:4011): protocol: read tcp 188.166.225.170:48732->89.149.201.147:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 16 13:48:34.164198381 (21h52m31.05790699s) [noise=true , address=89.149.201.147:4020 , piece_id=CQAAHSLJFZE4GHOKB5UO7A2ANJ6N4HHFZDQ33WV25JF76L7JMMDQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: expected piece hash; context deadline exceeded; EOF
Jan 16 13:48:34.164358165 (21h52m31.058066774s) [noise=true , address=89.149.201.147:4014 , piece_id=BNGVISOX55CNKPIHIJJCXFNIMGW5FCE6GWEY4QKLIAWMVYMRAYFA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: context deadline exceeded; context deadline exceeded
Jan 16 13:48:34.164382872 (21h52m31.058091481s) [seg_pos={0 0}, upload=19196]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil>
Jan 16 13:48:34.330174768 (21h52m31.223883387s) [upload=19196]: Done uploading segments...
Jan 16 13:48:34.330206869 (21h52m31.223915478s) [upload=19196]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25)
Duration: 5.273555385s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
54 out of 60 is failed (threshold=3) [..XXX.X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXX] duration:
30.001302636s trace:
https://jaeger.dev.storj.io/trace/3a03d81982cc525e
FAIL (2026-01-16 14:06:57 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001323377s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
20.482790226s trace:
https://jaeger.dev.storj.io/trace/a3cda5286a23a90
FAIL (2026-01-16 14:06:56 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 20.48281349s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
9 out of 60 is failed (threshold=3) [...........X.............X..X..X...X..........X.X.X.....X...] duration:
12.161226347s trace:
https://jaeger.dev.storj.io/trace/346659e7312f1ccc
FAIL (2026-01-16 14:06:54 UTC)
orbiter warning: context deadline exceeded
Duration: 12.161255021s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.00114355s trace:
https://jaeger.dev.storj.io/trace/3d078305d42d1385
FAIL (2026-01-16 14:06:49 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001163107s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
60 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001477608s trace:
https://jaeger.dev.storj.io/trace/7103a15ccf6df85c
FAIL (2026-01-16 14:06:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001497105s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
9 out of 60 is failed (threshold=3) [............X.............X..X..X...X..........X.X.X.....X..] duration:
19.179888427s trace:
https://jaeger.dev.storj.io/trace/346659e7312f1ccc
FAIL (2026-01-16 14:06:28 UTC)
orbiter warning: context deadline exceeded
Duration: 19.180182687s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
21.032105488s trace:
https://jaeger.dev.storj.io/trace/a3cda5286a23a90
FAIL (2026-01-16 14:06:22 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 21.032133592s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
7 out of 60 is failed (threshold=3) [...X.............X........X..X.XX.....X.....................] duration:
4.383313133s trace:
https://jaeger.dev.storj.io/trace/18ea158581539e05
FAIL (2026-01-16 14:06:14 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 25)
======LOGS======
: Uploading piece...
Jan 16 13:47:34.817121271 (21h51m31.71082988s) [noise=true , address=81.171.14.70:4030 , piece_id=SLIXLC6YJ5PLS4MCXJIACYV5MIFOQSBARATYC3WAEZEZSVWICMCQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817237642 (21h51m31.710946251s) [noise=true , address=81.171.14.70:4040 , piece_id=4ZRIS4UHZC6SMLPEIXS6KU6UAKYVIGI6ZMJFRFJ25PY4RB32WVEQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817351119 (21h51m31.711059728s) [noise=true , address=89.149.201.147:4011 , piece_id=TFRZQWAVN2NUSEWM4L33PKZKUYWQYCV3KCRHTKVTABLDZ7GSVVSQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817710643 (21h51m31.711419282s) [noise=true , address=185.167.204.205:4018 , piece_id=3J3LKBXHO5E2IYUVRO7NG7BSYXYBQEFRHV23OQHXHFBYBIYOBQ2A, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.817769405 (21h51m31.711478014s) [noise=true , address=185.167.204.206:4067 , piece_id=BSRJBOCNWZIYTEP6HY3NOGZORQTPCWKN43E6HPVC37LY5I5CJYNA, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818062553 (21h51m31.711771162s) [noise=true , address=185.167.204.206:4062 , piece_id=3Z6VWEXXIA3ISY73C5JV5S5ROLCBR6DSQWZVI6Q3QSSYLRBNM6LQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818137756 (21h51m31.711846365s) [noise=true , address=89.149.201.147:4012 , piece_id=SPAZ2DY75JXHC2PWPRI3QG7HYCGHM4GVXN5DY5ZWYDTM6637EV7Q, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818415194 (21h51m31.712123793s) [noise=true , address=185.167.204.205:4006 , piece_id=XCZ2NHQZKDA7AADK3JP26Y3WOZMXQ5IJV5THD5V2P5TX3F7WAFOQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.818739812 (21h51m31.712448421s) [noise=true , address=89.149.201.147:4023 , piece_id=LQXOUV6UVIHCQTXPVMC7NYINBLEZWF4H7W6BSOWGK6BFG5ARQ3VQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819043791 (21h51m31.7127524s) [noise=true , address=185.167.204.205:4026 , piece_id=SVVM4Q6OCLSV7TWVURRBYN765J44OTG7AFOQS3RZX7GKFMDJ6ZJQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819337630 (21h51m31.713046239s) [noise=true , address=185.167.204.206:4050 , piece_id=E5FLLMTDTPGGIAKZDIIWPBOFWQL7YXZSUA7Z5YSN2KWJOASDJF6Q, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819543422 (21h51m31.713252031s) [noise=true , address=81.171.14.70:4042 , piece_id=PFKTRW3DI6O7EQ4LSDBQM5YQDJ6JQLXCNZTBCNMTYD3SKQWVGGLQ, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:34.819810982 (21h51m31.713519591s) [noise=true , address=81.171.14.70:4027 , piece_id=7YO3ZAQFFZOSWSG2JH3KOHWDEFQJLOCCQ5FUGSIT7UUREHROOGYA, seg_pos={0 0}, upload=19196]: Uploading piece...
Jan 16 13:47:35.646384760 (21h51m32.540093369s) [upload=19196]: Next returned nil segment
Jan 16 13:47:35.646399467 (21h51m32.540108076s) [upload=19196]: Waiting for error group managing segments...
Jan 16 13:47:36.044357389 (21h51m32.938065998s) [noise=true , address=81.171.14.70:4045 , piece_id=GC5TS2METLE6BBQQGYRMWPTMVH3RMRKQJA2RXKQFCCTCJB2JXMBA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.049970457 (21h51m32.943679076s) [noise=true , address=81.171.14.70:4042 , piece_id=PFKTRW3DI6O7EQ4LSDBQM5YQDJ6JQLXCNZTBCNMTYD3SKQWVGGLQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.057550509 (21h51m32.951259118s) [noise=true , address=81.171.14.70:4040 , piece_id=4ZRIS4UHZC6SMLPEIXS6KU6UAKYVIGI6ZMJFRFJ25PY4RB32WVEQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.093832752 (21h51m32.987541361s) [noise=true , address=81.171.14.70:4026 , piece_id=YZUBTL2WRUNDUPW376BG676VAWOU2GMIMQITAKTWIKJ3HKZXNLZA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.370239906 (21h51m33.263948515s) [noise=true , address=185.167.204.206:4067 , piece_id=BSRJBOCNWZIYTEP6HY3NOGZORQTPCWKN43E6HPVC37LY5I5CJYNA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.384125184 (21h51m33.277833793s) [noise=true , address=185.167.204.206:4062 , piece_id=3Z6VWEXXIA3ISY73C5JV5S5ROLCBR6DSQWZVI6Q3QSSYLRBNM6LQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.391293162 (21h51m33.285001771s) [noise=true , address=185.167.204.206:4064 , piece_id=2PDTBJHQW5AHYD2SOM6Q5D6BYGQJCWSWQNLRXCNRPWYBHKBS6B3Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.391398192 (21h51m33.285106801s) [noise=true , address=185.167.204.206:4040 , piece_id=XCGSNFY225UO7MYFSYWQQLVMP73RIUVII7ZYTTGWZ3TQU2DPMOTA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.392101912 (21h51m33.285810521s) [noise=true , address=185.167.204.206:4061 , piece_id=EN3JISM5WLYJWU3UG3BFTNCBOP7MTIFJBRYU4KTKDFVSWCEJRVJA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.400384883 (21h51m33.294093492s) [noise=true , address=185.167.204.206:4039 , piece_id=BJWZY2S2EZSQO6CJM25WV4RIYTMS3P4CG5RHC44JXDMUGA3Z3JEQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.405248333 (21h51m33.298956942s) [noise=true , address=185.167.204.206:4050 , piece_id=E5FLLMTDTPGGIAKZDIIWPBOFWQL7YXZSUA7Z5YSN2KWJOASDJF6Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.414198665 (21h51m33.307907274s) [noise=true , address=185.167.204.205:4006 , piece_id=XCZ2NHQZKDA7AADK3JP26Y3WOZMXQ5IJV5THD5V2P5TX3F7WAFOQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.423293562 (21h51m33.317002171s) [noise=true , address=185.167.204.206:4045 , piece_id=FHIBGSVU2OZBBQETRZNJK6QW7DTCV7ITCEMUGEYQAN6ZYMQQYTOQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.423504314 (21h51m33.317212923s) [noise=true , address=185.167.204.205:4018 , piece_id=3J3LKBXHO5E2IYUVRO7NG7BSYXYBQEFRHV23OQHXHFBYBIYOBQ2A, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.435923832 (21h51m33.329632461s) [noise=true , address=185.167.204.205:4019 , piece_id=NGOWYQ3TZHPLUJ73WVIC3YYL65EYJVBKOPUWCI4DLEGRPQIECBBA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.436584970 (21h51m33.330293579s) [noise=true , address=185.167.204.205:4026 , piece_id=SVVM4Q6OCLSV7TWVURRBYN765J44OTG7AFOQS3RZX7GKFMDJ6ZJQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.491250847 (21h51m33.384959456s) [noise=true , address=81.171.14.70:4031 , piece_id=EF67IQMYFBPUP6UZXDGSQLHR6X4G6OSCKEH4DYWCKI4UIHO4UVDA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.518740798 (21h51m33.412449407s) [noise=true , address=81.171.14.70:4030 , piece_id=SLIXLC6YJ5PLS4MCXJIACYV5MIFOQSBARATYC3WAEZEZSVWICMCQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.526006844 (21h51m33.419715453s) [noise=true , address=81.171.14.70:4025 , piece_id=DCILR24FF45JKLU36SHTMAJLY3NTFNGT4UWNFJXATADZJY2W3TBA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.564445552 (21h51m33.458154161s) [noise=true , address=81.171.14.70:4027 , piece_id=7YO3ZAQFFZOSWSG2JH3KOHWDEFQJLOCCQ5FUGSIT7UUREHROOGYA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.587122821 (21h51m33.48083146s) [noise=true , address=81.171.14.70:4038 , piece_id=VAUCLB55KGVJVL4BSWY4ZY2747BDYFXI2RK3JMPDO4ZOCVOT43EA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.679729699 (21h51m33.573438309s) [noise=true , address=89.149.201.147:4012 , piece_id=SPAZ2DY75JXHC2PWPRI3QG7HYCGHM4GVXN5DY5ZWYDTM6637EV7Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:36.775377568 (21h51m33.669086177s) [noise=true , address=89.149.201.147:4010 , piece_id=WBP6VM7X2ZISV4K4SE6LL3ZSPYHJQK24IOGILVH6S2ZLWRRJVN7Q, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:37.160795473 (21h51m34.054504082s) [noise=true , address=89.149.201.147:4023 , piece_id=LQXOUV6UVIHCQTXPVMC7NYINBLEZWF4H7W6BSOWGK6BFG5ARQ3VQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:47:37.251044932 (21h51m34.144753541s) [noise=true , address=89.149.201.147:4007 , piece_id=J4W7LHO6YZCI3KPZ3F3DN6P4EAV5RIMRQ2X7ZOKLAAQVNGD7HVMA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: <nil>
Jan 16 13:48:34.164143586 (21h52m31.057852195s) [noise=true , address=89.149.201.147:4011 , piece_id=TFRZQWAVN2NUSEWM4L33PKZKUYWQYCV3KCRHTKVTABLDZ7GSVVSQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: ecclient: upload failed (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg, address:89.149.201.147:4011): protocol: read tcp 188.166.225.170:48732->89.149.201.147:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 16 13:48:34.164198381 (21h52m31.05790699s) [noise=true , address=89.149.201.147:4020 , piece_id=CQAAHSLJFZE4GHOKB5UO7A2ANJ6N4HHFZDQ33WV25JF76L7JMMDQ, seg_pos={0 0}, upload=19196]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: expected piece hash; context deadline exceeded; EOF
Jan 16 13:48:34.164358165 (21h52m31.058066774s) [noise=true , address=89.149.201.147:4014 , piece_id=BNGVISOX55CNKPIHIJJCXFNIMGW5FCE6GWEY4QKLIAWMVYMRAYFA, seg_pos={0 0}, upload=19196]: Done uploading piece. err: ecclient: upload failed (node:1wgtphHKaTejC1qrJ98sDysBwtj2Zeqw2c5UFk1HMqfY8Buvr, address:89.149.201.147:4014): protocol: context deadline exceeded; context deadline exceeded
Jan 16 13:48:34.164382872 (21h52m31.058091481s) [seg_pos={0 0}, upload=19196]: Done waiting for segment. successful: 25 optimal: 26 errs: <nil>
Jan 16 13:48:34.330174768 (21h52m31.223883387s) [upload=19196]: Done uploading segments...
Jan 16 13:48:34.330206869 (21h52m31.223915478s) [upload=19196]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 25)
Duration: 4.383347146s
core
uplink
eu-select-1
eu1-placement-15
|