Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 428277 reports (Page 292 of 8566)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X..XX................................................] duration: 5.212893558s trace: https://jaeger.dev.storj.io/trace/233c74a6b9db5cf8 FAIL (2026-01-08 06:44:26 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 53) Duration: 5.212927583s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [.X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.002015822s trace: https://jaeger.dev.storj.io/trace/4b5490d679920ebf FAIL (2026-01-08 06:44:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002042133s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 8 out of 60 is failed (threshold=3) [............XXXXXXXX........................................] duration: 7.719598481s trace: https://jaeger.dev.storj.io/trace/635cbd49e9d724cb FAIL (2026-01-08 06:44:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 21) ======LOGS====== _id=TM5C7SQQA5IAVFWRG2GV2WYRB2IRLWDLVB3MFWK7XILSBDUDIAVQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.977864602 (806h43m10.848947647s) [noise=true , address=185.167.204.206:4059 , piece_id=FKQBFAG3DNQ5OI5DJ7GAZINSWVSVDZED56SJVHBY2ULJO5HUAGNQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.978752623 (806h43m10.849835658s) [noise=true , address=185.167.204.206:4064 , piece_id=3FMVMJWMWGAHNVQX2AIBVII5Z6XTPJUCYABKNXJ4NWRC2LQ6I3FA, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.979487392 (806h43m10.850570427s) [noise=true , address=89.149.201.147:4020 , piece_id=U3G7AY7GMBA527UQXXY6DOLNKJBUR4CZD7T3MOJSQNFFGYJXKR3A, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.980303676 (806h43m10.851386721s) [noise=true , address=185.167.204.206:4061 , piece_id=LJVZ2GBV4S5T6XXBYDEELH6BKTBY2TTOBBXWNNNH2VSE2W5RI5YA, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.980907016 (806h43m10.851990051s) [noise=true , address=89.149.201.147:4011 , piece_id=PXK522IVCJHJDLWXUIP3PQF7PJXVPXLPGM6HE43CY6G2VSLOAWPQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.981714563 (806h43m10.852797608s) [noise=true , address=81.171.14.70:4043 , piece_id=AM727IICNSFNSVCQ3IZ55CAZK6YH3JUFBOJAUHMMYRJ6NEZZREDQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.982411280 (806h43m10.853494325s) [noise=true , address=89.149.201.147:4004 , piece_id=BRZPEKISSTSV7IPNBW7ZJJIIGOTR4SQDJDMQZ3OXOSUEQJZB5CMQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.983960189 (806h43m10.855043224s) [noise=true , address=81.171.14.70:4048 , piece_id=HLP5YVTQMQXLGROPTZJ4RPKFYCCBB3W5ZPHASDVMTJ6ZCTSGMDKQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:11.816868763 (806h43m11.687951808s) [upload=705619]: Next returned nil segment Jan 8 06:12:11.816884092 (806h43m11.687967137s) [upload=705619]: Waiting for error group managing segments... Jan 8 06:12:12.207503101 (806h43m12.078586136s) [noise=true , address=185.167.204.205:4014 , piece_id=42DLCGFRUY5XHUEJPKO6E6BKJSWTUC5IDYRSAE3OXFFVCYA2E67A, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.207636776 (806h43m12.078719821s) [noise=true , address=185.167.204.205:4011 , piece_id=36G2RTRXP6WEHPMQKMEQ3334GYKN6RFC7VYMW6BSCLBQ37OT43YQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.208187303 (806h43m12.079270338s) [noise=true , address=185.167.204.206:4041 , piece_id=CDCLTZOQNX4QLRZ5JT7ZIBQKKBRWVX3SWJSKC46EAIMPSV2SYONA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.208277846 (806h43m12.079360881s) [noise=true , address=185.167.204.205:4003 , piece_id=EGXKQAKMGKB3SDIEFFEVFE6XYGHZ4ZR6A5B6MFDB7XFY7KID4QYA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.211357060 (806h43m12.082440095s) [noise=true , address=185.167.204.206:4062 , piece_id=MJLGFBLTJEG7UHJMFLI72MHHH6FWFLP3O4VNTUAS5Z4KCH27EHOQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.215506782 (806h43m12.086589817s) [noise=true , address=185.167.204.206:4068 , piece_id=L6DRFJFP5OFG2UEGHUGNCRWVGJ7XQFGJJUXNT7WGSEJXS5TMKNRQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.221151259 (806h43m12.092234294s) [noise=true , address=185.167.204.206:4059 , piece_id=FKQBFAG3DNQ5OI5DJ7GAZINSWVSVDZED56SJVHBY2ULJO5HUAGNQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.247553292 (806h43m12.118636337s) [noise=true , address=185.167.204.206:4064 , piece_id=3FMVMJWMWGAHNVQX2AIBVII5Z6XTPJUCYABKNXJ4NWRC2LQ6I3FA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.268061592 (806h43m12.139144637s) [noise=true , address=185.167.204.206:4061 , piece_id=LJVZ2GBV4S5T6XXBYDEELH6BKTBY2TTOBBXWNNNH2VSE2W5RI5YA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.268319703 (806h43m12.139402739s) [noise=true , address=185.167.204.205:4017 , piece_id=ACCHFJNRCMWJWXPZJDXALXSUVPLETHOSDAGKPKDAKRMWJ2JI6RMQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.270984638 (806h43m12.142067684s) [noise=true , address=185.167.204.206:4037 , piece_id=HAKQW2RQFJJGBIG26EV4B5TJJBZRM2YMXVU5PZQIKNA5YYEAZVHA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.272396927 (806h43m12.143479962s) [noise=true , address=185.167.204.205:4034 , piece_id=EMHALXT4VKLMJNTQV4XMRGLVOPBRF7O45YKF5Q3RYUBF6DGHPTJQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:13.719906169 (806h43m13.590989215s) [noise=true , address=89.149.201.147:4007 , piece_id=XDKNFX56PLHN6AXAKPM3LSRUHAFHIOZIJF2CAFXOMDLUH7URSS3Q, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:15.774082352 (806h43m15.645165397s) [noise=true , address=89.149.201.147:4005 , piece_id=3WEJWRZJOWY6I7ZNNWHK5QYTF5MULZNKHWVZMBEJJPFNB6WU4UCA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:16.530239032 (806h43m16.401322067s) [noise=true , address=89.149.201.147:4001 , piece_id=LWDR4KCC5QHQDLFVZYVTYJH26GRABV2JFC3525OVIORRG2SRHQMA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:16.928628457 (806h43m16.799711492s) [noise=true , address=81.171.14.70:4041 , piece_id=J3IG4LCWN7UM2NCQAY4FKTPNYH5NRW7TDB6ZGMNPQEI7HY5CZGRQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:17.995225112 (806h43m17.866308157s) [noise=true , address=89.149.201.147:4011 , piece_id=PXK522IVCJHJDLWXUIP3PQF7PJXVPXLPGM6HE43CY6G2VSLOAWPQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:18.045551680 (806h43m17.916634725s) [noise=true , address=81.171.14.70:4027 , piece_id=5P2A7PSIMWZC2I6SAGUEUEJDDW67BD7AKQ34Z3XZHFPJGSHLJLVQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:19.316134332 (806h43m19.187217367s) [noise=true , address=89.149.201.147:4004 , piece_id=BRZPEKISSTSV7IPNBW7ZJJIIGOTR4SQDJDMQZ3OXOSUEQJZB5CMQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:19.607969825 (806h43m19.47905288s) [noise=true , address=81.171.14.70:4035 , piece_id=HAUGVFLDMZCSFVPSOYLS4PBCVHMMMOTPD4ODSDCS3P4NSTX6KPJQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:13:00.649061845 (806h44m0.52014488s) [noise=true , address=89.149.201.147:4010 , piece_id=TM5C7SQQA5IAVFWRG2GV2WYRB2IRLWDLVB3MFWK7XILSBDUDIAVQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:13:10.291882458 (806h44m10.162965513s) [noise=true , address=89.149.201.147:4015 , piece_id=GLHXEDCIEKGM2SPTCNCTJYS2GIEUMIT5LQXCHMDTVNNIY5KLASVQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K, address:89.149.201.147:4015): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.291987709 (806h44m10.163070774s) [noise=true , address=81.171.14.70:4034 , piece_id=VGWIA5ZBLFNA33QXVLF54XTCOCAHSU6TVV3O52DTZ2L3ROIPHNAA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12jcAHt22bETT27CCXJ8ZaaYxxoKBonBYsFcoq6A34DVVjvhQAN, address:81.171.14.70:4034): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.292023307 (806h44m10.163106342s) [noise=true , address=81.171.14.70:4038 , piece_id=QOQ3IYLTGQ27HOLQKQZOOILZXWAPR6PEHLQTHLGLBLXLLQDBA5RA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB, address:81.171.14.70:4038): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.292120161 (806h44m10.163203216s) [noise=true , address=89.149.201.147:4020 , piece_id=U3G7AY7GMBA527UQXXY6DOLNKJBUR4CZD7T3MOJSQNFFGYJXKR3A, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.292271189 (806h44m10.163354245s) [noise=true , address=81.171.14.70:4048 , piece_id=HLP5YVTQMQXLGROPTZJ4RPKFYCCBB3W5ZPHASDVMTJ6ZCTSGMDKQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: read tcp 188.166.225.170:40684->81.171.14.70:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 8 06:13:10.292450230 (806h44m10.163533285s) [noise=true , address=81.171.14.70:4043 , piece_id=AM727IICNSFNSVCQ3IZ55CAZK6YH3JUFBOJAUHMMYRJ6NEZZREDQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko, address:81.171.14.70:4043): protocol: read tcp 188.166.225.170:38496->81.171.14.70:4043: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 8 06:13:10.292509182 (806h44m10.163592237s) [noise=true , address=81.171.14.70:4029 , piece_id=6UO7K2VEO7HMZPJM4S34Z6Z3UWKFMLD3BLATMSIRL4WDIXKVTZCA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:60086->81.171.14.70:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 8 06:13:10.292533999 (806h44m10.163617034s) [seg_pos={0 0}, upload=705619]: Done waiting for segment. successful: 21 optimal: 26 errs: <nil> Jan 8 06:13:10.462867035 (806h44m10.33395007s) [upload=705619]: Done uploading segments... Jan 8 06:13:10.462901039 (806h44m10.333984085s) [upload=705619]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 21) Duration: 7.719628028s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.226216046s trace: https://jaeger.dev.storj.io/trace/3738a06b2bfcba74 FAIL (2026-01-08 06:44:12 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12hkSjuH8cdS4byda1bykpe2QJz42HXaK1uRtkTbCQdynovAKSz, Piece ID: 4GTLFPYP5LTIZPMRSOVJNHNBOMLFZTIF7CD5QHSAWAQT2DQYCAEQ) context deadline exceeded; piecestore: (Node ID: 1bWLCSW1tixYzHThBQsUPvXtTpnf4egF8uGHFUXDQkCeE4BwBU, Piece ID: RX5TQUH56ZXKJADRRFT7DX6J6VV32WB3EFRCM456NZ2TD2WHKM3A) context deadline exceeded; piecestore: (Node ID: 1XxiRqM4j35HGGzWPnmrHHfbWS8jryWrW5aY66SgDuDQKHYopv, Piece ID: GAOHJL4R553KJFKTWIRCGC3NHHM4TBFNH6GJYFREM23JQQS645TQ) context deadline exceeded; piecestore: (Node ID: 15LfiPEGogvaJkFN67RXm8egaXMGxsc9XmPo26GFwT5TZKmg75, Piece ID: HD6GDZKL6A43NRG367DI674KCKPZAN4PP6S5ZRHHYR33EPAQ4KRQ) context deadline exceeded; piecestore: (Node ID: 1m2RK4RXMBjqBpWj8Xof42VU2fyPNygiY1WR3gcsVRxd6jKe4E, Piece ID: SCDRYTITO42A42357QKLW7RUWW6VKZLZLYQ3H2NIPXAKYGOIC5HQ) context deadline exceeded; piecestore: (Node ID: 1vtdYMUQ1aEzqZeSWkWD9RceYHunHdt9QwAzy7CwyYruPtSNRw, Piece ID: SYC3VAYM2YKT5ZM2VAQQC7663JAM47LYYCU62ZFRVH5UJDTTXMRQ) context deadline exceeded; piecestore: (Node ID: 1dstfvWPzYyG7oHnsByWcpgv6C4NVw9RPt8MdxtZ634FaLrSt4, Piece ID: D6G2XNNG5QPU2VSWNAP6XPRA3LM5GREUKMQG3KEWTKXFBNSEPIZA) context deadline exceeded; piecestore: (Node ID: 12Bf3H3JPRVT3GKxieKhLpk7ZfXc3FrdtweLJKRpFo6yNTomKDu, Piece ID: UD3L4QJO2DEW7V3SJ7HHPO5P6VWX4MURJKKEQ5KIQUHKAUZOZVTQ) context deadline exceeded; piecestore: (Node ID: 1ZtoenGAVBrgVoJkmkyxDqire258rdYcqC3avm8xAPuBGiD1bU, Piece ID: 4GAT5EPHUEH7UO3IHA5FU23RMY7NGYLJU2CBCX3RS2BCAPIDP7VA) context deadline exceeded; piecestore: (Node ID: 12vhu4j9fCJhbUW9rxZshXEbh9gbG1j1sPzgPhDdiD7ecVTQaHz, Piece ID: 6N3EAXOD5X75DQR6W4AHVJ54XQ7BGOF2BRN32IDBHLLYOOYI3ILQ) context deadline exceeded; piecestore: (Node ID: 1kLgnR5vmDgUuaf1aWAUyDsXuHn8aEwVXEQiruQrkfKpXqTiQ1, Piece ID: DGJRCP2CNO7DMWSJ5XMYFWV6CF22SLA4LPCVWLTBRKCQ6NV7HS7Q) context deadline exceeded Duration: 6.226245962s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [XXX.X.....X..X.......XXX.XX.XX..X.XXX.XXXX.XXXXXXX..X.XXX.X.] duration: 18.593765024s trace: https://jaeger.dev.storj.io/trace/71999335d393d040 FAIL (2026-01-08 06:44:03 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 18.593795723s edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 3.478137448s trace: https://jaeger.dev.storj.io/trace/6173eb4c5ef6f546 FAIL (2026-01-08 06:43:55 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 53) Duration: 3.478178777s core uplink ap1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [.XXX.X.....X..X.......XXX.XX.XX..X.XXX.XXXX.XXXXXXX..X.XXX.X] duration: 10.831454183s trace: https://jaeger.dev.storj.io/trace/71999335d393d040 FAIL (2026-01-08 06:43:39 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 10.831476556s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [X.X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001413306s trace: https://jaeger.dev.storj.io/trace/e8ce2846c96d905 FAIL (2026-01-08 06:43:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001437201s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.................................X....X....X................] duration: 7.415538641s trace: https://jaeger.dev.storj.io/trace/23102568ffb9ad71 FAIL (2026-01-08 06:43:30 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 7.415568249s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [X........X..X...............X....XXX...XX.X......X..X....X..] duration: 19.509508023s trace: https://jaeger.dev.storj.io/trace/5b798fac87e6bbd1 FAIL (2026-01-08 06:43:28 UTC) orbiter warning: context deadline exceeded Duration: 19.509530545s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..........X......XXX..........XX......X...........X.........] duration: 6.061819127s trace: https://jaeger.dev.storj.io/trace/11a85404b66ec466 FAIL (2026-01-08 06:43:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: JMZE2QQYMZJLKRLN7DURUWAHXWYRPA4FRQ7H4ITTQUFDIAMKHM5Q) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: HVF4HNEVWBLOPEX6LMJNZZDYRZOLTT3NGFMIGMS46S3TVHDDLNDA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 3OF7GH4SQ2OXWT4AMSH6U542D4RF4T4UBNAKG47QHUGF3AA662VQ) context deadline exceeded ======LOGS====== Jan 8 06:38:10.969187554 (807h9m10.840270599s) [upload=705993]: Starting upload Jan 8 06:38:10.969275200 (807h9m10.840358245s) [upload=705993]: Done starting upload Jan 8 06:38:10.969338862 (807h9m10.840421907s) [upload=705993]: Uploading segments... Jan 8 06:38:10.969657759 (807h9m10.840740804s) [upload=705993]: Got next segment. Inline: false Jan 8 06:38:10.969705860 (807h9m10.840788896s) [upload=705993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 8 06:38:10.969712884 (807h9m10.840795929s) [upload=705993]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 8 06:38:11.214842718 (807h9m11.085925753s) [seg_pos={0 0}, upload=705993]: Begin upload segment... Jan 8 06:38:11.214885970 (807h9m11.085969005s) [seg_pos={0 0}, upload=705993]: Done begin upload segment. Jan 8 06:38:11.214913212 (807h9m11.085996247s) [noise=true , address=45.91.116.9:4262 , piece_id=HVF4HNEVWBLOPEX6LMJNZZDYRZOLTT3NGFMIGMS46S3TVHDDLNDA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.214935674 (807h9m11.086018709s) [noise=true , address=145.79.227.15:4875 , piece_id=66NSAERWEKNI5HJR22SKYZ3UT4ZGIVPHX4GNC7OCWNWUHHYOXQSA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215307472 (807h9m11.086390518s) [noise=true , address=145.79.227.49:4338 , piece_id=BA6QWKQJDAFRXCNQQTEPPTUDOUQRTOV6LJGKUI6PDG54IGIS44KQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215445766 (807h9m11.086528801s) [noise=true , address=145.79.227.47:4267 , piece_id=MAPONRBWD2MPQFXIGOXQ5T4QVUKE726LLXUBWA2WAZOOWSTALAJA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215800942 (807h9m11.086883977s) [noise=true , address=45.91.116.10:4082 , piece_id=H3UICIRBXOP5XWROFAS2BBDZSKLRHX6S3E4OTGXGNGUHVJHRLPKQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215815641 (807h9m11.086898686s) [noise=true , address=145.79.227.46:4602 , piece_id=4DLQBWUICSVM7CDVEDJQJRVHOLA62VDZV55ORHOFUXJKZXNODWRQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215868070 (807h9m11.086951115s) [noise=true , address=77.90.31.16:4736 , piece_id=3OF7GH4SQ2OXWT4AMSH6U542D4RF4T4UBNAKG47QHUGF3AA662VQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216134628 (807h9m11.087217663s) [noise=true , address=45.91.116.14:4191 , piece_id=JMZE2QQYMZJLKRLN7DURUWAHXWYRPA4FRQ7H4ITTQUFDIAMKHM5Q, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216232675 (807h9m11.08731572s) [noise=true , address=145.79.227.50:4375 , piece_id=UIYTXVHAKZ5JMBKMLLNIXNLSCGOGZCWFW6UOLDRLLXTV6G4H4SBA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216462912 (807h9m11.087545947s) [noise=true , address=45.91.116.5:4298 , piece_id=RUKJRTACXQ5XW6RIJ4ZP2DJN5GK6FISSBL73VTQ4QHSTI4KGKEGA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216686669 (807h9m11.087769734s) [noise=true , address=145.79.227.33:4871 , piece_id=OYZWFPL5IJB2CEPUWRK3ENTHYB23ZC65GJ7MRGW222JZRXV4Z4MA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216829160 (807h9m11.087912205s) [noise=true , address=45.91.116.6:4976 , piece_id=JUCN3Z7B66IVMMHFE7KPJUMMQCXIQ2I2LZHQURVGV3INEW6BNXGA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.217197411 (807h9m11.088280446s) [noise=true , address=145.79.227.30:4762 , piece_id=FCFXHYISPDWSMLCEN5D64VSGYN3GHCLMBCZCGB3U77OMMYJCBVFQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.217587214 (807h9m11.088670259s) [noise=true , address=145.79.227.28:4727 , piece_id=ELCOLD2FC5EZNIVQEIJUX4UXAPWMP2NGKUHYPYSYREH5Y4KUABBA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.217682535 (807h9m11.08876558s) [noise=true , address=77.90.31.13:4629 , piece_id=VAZ33WQ2GAX63OREDM2UMJHAYM2V3YHXY72DOK2QGKY42FT44XDQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.218057970 (807h9m11.089141015s) [noise=true , address=145.79.227.45:4230 , piece_id=TIY5UNNH7LQCR4M4TIBI55WDYAAOUY45GGBF5CYAFBETAWH6RXBA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:12.906151024 (807h9m12.777234069s) [upload=705993]: Next returned nil segment Jan 8 06:38:12.906163417 (807h9m12.777246462s) [upload=705993]: Waiting for error group managing segments... Jan 8 06:38:13.191636160 (807h9m13.062719195s) [noise=true , address=145.79.227.46:4602 , piece_id=4DLQBWUICSVM7CDVEDJQJRVHOLA62VDZV55ORHOFUXJKZXNODWRQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.198526130 (807h9m13.069609175s) [noise=true , address=145.79.227.49:4338 , piece_id=BA6QWKQJDAFRXCNQQTEPPTUDOUQRTOV6LJGKUI6PDG54IGIS44KQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.220694191 (807h9m13.091777236s) [noise=true , address=145.79.227.15:4875 , piece_id=66NSAERWEKNI5HJR22SKYZ3UT4ZGIVPHX4GNC7OCWNWUHHYOXQSA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.223448024 (807h9m13.09453107s) [noise=true , address=145.79.227.28:4727 , piece_id=ELCOLD2FC5EZNIVQEIJUX4UXAPWMP2NGKUHYPYSYREH5Y4KUABBA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.226749842 (807h9m13.097832877s) [noise=true , address=145.79.227.45:4230 , piece_id=TIY5UNNH7LQCR4M4TIBI55WDYAAOUY45GGBF5CYAFBETAWH6RXBA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.240985449 (807h9m13.112068484s) [noise=true , address=145.79.227.47:4267 , piece_id=MAPONRBWD2MPQFXIGOXQ5T4QVUKE726LLXUBWA2WAZOOWSTALAJA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.305163744 (807h9m13.176246779s) [noise=true , address=145.79.227.50:4375 , piece_id=UIYTXVHAKZ5JMBKMLLNIXNLSCGOGZCWFW6UOLDRLLXTV6G4H4SBA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.452825885 (807h9m13.32390893s) [noise=true , address=145.79.227.33:4871 , piece_id=OYZWFPL5IJB2CEPUWRK3ENTHYB23ZC65GJ7MRGW222JZRXV4Z4MA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.477222951 (807h9m13.348305996s) [noise=true , address=77.90.31.13:4629 , piece_id=VAZ33WQ2GAX63OREDM2UMJHAYM2V3YHXY72DOK2QGKY42FT44XDQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.487139634 (807h9m13.358222669s) [noise=true , address=77.90.31.16:4736 , piece_id=3OF7GH4SQ2OXWT4AMSH6U542D4RF4T4UBNAKG47QHUGF3AA662VQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.556463607 (807h9m13.427546652s) [noise=true , address=145.79.227.30:4762 , piece_id=FCFXHYISPDWSMLCEN5D64VSGYN3GHCLMBCZCGB3U77OMMYJCBVFQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.627164652 (807h9m13.498247697s) [noise=true , address=45.91.116.14:4191 , piece_id=JMZE2QQYMZJLKRLN7DURUWAHXWYRPA4FRQ7H4ITTQUFDIAMKHM5Q, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.640632367 (807h9m13.511715412s) [noise=true , address=45.91.116.9:4262 , piece_id=HVF4HNEVWBLOPEX6LMJNZZDYRZOLTT3NGFMIGMS46S3TVHDDLNDA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.640653266 (807h9m13.511736301s) [seg_pos={0 0}, upload=705993]: Segment reached optimal threshold of 13 pieces. Jan 8 06:38:13.640904686 (807h9m13.511987721s) [noise=true , address=45.91.116.6:4976 , piece_id=JUCN3Z7B66IVMMHFE7KPJUMMQCXIQ2I2LZHQURVGV3INEW6BNXGA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 8 06:38:13.640913031 (807h9m13.511996067s) [noise=true , address=45.91.116.6:4976 , piece_id=JUCN3Z7B66IVMMHFE7KPJUMMQCXIQ2I2LZHQURVGV3INEW6BNXGA, seg_pos={0 0}, upload=705993]: Long tail cancellation due to optimal threshold Jan 8 06:38:13.640957756 (807h9m13.512040801s) [noise=true , address=45.91.116.10:4082 , piece_id=H3UICIRBXOP5XWROFAS2BBDZSKLRHX6S3E4OTGXGNGUHVJHRLPKQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jan 8 06:38:13.640980901 (807h9m13.512063936s) [noise=true , address=45.91.116.10:4082 , piece_id=H3UICIRBXOP5XWROFAS2BBDZSKLRHX6S3E4OTGXGNGUHVJHRLPKQ, seg_pos={0 0}, upload=705993]: Long tail cancellation due to optimal threshold Jan 8 06:38:13.641034412 (807h9m13.512117447s) [noise=true , address=45.91.116.5:4298 , piece_id=RUKJRTACXQ5XW6RIJ4ZP2DJN5GK6FISSBL73VTQ4QHSTI4KGKEGA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jan 8 06:38:13.641040725 (807h9m13.51212377s) [noise=true , address=45.91.116.5:4298 , piece_id=RUKJRTACXQ5XW6RIJ4ZP2DJN5GK6FISSBL73VTQ4QHSTI4KGKEGA, seg_pos={0 0}, upload=705993]: Long tail cancellation due to optimal threshold Jan 8 06:38:13.641058478 (807h9m13.512141513s) [seg_pos={0 0}, upload=705993]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 8 06:38:13.641146847 (807h9m13.512229892s) [upload=705993]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 8 06:38:13.641152558 (807h9m13.512235593s) [upload=705993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 8 06:38:13.924716336 (807h9m13.795799371s) [upload=705993]: Done uploading segments... Jan 8 06:38:13.924735052 (807h9m13.795818097s) [upload=705993]: Upload finished. err: <nil> Duration: 6.061846649s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.181337037s trace: https://jaeger.dev.storj.io/trace/3738a06b2bfcba74 FAIL (2026-01-08 06:43:26 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12hkSjuH8cdS4byda1bykpe2QJz42HXaK1uRtkTbCQdynovAKSz, Piece ID: 4GTLFPYP5LTIZPMRSOVJNHNBOMLFZTIF7CD5QHSAWAQT2DQYCAEQ) context deadline exceeded; piecestore: (Node ID: 1bWLCSW1tixYzHThBQsUPvXtTpnf4egF8uGHFUXDQkCeE4BwBU, Piece ID: RX5TQUH56ZXKJADRRFT7DX6J6VV32WB3EFRCM456NZ2TD2WHKM3A) context deadline exceeded; piecestore: (Node ID: 1XxiRqM4j35HGGzWPnmrHHfbWS8jryWrW5aY66SgDuDQKHYopv, Piece ID: GAOHJL4R553KJFKTWIRCGC3NHHM4TBFNH6GJYFREM23JQQS645TQ) context deadline exceeded; piecestore: (Node ID: 15LfiPEGogvaJkFN67RXm8egaXMGxsc9XmPo26GFwT5TZKmg75, Piece ID: HD6GDZKL6A43NRG367DI674KCKPZAN4PP6S5ZRHHYR33EPAQ4KRQ) context deadline exceeded; piecestore: (Node ID: 1m2RK4RXMBjqBpWj8Xof42VU2fyPNygiY1WR3gcsVRxd6jKe4E, Piece ID: SCDRYTITO42A42357QKLW7RUWW6VKZLZLYQ3H2NIPXAKYGOIC5HQ) context deadline exceeded; piecestore: (Node ID: 1vtdYMUQ1aEzqZeSWkWD9RceYHunHdt9QwAzy7CwyYruPtSNRw, Piece ID: SYC3VAYM2YKT5ZM2VAQQC7663JAM47LYYCU62ZFRVH5UJDTTXMRQ) context deadline exceeded; piecestore: (Node ID: 1dstfvWPzYyG7oHnsByWcpgv6C4NVw9RPt8MdxtZ634FaLrSt4, Piece ID: D6G2XNNG5QPU2VSWNAP6XPRA3LM5GREUKMQG3KEWTKXFBNSEPIZA) context deadline exceeded; piecestore: (Node ID: 12Bf3H3JPRVT3GKxieKhLpk7ZfXc3FrdtweLJKRpFo6yNTomKDu, Piece ID: UD3L4QJO2DEW7V3SJ7HHPO5P6VWX4MURJKKEQ5KIQUHKAUZOZVTQ) context deadline exceeded; piecestore: (Node ID: 1ZtoenGAVBrgVoJkmkyxDqire258rdYcqC3avm8xAPuBGiD1bU, Piece ID: 4GAT5EPHUEH7UO3IHA5FU23RMY7NGYLJU2CBCX3RS2BCAPIDP7VA) context deadline exceeded; piecestore: (Node ID: 12vhu4j9fCJhbUW9rxZshXEbh9gbG1j1sPzgPhDdiD7ecVTQaHz, Piece ID: 6N3EAXOD5X75DQR6W4AHVJ54XQ7BGOF2BRN32IDBHLLYOOYI3ILQ) context deadline exceeded; piecestore: (Node ID: 1kLgnR5vmDgUuaf1aWAUyDsXuHn8aEwVXEQiruQrkfKpXqTiQ1, Piece ID: DGJRCP2CNO7DMWSJ5XMYFWV6CF22SLA4LPCVWLTBRKCQ6NV7HS7Q) context deadline exceeded Duration: 6.181364699s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X..XX...............................................] duration: 5.194884904s trace: https://jaeger.dev.storj.io/trace/233c74a6b9db5cf8 FAIL (2026-01-08 06:43:22 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 53) Duration: 5.194927023s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 8 out of 60 is failed (threshold=3) [.............XXXXXXXX.......................................] duration: 6.539936414s trace: https://jaeger.dev.storj.io/trace/635cbd49e9d724cb FAIL (2026-01-08 06:43:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 21) ======LOGS====== _id=TM5C7SQQA5IAVFWRG2GV2WYRB2IRLWDLVB3MFWK7XILSBDUDIAVQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.977864602 (806h43m10.848947647s) [noise=true , address=185.167.204.206:4059 , piece_id=FKQBFAG3DNQ5OI5DJ7GAZINSWVSVDZED56SJVHBY2ULJO5HUAGNQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.978752623 (806h43m10.849835658s) [noise=true , address=185.167.204.206:4064 , piece_id=3FMVMJWMWGAHNVQX2AIBVII5Z6XTPJUCYABKNXJ4NWRC2LQ6I3FA, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.979487392 (806h43m10.850570427s) [noise=true , address=89.149.201.147:4020 , piece_id=U3G7AY7GMBA527UQXXY6DOLNKJBUR4CZD7T3MOJSQNFFGYJXKR3A, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.980303676 (806h43m10.851386721s) [noise=true , address=185.167.204.206:4061 , piece_id=LJVZ2GBV4S5T6XXBYDEELH6BKTBY2TTOBBXWNNNH2VSE2W5RI5YA, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.980907016 (806h43m10.851990051s) [noise=true , address=89.149.201.147:4011 , piece_id=PXK522IVCJHJDLWXUIP3PQF7PJXVPXLPGM6HE43CY6G2VSLOAWPQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.981714563 (806h43m10.852797608s) [noise=true , address=81.171.14.70:4043 , piece_id=AM727IICNSFNSVCQ3IZ55CAZK6YH3JUFBOJAUHMMYRJ6NEZZREDQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.982411280 (806h43m10.853494325s) [noise=true , address=89.149.201.147:4004 , piece_id=BRZPEKISSTSV7IPNBW7ZJJIIGOTR4SQDJDMQZ3OXOSUEQJZB5CMQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.983960189 (806h43m10.855043224s) [noise=true , address=81.171.14.70:4048 , piece_id=HLP5YVTQMQXLGROPTZJ4RPKFYCCBB3W5ZPHASDVMTJ6ZCTSGMDKQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:11.816868763 (806h43m11.687951808s) [upload=705619]: Next returned nil segment Jan 8 06:12:11.816884092 (806h43m11.687967137s) [upload=705619]: Waiting for error group managing segments... Jan 8 06:12:12.207503101 (806h43m12.078586136s) [noise=true , address=185.167.204.205:4014 , piece_id=42DLCGFRUY5XHUEJPKO6E6BKJSWTUC5IDYRSAE3OXFFVCYA2E67A, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.207636776 (806h43m12.078719821s) [noise=true , address=185.167.204.205:4011 , piece_id=36G2RTRXP6WEHPMQKMEQ3334GYKN6RFC7VYMW6BSCLBQ37OT43YQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.208187303 (806h43m12.079270338s) [noise=true , address=185.167.204.206:4041 , piece_id=CDCLTZOQNX4QLRZ5JT7ZIBQKKBRWVX3SWJSKC46EAIMPSV2SYONA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.208277846 (806h43m12.079360881s) [noise=true , address=185.167.204.205:4003 , piece_id=EGXKQAKMGKB3SDIEFFEVFE6XYGHZ4ZR6A5B6MFDB7XFY7KID4QYA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.211357060 (806h43m12.082440095s) [noise=true , address=185.167.204.206:4062 , piece_id=MJLGFBLTJEG7UHJMFLI72MHHH6FWFLP3O4VNTUAS5Z4KCH27EHOQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.215506782 (806h43m12.086589817s) [noise=true , address=185.167.204.206:4068 , piece_id=L6DRFJFP5OFG2UEGHUGNCRWVGJ7XQFGJJUXNT7WGSEJXS5TMKNRQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.221151259 (806h43m12.092234294s) [noise=true , address=185.167.204.206:4059 , piece_id=FKQBFAG3DNQ5OI5DJ7GAZINSWVSVDZED56SJVHBY2ULJO5HUAGNQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.247553292 (806h43m12.118636337s) [noise=true , address=185.167.204.206:4064 , piece_id=3FMVMJWMWGAHNVQX2AIBVII5Z6XTPJUCYABKNXJ4NWRC2LQ6I3FA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.268061592 (806h43m12.139144637s) [noise=true , address=185.167.204.206:4061 , piece_id=LJVZ2GBV4S5T6XXBYDEELH6BKTBY2TTOBBXWNNNH2VSE2W5RI5YA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.268319703 (806h43m12.139402739s) [noise=true , address=185.167.204.205:4017 , piece_id=ACCHFJNRCMWJWXPZJDXALXSUVPLETHOSDAGKPKDAKRMWJ2JI6RMQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.270984638 (806h43m12.142067684s) [noise=true , address=185.167.204.206:4037 , piece_id=HAKQW2RQFJJGBIG26EV4B5TJJBZRM2YMXVU5PZQIKNA5YYEAZVHA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.272396927 (806h43m12.143479962s) [noise=true , address=185.167.204.205:4034 , piece_id=EMHALXT4VKLMJNTQV4XMRGLVOPBRF7O45YKF5Q3RYUBF6DGHPTJQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:13.719906169 (806h43m13.590989215s) [noise=true , address=89.149.201.147:4007 , piece_id=XDKNFX56PLHN6AXAKPM3LSRUHAFHIOZIJF2CAFXOMDLUH7URSS3Q, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:15.774082352 (806h43m15.645165397s) [noise=true , address=89.149.201.147:4005 , piece_id=3WEJWRZJOWY6I7ZNNWHK5QYTF5MULZNKHWVZMBEJJPFNB6WU4UCA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:16.530239032 (806h43m16.401322067s) [noise=true , address=89.149.201.147:4001 , piece_id=LWDR4KCC5QHQDLFVZYVTYJH26GRABV2JFC3525OVIORRG2SRHQMA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:16.928628457 (806h43m16.799711492s) [noise=true , address=81.171.14.70:4041 , piece_id=J3IG4LCWN7UM2NCQAY4FKTPNYH5NRW7TDB6ZGMNPQEI7HY5CZGRQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:17.995225112 (806h43m17.866308157s) [noise=true , address=89.149.201.147:4011 , piece_id=PXK522IVCJHJDLWXUIP3PQF7PJXVPXLPGM6HE43CY6G2VSLOAWPQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:18.045551680 (806h43m17.916634725s) [noise=true , address=81.171.14.70:4027 , piece_id=5P2A7PSIMWZC2I6SAGUEUEJDDW67BD7AKQ34Z3XZHFPJGSHLJLVQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:19.316134332 (806h43m19.187217367s) [noise=true , address=89.149.201.147:4004 , piece_id=BRZPEKISSTSV7IPNBW7ZJJIIGOTR4SQDJDMQZ3OXOSUEQJZB5CMQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:19.607969825 (806h43m19.47905288s) [noise=true , address=81.171.14.70:4035 , piece_id=HAUGVFLDMZCSFVPSOYLS4PBCVHMMMOTPD4ODSDCS3P4NSTX6KPJQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:13:00.649061845 (806h44m0.52014488s) [noise=true , address=89.149.201.147:4010 , piece_id=TM5C7SQQA5IAVFWRG2GV2WYRB2IRLWDLVB3MFWK7XILSBDUDIAVQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:13:10.291882458 (806h44m10.162965513s) [noise=true , address=89.149.201.147:4015 , piece_id=GLHXEDCIEKGM2SPTCNCTJYS2GIEUMIT5LQXCHMDTVNNIY5KLASVQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K, address:89.149.201.147:4015): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.291987709 (806h44m10.163070774s) [noise=true , address=81.171.14.70:4034 , piece_id=VGWIA5ZBLFNA33QXVLF54XTCOCAHSU6TVV3O52DTZ2L3ROIPHNAA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12jcAHt22bETT27CCXJ8ZaaYxxoKBonBYsFcoq6A34DVVjvhQAN, address:81.171.14.70:4034): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.292023307 (806h44m10.163106342s) [noise=true , address=81.171.14.70:4038 , piece_id=QOQ3IYLTGQ27HOLQKQZOOILZXWAPR6PEHLQTHLGLBLXLLQDBA5RA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB, address:81.171.14.70:4038): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.292120161 (806h44m10.163203216s) [noise=true , address=89.149.201.147:4020 , piece_id=U3G7AY7GMBA527UQXXY6DOLNKJBUR4CZD7T3MOJSQNFFGYJXKR3A, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.292271189 (806h44m10.163354245s) [noise=true , address=81.171.14.70:4048 , piece_id=HLP5YVTQMQXLGROPTZJ4RPKFYCCBB3W5ZPHASDVMTJ6ZCTSGMDKQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: read tcp 188.166.225.170:40684->81.171.14.70:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 8 06:13:10.292450230 (806h44m10.163533285s) [noise=true , address=81.171.14.70:4043 , piece_id=AM727IICNSFNSVCQ3IZ55CAZK6YH3JUFBOJAUHMMYRJ6NEZZREDQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko, address:81.171.14.70:4043): protocol: read tcp 188.166.225.170:38496->81.171.14.70:4043: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 8 06:13:10.292509182 (806h44m10.163592237s) [noise=true , address=81.171.14.70:4029 , piece_id=6UO7K2VEO7HMZPJM4S34Z6Z3UWKFMLD3BLATMSIRL4WDIXKVTZCA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:60086->81.171.14.70:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 8 06:13:10.292533999 (806h44m10.163617034s) [seg_pos={0 0}, upload=705619]: Done waiting for segment. successful: 21 optimal: 26 errs: <nil> Jan 8 06:13:10.462867035 (806h44m10.33395007s) [upload=705619]: Done uploading segments... Jan 8 06:13:10.462901039 (806h44m10.333984085s) [upload=705619]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 21) Duration: 6.539976582s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [XX........X..X...............X....XXX...XX.X......X..X....X.] duration: 31.515444074s trace: https://jaeger.dev.storj.io/trace/5b798fac87e6bbd1 FAIL (2026-01-08 06:43:09 UTC) orbiter warning: context deadline exceeded Duration: 31.515460184s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [..................................X....X....X...............] duration: 9.038620285s trace: https://jaeger.dev.storj.io/trace/23102568ffb9ad71 FAIL (2026-01-08 06:43:06 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 9.038641927s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...........X......XXX..........XX......X...........X........] duration: 5.755596713s trace: https://jaeger.dev.storj.io/trace/11a85404b66ec466 FAIL (2026-01-08 06:43:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: JMZE2QQYMZJLKRLN7DURUWAHXWYRPA4FRQ7H4ITTQUFDIAMKHM5Q) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: HVF4HNEVWBLOPEX6LMJNZZDYRZOLTT3NGFMIGMS46S3TVHDDLNDA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 3OF7GH4SQ2OXWT4AMSH6U542D4RF4T4UBNAKG47QHUGF3AA662VQ) context deadline exceeded ======LOGS====== Jan 8 06:38:10.969187554 (807h9m10.840270599s) [upload=705993]: Starting upload Jan 8 06:38:10.969275200 (807h9m10.840358245s) [upload=705993]: Done starting upload Jan 8 06:38:10.969338862 (807h9m10.840421907s) [upload=705993]: Uploading segments... Jan 8 06:38:10.969657759 (807h9m10.840740804s) [upload=705993]: Got next segment. Inline: false Jan 8 06:38:10.969705860 (807h9m10.840788896s) [upload=705993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 8 06:38:10.969712884 (807h9m10.840795929s) [upload=705993]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 8 06:38:11.214842718 (807h9m11.085925753s) [seg_pos={0 0}, upload=705993]: Begin upload segment... Jan 8 06:38:11.214885970 (807h9m11.085969005s) [seg_pos={0 0}, upload=705993]: Done begin upload segment. Jan 8 06:38:11.214913212 (807h9m11.085996247s) [noise=true , address=45.91.116.9:4262 , piece_id=HVF4HNEVWBLOPEX6LMJNZZDYRZOLTT3NGFMIGMS46S3TVHDDLNDA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.214935674 (807h9m11.086018709s) [noise=true , address=145.79.227.15:4875 , piece_id=66NSAERWEKNI5HJR22SKYZ3UT4ZGIVPHX4GNC7OCWNWUHHYOXQSA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215307472 (807h9m11.086390518s) [noise=true , address=145.79.227.49:4338 , piece_id=BA6QWKQJDAFRXCNQQTEPPTUDOUQRTOV6LJGKUI6PDG54IGIS44KQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215445766 (807h9m11.086528801s) [noise=true , address=145.79.227.47:4267 , piece_id=MAPONRBWD2MPQFXIGOXQ5T4QVUKE726LLXUBWA2WAZOOWSTALAJA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215800942 (807h9m11.086883977s) [noise=true , address=45.91.116.10:4082 , piece_id=H3UICIRBXOP5XWROFAS2BBDZSKLRHX6S3E4OTGXGNGUHVJHRLPKQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215815641 (807h9m11.086898686s) [noise=true , address=145.79.227.46:4602 , piece_id=4DLQBWUICSVM7CDVEDJQJRVHOLA62VDZV55ORHOFUXJKZXNODWRQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215868070 (807h9m11.086951115s) [noise=true , address=77.90.31.16:4736 , piece_id=3OF7GH4SQ2OXWT4AMSH6U542D4RF4T4UBNAKG47QHUGF3AA662VQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216134628 (807h9m11.087217663s) [noise=true , address=45.91.116.14:4191 , piece_id=JMZE2QQYMZJLKRLN7DURUWAHXWYRPA4FRQ7H4ITTQUFDIAMKHM5Q, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216232675 (807h9m11.08731572s) [noise=true , address=145.79.227.50:4375 , piece_id=UIYTXVHAKZ5JMBKMLLNIXNLSCGOGZCWFW6UOLDRLLXTV6G4H4SBA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216462912 (807h9m11.087545947s) [noise=true , address=45.91.116.5:4298 , piece_id=RUKJRTACXQ5XW6RIJ4ZP2DJN5GK6FISSBL73VTQ4QHSTI4KGKEGA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216686669 (807h9m11.087769734s) [noise=true , address=145.79.227.33:4871 , piece_id=OYZWFPL5IJB2CEPUWRK3ENTHYB23ZC65GJ7MRGW222JZRXV4Z4MA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216829160 (807h9m11.087912205s) [noise=true , address=45.91.116.6:4976 , piece_id=JUCN3Z7B66IVMMHFE7KPJUMMQCXIQ2I2LZHQURVGV3INEW6BNXGA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.217197411 (807h9m11.088280446s) [noise=true , address=145.79.227.30:4762 , piece_id=FCFXHYISPDWSMLCEN5D64VSGYN3GHCLMBCZCGB3U77OMMYJCBVFQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.217587214 (807h9m11.088670259s) [noise=true , address=145.79.227.28:4727 , piece_id=ELCOLD2FC5EZNIVQEIJUX4UXAPWMP2NGKUHYPYSYREH5Y4KUABBA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.217682535 (807h9m11.08876558s) [noise=true , address=77.90.31.13:4629 , piece_id=VAZ33WQ2GAX63OREDM2UMJHAYM2V3YHXY72DOK2QGKY42FT44XDQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.218057970 (807h9m11.089141015s) [noise=true , address=145.79.227.45:4230 , piece_id=TIY5UNNH7LQCR4M4TIBI55WDYAAOUY45GGBF5CYAFBETAWH6RXBA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:12.906151024 (807h9m12.777234069s) [upload=705993]: Next returned nil segment Jan 8 06:38:12.906163417 (807h9m12.777246462s) [upload=705993]: Waiting for error group managing segments... Jan 8 06:38:13.191636160 (807h9m13.062719195s) [noise=true , address=145.79.227.46:4602 , piece_id=4DLQBWUICSVM7CDVEDJQJRVHOLA62VDZV55ORHOFUXJKZXNODWRQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.198526130 (807h9m13.069609175s) [noise=true , address=145.79.227.49:4338 , piece_id=BA6QWKQJDAFRXCNQQTEPPTUDOUQRTOV6LJGKUI6PDG54IGIS44KQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.220694191 (807h9m13.091777236s) [noise=true , address=145.79.227.15:4875 , piece_id=66NSAERWEKNI5HJR22SKYZ3UT4ZGIVPHX4GNC7OCWNWUHHYOXQSA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.223448024 (807h9m13.09453107s) [noise=true , address=145.79.227.28:4727 , piece_id=ELCOLD2FC5EZNIVQEIJUX4UXAPWMP2NGKUHYPYSYREH5Y4KUABBA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.226749842 (807h9m13.097832877s) [noise=true , address=145.79.227.45:4230 , piece_id=TIY5UNNH7LQCR4M4TIBI55WDYAAOUY45GGBF5CYAFBETAWH6RXBA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.240985449 (807h9m13.112068484s) [noise=true , address=145.79.227.47:4267 , piece_id=MAPONRBWD2MPQFXIGOXQ5T4QVUKE726LLXUBWA2WAZOOWSTALAJA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.305163744 (807h9m13.176246779s) [noise=true , address=145.79.227.50:4375 , piece_id=UIYTXVHAKZ5JMBKMLLNIXNLSCGOGZCWFW6UOLDRLLXTV6G4H4SBA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.452825885 (807h9m13.32390893s) [noise=true , address=145.79.227.33:4871 , piece_id=OYZWFPL5IJB2CEPUWRK3ENTHYB23ZC65GJ7MRGW222JZRXV4Z4MA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.477222951 (807h9m13.348305996s) [noise=true , address=77.90.31.13:4629 , piece_id=VAZ33WQ2GAX63OREDM2UMJHAYM2V3YHXY72DOK2QGKY42FT44XDQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.487139634 (807h9m13.358222669s) [noise=true , address=77.90.31.16:4736 , piece_id=3OF7GH4SQ2OXWT4AMSH6U542D4RF4T4UBNAKG47QHUGF3AA662VQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.556463607 (807h9m13.427546652s) [noise=true , address=145.79.227.30:4762 , piece_id=FCFXHYISPDWSMLCEN5D64VSGYN3GHCLMBCZCGB3U77OMMYJCBVFQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.627164652 (807h9m13.498247697s) [noise=true , address=45.91.116.14:4191 , piece_id=JMZE2QQYMZJLKRLN7DURUWAHXWYRPA4FRQ7H4ITTQUFDIAMKHM5Q, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.640632367 (807h9m13.511715412s) [noise=true , address=45.91.116.9:4262 , piece_id=HVF4HNEVWBLOPEX6LMJNZZDYRZOLTT3NGFMIGMS46S3TVHDDLNDA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.640653266 (807h9m13.511736301s) [seg_pos={0 0}, upload=705993]: Segment reached optimal threshold of 13 pieces. Jan 8 06:38:13.640904686 (807h9m13.511987721s) [noise=true , address=45.91.116.6:4976 , piece_id=JUCN3Z7B66IVMMHFE7KPJUMMQCXIQ2I2LZHQURVGV3INEW6BNXGA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 8 06:38:13.640913031 (807h9m13.511996067s) [noise=true , address=45.91.116.6:4976 , piece_id=JUCN3Z7B66IVMMHFE7KPJUMMQCXIQ2I2LZHQURVGV3INEW6BNXGA, seg_pos={0 0}, upload=705993]: Long tail cancellation due to optimal threshold Jan 8 06:38:13.640957756 (807h9m13.512040801s) [noise=true , address=45.91.116.10:4082 , piece_id=H3UICIRBXOP5XWROFAS2BBDZSKLRHX6S3E4OTGXGNGUHVJHRLPKQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jan 8 06:38:13.640980901 (807h9m13.512063936s) [noise=true , address=45.91.116.10:4082 , piece_id=H3UICIRBXOP5XWROFAS2BBDZSKLRHX6S3E4OTGXGNGUHVJHRLPKQ, seg_pos={0 0}, upload=705993]: Long tail cancellation due to optimal threshold Jan 8 06:38:13.641034412 (807h9m13.512117447s) [noise=true , address=45.91.116.5:4298 , piece_id=RUKJRTACXQ5XW6RIJ4ZP2DJN5GK6FISSBL73VTQ4QHSTI4KGKEGA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jan 8 06:38:13.641040725 (807h9m13.51212377s) [noise=true , address=45.91.116.5:4298 , piece_id=RUKJRTACXQ5XW6RIJ4ZP2DJN5GK6FISSBL73VTQ4QHSTI4KGKEGA, seg_pos={0 0}, upload=705993]: Long tail cancellation due to optimal threshold Jan 8 06:38:13.641058478 (807h9m13.512141513s) [seg_pos={0 0}, upload=705993]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 8 06:38:13.641146847 (807h9m13.512229892s) [upload=705993]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 8 06:38:13.641152558 (807h9m13.512235593s) [upload=705993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 8 06:38:13.924716336 (807h9m13.795799371s) [upload=705993]: Done uploading segments... Jan 8 06:38:13.924735052 (807h9m13.795818097s) [upload=705993]: Upload finished. err: <nil> Duration: 5.75562078s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [X.XXX.X.....X..X.......XXX.XX.XX..X.XXX.XXXX.XXXXXXX..X.XXX.] duration: 16.550298389s trace: https://jaeger.dev.storj.io/trace/75210dfa1178ee41 FAIL (2026-01-08 06:42:59 UTC) orbiter warning: context deadline exceeded Duration: 16.550346752s edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 3.664582808s trace: https://jaeger.dev.storj.io/trace/6173eb4c5ef6f546 FAIL (2026-01-08 06:42:58 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 53) Duration: 3.664611764s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [............X......XXX..........XX......X...........X.......] duration: 5.687836378s trace: https://jaeger.dev.storj.io/trace/11a85404b66ec466 FAIL (2026-01-08 06:42:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: JMZE2QQYMZJLKRLN7DURUWAHXWYRPA4FRQ7H4ITTQUFDIAMKHM5Q) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: HVF4HNEVWBLOPEX6LMJNZZDYRZOLTT3NGFMIGMS46S3TVHDDLNDA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 3OF7GH4SQ2OXWT4AMSH6U542D4RF4T4UBNAKG47QHUGF3AA662VQ) context deadline exceeded ======LOGS====== Jan 8 06:38:10.969187554 (807h9m10.840270599s) [upload=705993]: Starting upload Jan 8 06:38:10.969275200 (807h9m10.840358245s) [upload=705993]: Done starting upload Jan 8 06:38:10.969338862 (807h9m10.840421907s) [upload=705993]: Uploading segments... Jan 8 06:38:10.969657759 (807h9m10.840740804s) [upload=705993]: Got next segment. Inline: false Jan 8 06:38:10.969705860 (807h9m10.840788896s) [upload=705993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 8 06:38:10.969712884 (807h9m10.840795929s) [upload=705993]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 8 06:38:11.214842718 (807h9m11.085925753s) [seg_pos={0 0}, upload=705993]: Begin upload segment... Jan 8 06:38:11.214885970 (807h9m11.085969005s) [seg_pos={0 0}, upload=705993]: Done begin upload segment. Jan 8 06:38:11.214913212 (807h9m11.085996247s) [noise=true , address=45.91.116.9:4262 , piece_id=HVF4HNEVWBLOPEX6LMJNZZDYRZOLTT3NGFMIGMS46S3TVHDDLNDA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.214935674 (807h9m11.086018709s) [noise=true , address=145.79.227.15:4875 , piece_id=66NSAERWEKNI5HJR22SKYZ3UT4ZGIVPHX4GNC7OCWNWUHHYOXQSA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215307472 (807h9m11.086390518s) [noise=true , address=145.79.227.49:4338 , piece_id=BA6QWKQJDAFRXCNQQTEPPTUDOUQRTOV6LJGKUI6PDG54IGIS44KQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215445766 (807h9m11.086528801s) [noise=true , address=145.79.227.47:4267 , piece_id=MAPONRBWD2MPQFXIGOXQ5T4QVUKE726LLXUBWA2WAZOOWSTALAJA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215800942 (807h9m11.086883977s) [noise=true , address=45.91.116.10:4082 , piece_id=H3UICIRBXOP5XWROFAS2BBDZSKLRHX6S3E4OTGXGNGUHVJHRLPKQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215815641 (807h9m11.086898686s) [noise=true , address=145.79.227.46:4602 , piece_id=4DLQBWUICSVM7CDVEDJQJRVHOLA62VDZV55ORHOFUXJKZXNODWRQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215868070 (807h9m11.086951115s) [noise=true , address=77.90.31.16:4736 , piece_id=3OF7GH4SQ2OXWT4AMSH6U542D4RF4T4UBNAKG47QHUGF3AA662VQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216134628 (807h9m11.087217663s) [noise=true , address=45.91.116.14:4191 , piece_id=JMZE2QQYMZJLKRLN7DURUWAHXWYRPA4FRQ7H4ITTQUFDIAMKHM5Q, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216232675 (807h9m11.08731572s) [noise=true , address=145.79.227.50:4375 , piece_id=UIYTXVHAKZ5JMBKMLLNIXNLSCGOGZCWFW6UOLDRLLXTV6G4H4SBA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216462912 (807h9m11.087545947s) [noise=true , address=45.91.116.5:4298 , piece_id=RUKJRTACXQ5XW6RIJ4ZP2DJN5GK6FISSBL73VTQ4QHSTI4KGKEGA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216686669 (807h9m11.087769734s) [noise=true , address=145.79.227.33:4871 , piece_id=OYZWFPL5IJB2CEPUWRK3ENTHYB23ZC65GJ7MRGW222JZRXV4Z4MA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216829160 (807h9m11.087912205s) [noise=true , address=45.91.116.6:4976 , piece_id=JUCN3Z7B66IVMMHFE7KPJUMMQCXIQ2I2LZHQURVGV3INEW6BNXGA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.217197411 (807h9m11.088280446s) [noise=true , address=145.79.227.30:4762 , piece_id=FCFXHYISPDWSMLCEN5D64VSGYN3GHCLMBCZCGB3U77OMMYJCBVFQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.217587214 (807h9m11.088670259s) [noise=true , address=145.79.227.28:4727 , piece_id=ELCOLD2FC5EZNIVQEIJUX4UXAPWMP2NGKUHYPYSYREH5Y4KUABBA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.217682535 (807h9m11.08876558s) [noise=true , address=77.90.31.13:4629 , piece_id=VAZ33WQ2GAX63OREDM2UMJHAYM2V3YHXY72DOK2QGKY42FT44XDQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.218057970 (807h9m11.089141015s) [noise=true , address=145.79.227.45:4230 , piece_id=TIY5UNNH7LQCR4M4TIBI55WDYAAOUY45GGBF5CYAFBETAWH6RXBA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:12.906151024 (807h9m12.777234069s) [upload=705993]: Next returned nil segment Jan 8 06:38:12.906163417 (807h9m12.777246462s) [upload=705993]: Waiting for error group managing segments... Jan 8 06:38:13.191636160 (807h9m13.062719195s) [noise=true , address=145.79.227.46:4602 , piece_id=4DLQBWUICSVM7CDVEDJQJRVHOLA62VDZV55ORHOFUXJKZXNODWRQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.198526130 (807h9m13.069609175s) [noise=true , address=145.79.227.49:4338 , piece_id=BA6QWKQJDAFRXCNQQTEPPTUDOUQRTOV6LJGKUI6PDG54IGIS44KQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.220694191 (807h9m13.091777236s) [noise=true , address=145.79.227.15:4875 , piece_id=66NSAERWEKNI5HJR22SKYZ3UT4ZGIVPHX4GNC7OCWNWUHHYOXQSA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.223448024 (807h9m13.09453107s) [noise=true , address=145.79.227.28:4727 , piece_id=ELCOLD2FC5EZNIVQEIJUX4UXAPWMP2NGKUHYPYSYREH5Y4KUABBA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.226749842 (807h9m13.097832877s) [noise=true , address=145.79.227.45:4230 , piece_id=TIY5UNNH7LQCR4M4TIBI55WDYAAOUY45GGBF5CYAFBETAWH6RXBA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.240985449 (807h9m13.112068484s) [noise=true , address=145.79.227.47:4267 , piece_id=MAPONRBWD2MPQFXIGOXQ5T4QVUKE726LLXUBWA2WAZOOWSTALAJA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.305163744 (807h9m13.176246779s) [noise=true , address=145.79.227.50:4375 , piece_id=UIYTXVHAKZ5JMBKMLLNIXNLSCGOGZCWFW6UOLDRLLXTV6G4H4SBA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.452825885 (807h9m13.32390893s) [noise=true , address=145.79.227.33:4871 , piece_id=OYZWFPL5IJB2CEPUWRK3ENTHYB23ZC65GJ7MRGW222JZRXV4Z4MA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.477222951 (807h9m13.348305996s) [noise=true , address=77.90.31.13:4629 , piece_id=VAZ33WQ2GAX63OREDM2UMJHAYM2V3YHXY72DOK2QGKY42FT44XDQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.487139634 (807h9m13.358222669s) [noise=true , address=77.90.31.16:4736 , piece_id=3OF7GH4SQ2OXWT4AMSH6U542D4RF4T4UBNAKG47QHUGF3AA662VQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.556463607 (807h9m13.427546652s) [noise=true , address=145.79.227.30:4762 , piece_id=FCFXHYISPDWSMLCEN5D64VSGYN3GHCLMBCZCGB3U77OMMYJCBVFQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.627164652 (807h9m13.498247697s) [noise=true , address=45.91.116.14:4191 , piece_id=JMZE2QQYMZJLKRLN7DURUWAHXWYRPA4FRQ7H4ITTQUFDIAMKHM5Q, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.640632367 (807h9m13.511715412s) [noise=true , address=45.91.116.9:4262 , piece_id=HVF4HNEVWBLOPEX6LMJNZZDYRZOLTT3NGFMIGMS46S3TVHDDLNDA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.640653266 (807h9m13.511736301s) [seg_pos={0 0}, upload=705993]: Segment reached optimal threshold of 13 pieces. Jan 8 06:38:13.640904686 (807h9m13.511987721s) [noise=true , address=45.91.116.6:4976 , piece_id=JUCN3Z7B66IVMMHFE7KPJUMMQCXIQ2I2LZHQURVGV3INEW6BNXGA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 8 06:38:13.640913031 (807h9m13.511996067s) [noise=true , address=45.91.116.6:4976 , piece_id=JUCN3Z7B66IVMMHFE7KPJUMMQCXIQ2I2LZHQURVGV3INEW6BNXGA, seg_pos={0 0}, upload=705993]: Long tail cancellation due to optimal threshold Jan 8 06:38:13.640957756 (807h9m13.512040801s) [noise=true , address=45.91.116.10:4082 , piece_id=H3UICIRBXOP5XWROFAS2BBDZSKLRHX6S3E4OTGXGNGUHVJHRLPKQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jan 8 06:38:13.640980901 (807h9m13.512063936s) [noise=true , address=45.91.116.10:4082 , piece_id=H3UICIRBXOP5XWROFAS2BBDZSKLRHX6S3E4OTGXGNGUHVJHRLPKQ, seg_pos={0 0}, upload=705993]: Long tail cancellation due to optimal threshold Jan 8 06:38:13.641034412 (807h9m13.512117447s) [noise=true , address=45.91.116.5:4298 , piece_id=RUKJRTACXQ5XW6RIJ4ZP2DJN5GK6FISSBL73VTQ4QHSTI4KGKEGA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jan 8 06:38:13.641040725 (807h9m13.51212377s) [noise=true , address=45.91.116.5:4298 , piece_id=RUKJRTACXQ5XW6RIJ4ZP2DJN5GK6FISSBL73VTQ4QHSTI4KGKEGA, seg_pos={0 0}, upload=705993]: Long tail cancellation due to optimal threshold Jan 8 06:38:13.641058478 (807h9m13.512141513s) [seg_pos={0 0}, upload=705993]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 8 06:38:13.641146847 (807h9m13.512229892s) [upload=705993]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 8 06:38:13.641152558 (807h9m13.512235593s) [upload=705993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 8 06:38:13.924716336 (807h9m13.795799371s) [upload=705993]: Done uploading segments... Jan 8 06:38:13.924735052 (807h9m13.795818097s) [upload=705993]: Upload finished. err: <nil> Duration: 5.687872054s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X..XX..............................................] duration: 4.956172951s trace: https://jaeger.dev.storj.io/trace/233c74a6b9db5cf8 FAIL (2026-01-08 06:42:42 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 53) Duration: 4.956200454s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [XX.X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000349688s trace: https://jaeger.dev.storj.io/trace/d588cde17c7df91 FAIL (2026-01-08 06:42:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000375717s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [XXX........X..X...............X....XXX...XX.X......X..X....X] duration: 1m0.001684477s trace: https://jaeger.dev.storj.io/trace/5b798fac87e6bbd1 FAIL (2026-01-08 06:42:37 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001695649s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [...................................X....X....X..............] duration: 12.702797314s trace: https://jaeger.dev.storj.io/trace/23102568ffb9ad71 FAIL (2026-01-08 06:42:28 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 12.702822102s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 8 out of 60 is failed (threshold=3) [..............XXXXXXXX......................................] duration: 6.55646284s trace: https://jaeger.dev.storj.io/trace/635cbd49e9d724cb FAIL (2026-01-08 06:42:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 21) ======LOGS====== _id=TM5C7SQQA5IAVFWRG2GV2WYRB2IRLWDLVB3MFWK7XILSBDUDIAVQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.977864602 (806h43m10.848947647s) [noise=true , address=185.167.204.206:4059 , piece_id=FKQBFAG3DNQ5OI5DJ7GAZINSWVSVDZED56SJVHBY2ULJO5HUAGNQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.978752623 (806h43m10.849835658s) [noise=true , address=185.167.204.206:4064 , piece_id=3FMVMJWMWGAHNVQX2AIBVII5Z6XTPJUCYABKNXJ4NWRC2LQ6I3FA, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.979487392 (806h43m10.850570427s) [noise=true , address=89.149.201.147:4020 , piece_id=U3G7AY7GMBA527UQXXY6DOLNKJBUR4CZD7T3MOJSQNFFGYJXKR3A, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.980303676 (806h43m10.851386721s) [noise=true , address=185.167.204.206:4061 , piece_id=LJVZ2GBV4S5T6XXBYDEELH6BKTBY2TTOBBXWNNNH2VSE2W5RI5YA, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.980907016 (806h43m10.851990051s) [noise=true , address=89.149.201.147:4011 , piece_id=PXK522IVCJHJDLWXUIP3PQF7PJXVPXLPGM6HE43CY6G2VSLOAWPQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.981714563 (806h43m10.852797608s) [noise=true , address=81.171.14.70:4043 , piece_id=AM727IICNSFNSVCQ3IZ55CAZK6YH3JUFBOJAUHMMYRJ6NEZZREDQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.982411280 (806h43m10.853494325s) [noise=true , address=89.149.201.147:4004 , piece_id=BRZPEKISSTSV7IPNBW7ZJJIIGOTR4SQDJDMQZ3OXOSUEQJZB5CMQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.983960189 (806h43m10.855043224s) [noise=true , address=81.171.14.70:4048 , piece_id=HLP5YVTQMQXLGROPTZJ4RPKFYCCBB3W5ZPHASDVMTJ6ZCTSGMDKQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:11.816868763 (806h43m11.687951808s) [upload=705619]: Next returned nil segment Jan 8 06:12:11.816884092 (806h43m11.687967137s) [upload=705619]: Waiting for error group managing segments... Jan 8 06:12:12.207503101 (806h43m12.078586136s) [noise=true , address=185.167.204.205:4014 , piece_id=42DLCGFRUY5XHUEJPKO6E6BKJSWTUC5IDYRSAE3OXFFVCYA2E67A, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.207636776 (806h43m12.078719821s) [noise=true , address=185.167.204.205:4011 , piece_id=36G2RTRXP6WEHPMQKMEQ3334GYKN6RFC7VYMW6BSCLBQ37OT43YQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.208187303 (806h43m12.079270338s) [noise=true , address=185.167.204.206:4041 , piece_id=CDCLTZOQNX4QLRZ5JT7ZIBQKKBRWVX3SWJSKC46EAIMPSV2SYONA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.208277846 (806h43m12.079360881s) [noise=true , address=185.167.204.205:4003 , piece_id=EGXKQAKMGKB3SDIEFFEVFE6XYGHZ4ZR6A5B6MFDB7XFY7KID4QYA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.211357060 (806h43m12.082440095s) [noise=true , address=185.167.204.206:4062 , piece_id=MJLGFBLTJEG7UHJMFLI72MHHH6FWFLP3O4VNTUAS5Z4KCH27EHOQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.215506782 (806h43m12.086589817s) [noise=true , address=185.167.204.206:4068 , piece_id=L6DRFJFP5OFG2UEGHUGNCRWVGJ7XQFGJJUXNT7WGSEJXS5TMKNRQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.221151259 (806h43m12.092234294s) [noise=true , address=185.167.204.206:4059 , piece_id=FKQBFAG3DNQ5OI5DJ7GAZINSWVSVDZED56SJVHBY2ULJO5HUAGNQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.247553292 (806h43m12.118636337s) [noise=true , address=185.167.204.206:4064 , piece_id=3FMVMJWMWGAHNVQX2AIBVII5Z6XTPJUCYABKNXJ4NWRC2LQ6I3FA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.268061592 (806h43m12.139144637s) [noise=true , address=185.167.204.206:4061 , piece_id=LJVZ2GBV4S5T6XXBYDEELH6BKTBY2TTOBBXWNNNH2VSE2W5RI5YA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.268319703 (806h43m12.139402739s) [noise=true , address=185.167.204.205:4017 , piece_id=ACCHFJNRCMWJWXPZJDXALXSUVPLETHOSDAGKPKDAKRMWJ2JI6RMQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.270984638 (806h43m12.142067684s) [noise=true , address=185.167.204.206:4037 , piece_id=HAKQW2RQFJJGBIG26EV4B5TJJBZRM2YMXVU5PZQIKNA5YYEAZVHA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.272396927 (806h43m12.143479962s) [noise=true , address=185.167.204.205:4034 , piece_id=EMHALXT4VKLMJNTQV4XMRGLVOPBRF7O45YKF5Q3RYUBF6DGHPTJQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:13.719906169 (806h43m13.590989215s) [noise=true , address=89.149.201.147:4007 , piece_id=XDKNFX56PLHN6AXAKPM3LSRUHAFHIOZIJF2CAFXOMDLUH7URSS3Q, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:15.774082352 (806h43m15.645165397s) [noise=true , address=89.149.201.147:4005 , piece_id=3WEJWRZJOWY6I7ZNNWHK5QYTF5MULZNKHWVZMBEJJPFNB6WU4UCA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:16.530239032 (806h43m16.401322067s) [noise=true , address=89.149.201.147:4001 , piece_id=LWDR4KCC5QHQDLFVZYVTYJH26GRABV2JFC3525OVIORRG2SRHQMA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:16.928628457 (806h43m16.799711492s) [noise=true , address=81.171.14.70:4041 , piece_id=J3IG4LCWN7UM2NCQAY4FKTPNYH5NRW7TDB6ZGMNPQEI7HY5CZGRQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:17.995225112 (806h43m17.866308157s) [noise=true , address=89.149.201.147:4011 , piece_id=PXK522IVCJHJDLWXUIP3PQF7PJXVPXLPGM6HE43CY6G2VSLOAWPQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:18.045551680 (806h43m17.916634725s) [noise=true , address=81.171.14.70:4027 , piece_id=5P2A7PSIMWZC2I6SAGUEUEJDDW67BD7AKQ34Z3XZHFPJGSHLJLVQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:19.316134332 (806h43m19.187217367s) [noise=true , address=89.149.201.147:4004 , piece_id=BRZPEKISSTSV7IPNBW7ZJJIIGOTR4SQDJDMQZ3OXOSUEQJZB5CMQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:19.607969825 (806h43m19.47905288s) [noise=true , address=81.171.14.70:4035 , piece_id=HAUGVFLDMZCSFVPSOYLS4PBCVHMMMOTPD4ODSDCS3P4NSTX6KPJQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:13:00.649061845 (806h44m0.52014488s) [noise=true , address=89.149.201.147:4010 , piece_id=TM5C7SQQA5IAVFWRG2GV2WYRB2IRLWDLVB3MFWK7XILSBDUDIAVQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:13:10.291882458 (806h44m10.162965513s) [noise=true , address=89.149.201.147:4015 , piece_id=GLHXEDCIEKGM2SPTCNCTJYS2GIEUMIT5LQXCHMDTVNNIY5KLASVQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K, address:89.149.201.147:4015): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.291987709 (806h44m10.163070774s) [noise=true , address=81.171.14.70:4034 , piece_id=VGWIA5ZBLFNA33QXVLF54XTCOCAHSU6TVV3O52DTZ2L3ROIPHNAA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12jcAHt22bETT27CCXJ8ZaaYxxoKBonBYsFcoq6A34DVVjvhQAN, address:81.171.14.70:4034): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.292023307 (806h44m10.163106342s) [noise=true , address=81.171.14.70:4038 , piece_id=QOQ3IYLTGQ27HOLQKQZOOILZXWAPR6PEHLQTHLGLBLXLLQDBA5RA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB, address:81.171.14.70:4038): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.292120161 (806h44m10.163203216s) [noise=true , address=89.149.201.147:4020 , piece_id=U3G7AY7GMBA527UQXXY6DOLNKJBUR4CZD7T3MOJSQNFFGYJXKR3A, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.292271189 (806h44m10.163354245s) [noise=true , address=81.171.14.70:4048 , piece_id=HLP5YVTQMQXLGROPTZJ4RPKFYCCBB3W5ZPHASDVMTJ6ZCTSGMDKQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: read tcp 188.166.225.170:40684->81.171.14.70:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 8 06:13:10.292450230 (806h44m10.163533285s) [noise=true , address=81.171.14.70:4043 , piece_id=AM727IICNSFNSVCQ3IZ55CAZK6YH3JUFBOJAUHMMYRJ6NEZZREDQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko, address:81.171.14.70:4043): protocol: read tcp 188.166.225.170:38496->81.171.14.70:4043: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 8 06:13:10.292509182 (806h44m10.163592237s) [noise=true , address=81.171.14.70:4029 , piece_id=6UO7K2VEO7HMZPJM4S34Z6Z3UWKFMLD3BLATMSIRL4WDIXKVTZCA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:60086->81.171.14.70:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 8 06:13:10.292533999 (806h44m10.163617034s) [seg_pos={0 0}, upload=705619]: Done waiting for segment. successful: 21 optimal: 26 errs: <nil> Jan 8 06:13:10.462867035 (806h44m10.33395007s) [upload=705619]: Done uploading segments... Jan 8 06:13:10.462901039 (806h44m10.333984085s) [upload=705619]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 21) Duration: 6.556491384s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [.X.XXX.X.....X..X.......XXX.XX.XX..X.XXX.XXXX.XXXXXXX..X.XXX] duration: 30.00150443s trace: https://jaeger.dev.storj.io/trace/75210dfa1178ee41 FAIL (2026-01-08 06:42:19 UTC) orbiter warning: context deadline exceeded Duration: 30.001525942s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.962923015s trace: https://jaeger.dev.storj.io/trace/3738a06b2bfcba74 FAIL (2026-01-08 06:42:18 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12hkSjuH8cdS4byda1bykpe2QJz42HXaK1uRtkTbCQdynovAKSz, Piece ID: 4GTLFPYP5LTIZPMRSOVJNHNBOMLFZTIF7CD5QHSAWAQT2DQYCAEQ) context deadline exceeded; piecestore: (Node ID: 1bWLCSW1tixYzHThBQsUPvXtTpnf4egF8uGHFUXDQkCeE4BwBU, Piece ID: RX5TQUH56ZXKJADRRFT7DX6J6VV32WB3EFRCM456NZ2TD2WHKM3A) context deadline exceeded; piecestore: (Node ID: 1XxiRqM4j35HGGzWPnmrHHfbWS8jryWrW5aY66SgDuDQKHYopv, Piece ID: GAOHJL4R553KJFKTWIRCGC3NHHM4TBFNH6GJYFREM23JQQS645TQ) context deadline exceeded; piecestore: (Node ID: 15LfiPEGogvaJkFN67RXm8egaXMGxsc9XmPo26GFwT5TZKmg75, Piece ID: HD6GDZKL6A43NRG367DI674KCKPZAN4PP6S5ZRHHYR33EPAQ4KRQ) context deadline exceeded; piecestore: (Node ID: 1m2RK4RXMBjqBpWj8Xof42VU2fyPNygiY1WR3gcsVRxd6jKe4E, Piece ID: SCDRYTITO42A42357QKLW7RUWW6VKZLZLYQ3H2NIPXAKYGOIC5HQ) context deadline exceeded; piecestore: (Node ID: 1vtdYMUQ1aEzqZeSWkWD9RceYHunHdt9QwAzy7CwyYruPtSNRw, Piece ID: SYC3VAYM2YKT5ZM2VAQQC7663JAM47LYYCU62ZFRVH5UJDTTXMRQ) context deadline exceeded; piecestore: (Node ID: 1dstfvWPzYyG7oHnsByWcpgv6C4NVw9RPt8MdxtZ634FaLrSt4, Piece ID: D6G2XNNG5QPU2VSWNAP6XPRA3LM5GREUKMQG3KEWTKXFBNSEPIZA) context deadline exceeded; piecestore: (Node ID: 12Bf3H3JPRVT3GKxieKhLpk7ZfXc3FrdtweLJKRpFo6yNTomKDu, Piece ID: UD3L4QJO2DEW7V3SJ7HHPO5P6VWX4MURJKKEQ5KIQUHKAUZOZVTQ) context deadline exceeded; piecestore: (Node ID: 1ZtoenGAVBrgVoJkmkyxDqire258rdYcqC3avm8xAPuBGiD1bU, Piece ID: 4GAT5EPHUEH7UO3IHA5FU23RMY7NGYLJU2CBCX3RS2BCAPIDP7VA) context deadline exceeded; piecestore: (Node ID: 12vhu4j9fCJhbUW9rxZshXEbh9gbG1j1sPzgPhDdiD7ecVTQaHz, Piece ID: 6N3EAXOD5X75DQR6W4AHVJ54XQ7BGOF2BRN32IDBHLLYOOYI3ILQ) context deadline exceeded; piecestore: (Node ID: 1kLgnR5vmDgUuaf1aWAUyDsXuHn8aEwVXEQiruQrkfKpXqTiQ1, Piece ID: DGJRCP2CNO7DMWSJ5XMYFWV6CF22SLA4LPCVWLTBRKCQ6NV7HS7Q) context deadline exceeded Duration: 5.962950578s core uplink us1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 3.628848775s trace: https://jaeger.dev.storj.io/trace/6173eb4c5ef6f546 FAIL (2026-01-08 06:42:16 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 53) Duration: 3.628877951s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.............X......XXX..........XX......X...........X......] duration: 5.807515879s trace: https://jaeger.dev.storj.io/trace/11a85404b66ec466 FAIL (2026-01-08 06:42:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: JMZE2QQYMZJLKRLN7DURUWAHXWYRPA4FRQ7H4ITTQUFDIAMKHM5Q) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: HVF4HNEVWBLOPEX6LMJNZZDYRZOLTT3NGFMIGMS46S3TVHDDLNDA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 3OF7GH4SQ2OXWT4AMSH6U542D4RF4T4UBNAKG47QHUGF3AA662VQ) context deadline exceeded ======LOGS====== Jan 8 06:38:10.969187554 (807h9m10.840270599s) [upload=705993]: Starting upload Jan 8 06:38:10.969275200 (807h9m10.840358245s) [upload=705993]: Done starting upload Jan 8 06:38:10.969338862 (807h9m10.840421907s) [upload=705993]: Uploading segments... Jan 8 06:38:10.969657759 (807h9m10.840740804s) [upload=705993]: Got next segment. Inline: false Jan 8 06:38:10.969705860 (807h9m10.840788896s) [upload=705993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 8 06:38:10.969712884 (807h9m10.840795929s) [upload=705993]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 8 06:38:11.214842718 (807h9m11.085925753s) [seg_pos={0 0}, upload=705993]: Begin upload segment... Jan 8 06:38:11.214885970 (807h9m11.085969005s) [seg_pos={0 0}, upload=705993]: Done begin upload segment. Jan 8 06:38:11.214913212 (807h9m11.085996247s) [noise=true , address=45.91.116.9:4262 , piece_id=HVF4HNEVWBLOPEX6LMJNZZDYRZOLTT3NGFMIGMS46S3TVHDDLNDA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.214935674 (807h9m11.086018709s) [noise=true , address=145.79.227.15:4875 , piece_id=66NSAERWEKNI5HJR22SKYZ3UT4ZGIVPHX4GNC7OCWNWUHHYOXQSA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215307472 (807h9m11.086390518s) [noise=true , address=145.79.227.49:4338 , piece_id=BA6QWKQJDAFRXCNQQTEPPTUDOUQRTOV6LJGKUI6PDG54IGIS44KQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215445766 (807h9m11.086528801s) [noise=true , address=145.79.227.47:4267 , piece_id=MAPONRBWD2MPQFXIGOXQ5T4QVUKE726LLXUBWA2WAZOOWSTALAJA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215800942 (807h9m11.086883977s) [noise=true , address=45.91.116.10:4082 , piece_id=H3UICIRBXOP5XWROFAS2BBDZSKLRHX6S3E4OTGXGNGUHVJHRLPKQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215815641 (807h9m11.086898686s) [noise=true , address=145.79.227.46:4602 , piece_id=4DLQBWUICSVM7CDVEDJQJRVHOLA62VDZV55ORHOFUXJKZXNODWRQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215868070 (807h9m11.086951115s) [noise=true , address=77.90.31.16:4736 , piece_id=3OF7GH4SQ2OXWT4AMSH6U542D4RF4T4UBNAKG47QHUGF3AA662VQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216134628 (807h9m11.087217663s) [noise=true , address=45.91.116.14:4191 , piece_id=JMZE2QQYMZJLKRLN7DURUWAHXWYRPA4FRQ7H4ITTQUFDIAMKHM5Q, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216232675 (807h9m11.08731572s) [noise=true , address=145.79.227.50:4375 , piece_id=UIYTXVHAKZ5JMBKMLLNIXNLSCGOGZCWFW6UOLDRLLXTV6G4H4SBA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216462912 (807h9m11.087545947s) [noise=true , address=45.91.116.5:4298 , piece_id=RUKJRTACXQ5XW6RIJ4ZP2DJN5GK6FISSBL73VTQ4QHSTI4KGKEGA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216686669 (807h9m11.087769734s) [noise=true , address=145.79.227.33:4871 , piece_id=OYZWFPL5IJB2CEPUWRK3ENTHYB23ZC65GJ7MRGW222JZRXV4Z4MA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216829160 (807h9m11.087912205s) [noise=true , address=45.91.116.6:4976 , piece_id=JUCN3Z7B66IVMMHFE7KPJUMMQCXIQ2I2LZHQURVGV3INEW6BNXGA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.217197411 (807h9m11.088280446s) [noise=true , address=145.79.227.30:4762 , piece_id=FCFXHYISPDWSMLCEN5D64VSGYN3GHCLMBCZCGB3U77OMMYJCBVFQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.217587214 (807h9m11.088670259s) [noise=true , address=145.79.227.28:4727 , piece_id=ELCOLD2FC5EZNIVQEIJUX4UXAPWMP2NGKUHYPYSYREH5Y4KUABBA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.217682535 (807h9m11.08876558s) [noise=true , address=77.90.31.13:4629 , piece_id=VAZ33WQ2GAX63OREDM2UMJHAYM2V3YHXY72DOK2QGKY42FT44XDQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.218057970 (807h9m11.089141015s) [noise=true , address=145.79.227.45:4230 , piece_id=TIY5UNNH7LQCR4M4TIBI55WDYAAOUY45GGBF5CYAFBETAWH6RXBA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:12.906151024 (807h9m12.777234069s) [upload=705993]: Next returned nil segment Jan 8 06:38:12.906163417 (807h9m12.777246462s) [upload=705993]: Waiting for error group managing segments... Jan 8 06:38:13.191636160 (807h9m13.062719195s) [noise=true , address=145.79.227.46:4602 , piece_id=4DLQBWUICSVM7CDVEDJQJRVHOLA62VDZV55ORHOFUXJKZXNODWRQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.198526130 (807h9m13.069609175s) [noise=true , address=145.79.227.49:4338 , piece_id=BA6QWKQJDAFRXCNQQTEPPTUDOUQRTOV6LJGKUI6PDG54IGIS44KQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.220694191 (807h9m13.091777236s) [noise=true , address=145.79.227.15:4875 , piece_id=66NSAERWEKNI5HJR22SKYZ3UT4ZGIVPHX4GNC7OCWNWUHHYOXQSA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.223448024 (807h9m13.09453107s) [noise=true , address=145.79.227.28:4727 , piece_id=ELCOLD2FC5EZNIVQEIJUX4UXAPWMP2NGKUHYPYSYREH5Y4KUABBA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.226749842 (807h9m13.097832877s) [noise=true , address=145.79.227.45:4230 , piece_id=TIY5UNNH7LQCR4M4TIBI55WDYAAOUY45GGBF5CYAFBETAWH6RXBA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.240985449 (807h9m13.112068484s) [noise=true , address=145.79.227.47:4267 , piece_id=MAPONRBWD2MPQFXIGOXQ5T4QVUKE726LLXUBWA2WAZOOWSTALAJA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.305163744 (807h9m13.176246779s) [noise=true , address=145.79.227.50:4375 , piece_id=UIYTXVHAKZ5JMBKMLLNIXNLSCGOGZCWFW6UOLDRLLXTV6G4H4SBA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.452825885 (807h9m13.32390893s) [noise=true , address=145.79.227.33:4871 , piece_id=OYZWFPL5IJB2CEPUWRK3ENTHYB23ZC65GJ7MRGW222JZRXV4Z4MA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.477222951 (807h9m13.348305996s) [noise=true , address=77.90.31.13:4629 , piece_id=VAZ33WQ2GAX63OREDM2UMJHAYM2V3YHXY72DOK2QGKY42FT44XDQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.487139634 (807h9m13.358222669s) [noise=true , address=77.90.31.16:4736 , piece_id=3OF7GH4SQ2OXWT4AMSH6U542D4RF4T4UBNAKG47QHUGF3AA662VQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.556463607 (807h9m13.427546652s) [noise=true , address=145.79.227.30:4762 , piece_id=FCFXHYISPDWSMLCEN5D64VSGYN3GHCLMBCZCGB3U77OMMYJCBVFQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.627164652 (807h9m13.498247697s) [noise=true , address=45.91.116.14:4191 , piece_id=JMZE2QQYMZJLKRLN7DURUWAHXWYRPA4FRQ7H4ITTQUFDIAMKHM5Q, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.640632367 (807h9m13.511715412s) [noise=true , address=45.91.116.9:4262 , piece_id=HVF4HNEVWBLOPEX6LMJNZZDYRZOLTT3NGFMIGMS46S3TVHDDLNDA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.640653266 (807h9m13.511736301s) [seg_pos={0 0}, upload=705993]: Segment reached optimal threshold of 13 pieces. Jan 8 06:38:13.640904686 (807h9m13.511987721s) [noise=true , address=45.91.116.6:4976 , piece_id=JUCN3Z7B66IVMMHFE7KPJUMMQCXIQ2I2LZHQURVGV3INEW6BNXGA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 8 06:38:13.640913031 (807h9m13.511996067s) [noise=true , address=45.91.116.6:4976 , piece_id=JUCN3Z7B66IVMMHFE7KPJUMMQCXIQ2I2LZHQURVGV3INEW6BNXGA, seg_pos={0 0}, upload=705993]: Long tail cancellation due to optimal threshold Jan 8 06:38:13.640957756 (807h9m13.512040801s) [noise=true , address=45.91.116.10:4082 , piece_id=H3UICIRBXOP5XWROFAS2BBDZSKLRHX6S3E4OTGXGNGUHVJHRLPKQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jan 8 06:38:13.640980901 (807h9m13.512063936s) [noise=true , address=45.91.116.10:4082 , piece_id=H3UICIRBXOP5XWROFAS2BBDZSKLRHX6S3E4OTGXGNGUHVJHRLPKQ, seg_pos={0 0}, upload=705993]: Long tail cancellation due to optimal threshold Jan 8 06:38:13.641034412 (807h9m13.512117447s) [noise=true , address=45.91.116.5:4298 , piece_id=RUKJRTACXQ5XW6RIJ4ZP2DJN5GK6FISSBL73VTQ4QHSTI4KGKEGA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jan 8 06:38:13.641040725 (807h9m13.51212377s) [noise=true , address=45.91.116.5:4298 , piece_id=RUKJRTACXQ5XW6RIJ4ZP2DJN5GK6FISSBL73VTQ4QHSTI4KGKEGA, seg_pos={0 0}, upload=705993]: Long tail cancellation due to optimal threshold Jan 8 06:38:13.641058478 (807h9m13.512141513s) [seg_pos={0 0}, upload=705993]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 8 06:38:13.641146847 (807h9m13.512229892s) [upload=705993]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 8 06:38:13.641152558 (807h9m13.512235593s) [upload=705993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 8 06:38:13.924716336 (807h9m13.795799371s) [upload=705993]: Done uploading segments... Jan 8 06:38:13.924735052 (807h9m13.795818097s) [upload=705993]: Upload finished. err: <nil> Duration: 5.80754281s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [....................................X....X....X.............] duration: 7.521276636s trace: https://jaeger.dev.storj.io/trace/23102568ffb9ad71 FAIL (2026-01-08 06:41:59 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 7.521303469s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X..XX.............................................] duration: 5.988300278s trace: https://jaeger.dev.storj.io/trace/233c74a6b9db5cf8 FAIL (2026-01-08 06:41:52 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 53) Duration: 5.988329453s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.518504496s trace: https://jaeger.dev.storj.io/trace/3738a06b2bfcba74 FAIL (2026-01-08 06:41:49 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12hkSjuH8cdS4byda1bykpe2QJz42HXaK1uRtkTbCQdynovAKSz, Piece ID: 4GTLFPYP5LTIZPMRSOVJNHNBOMLFZTIF7CD5QHSAWAQT2DQYCAEQ) context deadline exceeded; piecestore: (Node ID: 1bWLCSW1tixYzHThBQsUPvXtTpnf4egF8uGHFUXDQkCeE4BwBU, Piece ID: RX5TQUH56ZXKJADRRFT7DX6J6VV32WB3EFRCM456NZ2TD2WHKM3A) context deadline exceeded; piecestore: (Node ID: 1XxiRqM4j35HGGzWPnmrHHfbWS8jryWrW5aY66SgDuDQKHYopv, Piece ID: GAOHJL4R553KJFKTWIRCGC3NHHM4TBFNH6GJYFREM23JQQS645TQ) context deadline exceeded; piecestore: (Node ID: 15LfiPEGogvaJkFN67RXm8egaXMGxsc9XmPo26GFwT5TZKmg75, Piece ID: HD6GDZKL6A43NRG367DI674KCKPZAN4PP6S5ZRHHYR33EPAQ4KRQ) context deadline exceeded; piecestore: (Node ID: 1m2RK4RXMBjqBpWj8Xof42VU2fyPNygiY1WR3gcsVRxd6jKe4E, Piece ID: SCDRYTITO42A42357QKLW7RUWW6VKZLZLYQ3H2NIPXAKYGOIC5HQ) context deadline exceeded; piecestore: (Node ID: 1vtdYMUQ1aEzqZeSWkWD9RceYHunHdt9QwAzy7CwyYruPtSNRw, Piece ID: SYC3VAYM2YKT5ZM2VAQQC7663JAM47LYYCU62ZFRVH5UJDTTXMRQ) context deadline exceeded; piecestore: (Node ID: 1dstfvWPzYyG7oHnsByWcpgv6C4NVw9RPt8MdxtZ634FaLrSt4, Piece ID: D6G2XNNG5QPU2VSWNAP6XPRA3LM5GREUKMQG3KEWTKXFBNSEPIZA) context deadline exceeded; piecestore: (Node ID: 12Bf3H3JPRVT3GKxieKhLpk7ZfXc3FrdtweLJKRpFo6yNTomKDu, Piece ID: UD3L4QJO2DEW7V3SJ7HHPO5P6VWX4MURJKKEQ5KIQUHKAUZOZVTQ) context deadline exceeded; piecestore: (Node ID: 1ZtoenGAVBrgVoJkmkyxDqire258rdYcqC3avm8xAPuBGiD1bU, Piece ID: 4GAT5EPHUEH7UO3IHA5FU23RMY7NGYLJU2CBCX3RS2BCAPIDP7VA) context deadline exceeded; piecestore: (Node ID: 12vhu4j9fCJhbUW9rxZshXEbh9gbG1j1sPzgPhDdiD7ecVTQaHz, Piece ID: 6N3EAXOD5X75DQR6W4AHVJ54XQ7BGOF2BRN32IDBHLLYOOYI3ILQ) context deadline exceeded; piecestore: (Node ID: 1kLgnR5vmDgUuaf1aWAUyDsXuHn8aEwVXEQiruQrkfKpXqTiQ1, Piece ID: DGJRCP2CNO7DMWSJ5XMYFWV6CF22SLA4LPCVWLTBRKCQ6NV7HS7Q) context deadline exceeded Duration: 6.518528041s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 8 out of 60 is failed (threshold=3) [...............XXXXXXXX.....................................] duration: 6.665754953s trace: https://jaeger.dev.storj.io/trace/635cbd49e9d724cb FAIL (2026-01-08 06:41:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 21) ======LOGS====== _id=TM5C7SQQA5IAVFWRG2GV2WYRB2IRLWDLVB3MFWK7XILSBDUDIAVQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.977864602 (806h43m10.848947647s) [noise=true , address=185.167.204.206:4059 , piece_id=FKQBFAG3DNQ5OI5DJ7GAZINSWVSVDZED56SJVHBY2ULJO5HUAGNQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.978752623 (806h43m10.849835658s) [noise=true , address=185.167.204.206:4064 , piece_id=3FMVMJWMWGAHNVQX2AIBVII5Z6XTPJUCYABKNXJ4NWRC2LQ6I3FA, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.979487392 (806h43m10.850570427s) [noise=true , address=89.149.201.147:4020 , piece_id=U3G7AY7GMBA527UQXXY6DOLNKJBUR4CZD7T3MOJSQNFFGYJXKR3A, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.980303676 (806h43m10.851386721s) [noise=true , address=185.167.204.206:4061 , piece_id=LJVZ2GBV4S5T6XXBYDEELH6BKTBY2TTOBBXWNNNH2VSE2W5RI5YA, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.980907016 (806h43m10.851990051s) [noise=true , address=89.149.201.147:4011 , piece_id=PXK522IVCJHJDLWXUIP3PQF7PJXVPXLPGM6HE43CY6G2VSLOAWPQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.981714563 (806h43m10.852797608s) [noise=true , address=81.171.14.70:4043 , piece_id=AM727IICNSFNSVCQ3IZ55CAZK6YH3JUFBOJAUHMMYRJ6NEZZREDQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.982411280 (806h43m10.853494325s) [noise=true , address=89.149.201.147:4004 , piece_id=BRZPEKISSTSV7IPNBW7ZJJIIGOTR4SQDJDMQZ3OXOSUEQJZB5CMQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.983960189 (806h43m10.855043224s) [noise=true , address=81.171.14.70:4048 , piece_id=HLP5YVTQMQXLGROPTZJ4RPKFYCCBB3W5ZPHASDVMTJ6ZCTSGMDKQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:11.816868763 (806h43m11.687951808s) [upload=705619]: Next returned nil segment Jan 8 06:12:11.816884092 (806h43m11.687967137s) [upload=705619]: Waiting for error group managing segments... Jan 8 06:12:12.207503101 (806h43m12.078586136s) [noise=true , address=185.167.204.205:4014 , piece_id=42DLCGFRUY5XHUEJPKO6E6BKJSWTUC5IDYRSAE3OXFFVCYA2E67A, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.207636776 (806h43m12.078719821s) [noise=true , address=185.167.204.205:4011 , piece_id=36G2RTRXP6WEHPMQKMEQ3334GYKN6RFC7VYMW6BSCLBQ37OT43YQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.208187303 (806h43m12.079270338s) [noise=true , address=185.167.204.206:4041 , piece_id=CDCLTZOQNX4QLRZ5JT7ZIBQKKBRWVX3SWJSKC46EAIMPSV2SYONA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.208277846 (806h43m12.079360881s) [noise=true , address=185.167.204.205:4003 , piece_id=EGXKQAKMGKB3SDIEFFEVFE6XYGHZ4ZR6A5B6MFDB7XFY7KID4QYA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.211357060 (806h43m12.082440095s) [noise=true , address=185.167.204.206:4062 , piece_id=MJLGFBLTJEG7UHJMFLI72MHHH6FWFLP3O4VNTUAS5Z4KCH27EHOQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.215506782 (806h43m12.086589817s) [noise=true , address=185.167.204.206:4068 , piece_id=L6DRFJFP5OFG2UEGHUGNCRWVGJ7XQFGJJUXNT7WGSEJXS5TMKNRQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.221151259 (806h43m12.092234294s) [noise=true , address=185.167.204.206:4059 , piece_id=FKQBFAG3DNQ5OI5DJ7GAZINSWVSVDZED56SJVHBY2ULJO5HUAGNQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.247553292 (806h43m12.118636337s) [noise=true , address=185.167.204.206:4064 , piece_id=3FMVMJWMWGAHNVQX2AIBVII5Z6XTPJUCYABKNXJ4NWRC2LQ6I3FA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.268061592 (806h43m12.139144637s) [noise=true , address=185.167.204.206:4061 , piece_id=LJVZ2GBV4S5T6XXBYDEELH6BKTBY2TTOBBXWNNNH2VSE2W5RI5YA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.268319703 (806h43m12.139402739s) [noise=true , address=185.167.204.205:4017 , piece_id=ACCHFJNRCMWJWXPZJDXALXSUVPLETHOSDAGKPKDAKRMWJ2JI6RMQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.270984638 (806h43m12.142067684s) [noise=true , address=185.167.204.206:4037 , piece_id=HAKQW2RQFJJGBIG26EV4B5TJJBZRM2YMXVU5PZQIKNA5YYEAZVHA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.272396927 (806h43m12.143479962s) [noise=true , address=185.167.204.205:4034 , piece_id=EMHALXT4VKLMJNTQV4XMRGLVOPBRF7O45YKF5Q3RYUBF6DGHPTJQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:13.719906169 (806h43m13.590989215s) [noise=true , address=89.149.201.147:4007 , piece_id=XDKNFX56PLHN6AXAKPM3LSRUHAFHIOZIJF2CAFXOMDLUH7URSS3Q, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:15.774082352 (806h43m15.645165397s) [noise=true , address=89.149.201.147:4005 , piece_id=3WEJWRZJOWY6I7ZNNWHK5QYTF5MULZNKHWVZMBEJJPFNB6WU4UCA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:16.530239032 (806h43m16.401322067s) [noise=true , address=89.149.201.147:4001 , piece_id=LWDR4KCC5QHQDLFVZYVTYJH26GRABV2JFC3525OVIORRG2SRHQMA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:16.928628457 (806h43m16.799711492s) [noise=true , address=81.171.14.70:4041 , piece_id=J3IG4LCWN7UM2NCQAY4FKTPNYH5NRW7TDB6ZGMNPQEI7HY5CZGRQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:17.995225112 (806h43m17.866308157s) [noise=true , address=89.149.201.147:4011 , piece_id=PXK522IVCJHJDLWXUIP3PQF7PJXVPXLPGM6HE43CY6G2VSLOAWPQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:18.045551680 (806h43m17.916634725s) [noise=true , address=81.171.14.70:4027 , piece_id=5P2A7PSIMWZC2I6SAGUEUEJDDW67BD7AKQ34Z3XZHFPJGSHLJLVQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:19.316134332 (806h43m19.187217367s) [noise=true , address=89.149.201.147:4004 , piece_id=BRZPEKISSTSV7IPNBW7ZJJIIGOTR4SQDJDMQZ3OXOSUEQJZB5CMQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:19.607969825 (806h43m19.47905288s) [noise=true , address=81.171.14.70:4035 , piece_id=HAUGVFLDMZCSFVPSOYLS4PBCVHMMMOTPD4ODSDCS3P4NSTX6KPJQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:13:00.649061845 (806h44m0.52014488s) [noise=true , address=89.149.201.147:4010 , piece_id=TM5C7SQQA5IAVFWRG2GV2WYRB2IRLWDLVB3MFWK7XILSBDUDIAVQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:13:10.291882458 (806h44m10.162965513s) [noise=true , address=89.149.201.147:4015 , piece_id=GLHXEDCIEKGM2SPTCNCTJYS2GIEUMIT5LQXCHMDTVNNIY5KLASVQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K, address:89.149.201.147:4015): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.291987709 (806h44m10.163070774s) [noise=true , address=81.171.14.70:4034 , piece_id=VGWIA5ZBLFNA33QXVLF54XTCOCAHSU6TVV3O52DTZ2L3ROIPHNAA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12jcAHt22bETT27CCXJ8ZaaYxxoKBonBYsFcoq6A34DVVjvhQAN, address:81.171.14.70:4034): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.292023307 (806h44m10.163106342s) [noise=true , address=81.171.14.70:4038 , piece_id=QOQ3IYLTGQ27HOLQKQZOOILZXWAPR6PEHLQTHLGLBLXLLQDBA5RA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB, address:81.171.14.70:4038): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.292120161 (806h44m10.163203216s) [noise=true , address=89.149.201.147:4020 , piece_id=U3G7AY7GMBA527UQXXY6DOLNKJBUR4CZD7T3MOJSQNFFGYJXKR3A, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.292271189 (806h44m10.163354245s) [noise=true , address=81.171.14.70:4048 , piece_id=HLP5YVTQMQXLGROPTZJ4RPKFYCCBB3W5ZPHASDVMTJ6ZCTSGMDKQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: read tcp 188.166.225.170:40684->81.171.14.70:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 8 06:13:10.292450230 (806h44m10.163533285s) [noise=true , address=81.171.14.70:4043 , piece_id=AM727IICNSFNSVCQ3IZ55CAZK6YH3JUFBOJAUHMMYRJ6NEZZREDQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko, address:81.171.14.70:4043): protocol: read tcp 188.166.225.170:38496->81.171.14.70:4043: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 8 06:13:10.292509182 (806h44m10.163592237s) [noise=true , address=81.171.14.70:4029 , piece_id=6UO7K2VEO7HMZPJM4S34Z6Z3UWKFMLD3BLATMSIRL4WDIXKVTZCA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:60086->81.171.14.70:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 8 06:13:10.292533999 (806h44m10.163617034s) [seg_pos={0 0}, upload=705619]: Done waiting for segment. successful: 21 optimal: 26 errs: <nil> Jan 8 06:13:10.462867035 (806h44m10.33395007s) [upload=705619]: Done uploading segments... Jan 8 06:13:10.462901039 (806h44m10.333984085s) [upload=705619]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 21) Duration: 6.665772506s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 3.158288489s trace: https://jaeger.dev.storj.io/trace/6173eb4c5ef6f546 FAIL (2026-01-08 06:41:38 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 53) Duration: 3.158316862s core uplink ap1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.XXX........X..X...............X....XXX...XX.X......X..X....] duration: 20.293076842s trace: https://jaeger.dev.storj.io/trace/b037bcc8f577a49 FAIL (2026-01-08 06:41:37 UTC) orbiter warning: context deadline exceeded Duration: 20.293147326s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X.............X......XXX..........XX......X...........X.....] duration: 5.774740409s trace: https://jaeger.dev.storj.io/trace/11a85404b66ec466 FAIL (2026-01-08 06:41:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: JMZE2QQYMZJLKRLN7DURUWAHXWYRPA4FRQ7H4ITTQUFDIAMKHM5Q) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: HVF4HNEVWBLOPEX6LMJNZZDYRZOLTT3NGFMIGMS46S3TVHDDLNDA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 3OF7GH4SQ2OXWT4AMSH6U542D4RF4T4UBNAKG47QHUGF3AA662VQ) context deadline exceeded ======LOGS====== Jan 8 06:38:10.969187554 (807h9m10.840270599s) [upload=705993]: Starting upload Jan 8 06:38:10.969275200 (807h9m10.840358245s) [upload=705993]: Done starting upload Jan 8 06:38:10.969338862 (807h9m10.840421907s) [upload=705993]: Uploading segments... Jan 8 06:38:10.969657759 (807h9m10.840740804s) [upload=705993]: Got next segment. Inline: false Jan 8 06:38:10.969705860 (807h9m10.840788896s) [upload=705993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 8 06:38:10.969712884 (807h9m10.840795929s) [upload=705993]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 8 06:38:11.214842718 (807h9m11.085925753s) [seg_pos={0 0}, upload=705993]: Begin upload segment... Jan 8 06:38:11.214885970 (807h9m11.085969005s) [seg_pos={0 0}, upload=705993]: Done begin upload segment. Jan 8 06:38:11.214913212 (807h9m11.085996247s) [noise=true , address=45.91.116.9:4262 , piece_id=HVF4HNEVWBLOPEX6LMJNZZDYRZOLTT3NGFMIGMS46S3TVHDDLNDA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.214935674 (807h9m11.086018709s) [noise=true , address=145.79.227.15:4875 , piece_id=66NSAERWEKNI5HJR22SKYZ3UT4ZGIVPHX4GNC7OCWNWUHHYOXQSA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215307472 (807h9m11.086390518s) [noise=true , address=145.79.227.49:4338 , piece_id=BA6QWKQJDAFRXCNQQTEPPTUDOUQRTOV6LJGKUI6PDG54IGIS44KQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215445766 (807h9m11.086528801s) [noise=true , address=145.79.227.47:4267 , piece_id=MAPONRBWD2MPQFXIGOXQ5T4QVUKE726LLXUBWA2WAZOOWSTALAJA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215800942 (807h9m11.086883977s) [noise=true , address=45.91.116.10:4082 , piece_id=H3UICIRBXOP5XWROFAS2BBDZSKLRHX6S3E4OTGXGNGUHVJHRLPKQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215815641 (807h9m11.086898686s) [noise=true , address=145.79.227.46:4602 , piece_id=4DLQBWUICSVM7CDVEDJQJRVHOLA62VDZV55ORHOFUXJKZXNODWRQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215868070 (807h9m11.086951115s) [noise=true , address=77.90.31.16:4736 , piece_id=3OF7GH4SQ2OXWT4AMSH6U542D4RF4T4UBNAKG47QHUGF3AA662VQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216134628 (807h9m11.087217663s) [noise=true , address=45.91.116.14:4191 , piece_id=JMZE2QQYMZJLKRLN7DURUWAHXWYRPA4FRQ7H4ITTQUFDIAMKHM5Q, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216232675 (807h9m11.08731572s) [noise=true , address=145.79.227.50:4375 , piece_id=UIYTXVHAKZ5JMBKMLLNIXNLSCGOGZCWFW6UOLDRLLXTV6G4H4SBA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216462912 (807h9m11.087545947s) [noise=true , address=45.91.116.5:4298 , piece_id=RUKJRTACXQ5XW6RIJ4ZP2DJN5GK6FISSBL73VTQ4QHSTI4KGKEGA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216686669 (807h9m11.087769734s) [noise=true , address=145.79.227.33:4871 , piece_id=OYZWFPL5IJB2CEPUWRK3ENTHYB23ZC65GJ7MRGW222JZRXV4Z4MA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216829160 (807h9m11.087912205s) [noise=true , address=45.91.116.6:4976 , piece_id=JUCN3Z7B66IVMMHFE7KPJUMMQCXIQ2I2LZHQURVGV3INEW6BNXGA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.217197411 (807h9m11.088280446s) [noise=true , address=145.79.227.30:4762 , piece_id=FCFXHYISPDWSMLCEN5D64VSGYN3GHCLMBCZCGB3U77OMMYJCBVFQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.217587214 (807h9m11.088670259s) [noise=true , address=145.79.227.28:4727 , piece_id=ELCOLD2FC5EZNIVQEIJUX4UXAPWMP2NGKUHYPYSYREH5Y4KUABBA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.217682535 (807h9m11.08876558s) [noise=true , address=77.90.31.13:4629 , piece_id=VAZ33WQ2GAX63OREDM2UMJHAYM2V3YHXY72DOK2QGKY42FT44XDQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.218057970 (807h9m11.089141015s) [noise=true , address=145.79.227.45:4230 , piece_id=TIY5UNNH7LQCR4M4TIBI55WDYAAOUY45GGBF5CYAFBETAWH6RXBA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:12.906151024 (807h9m12.777234069s) [upload=705993]: Next returned nil segment Jan 8 06:38:12.906163417 (807h9m12.777246462s) [upload=705993]: Waiting for error group managing segments... Jan 8 06:38:13.191636160 (807h9m13.062719195s) [noise=true , address=145.79.227.46:4602 , piece_id=4DLQBWUICSVM7CDVEDJQJRVHOLA62VDZV55ORHOFUXJKZXNODWRQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.198526130 (807h9m13.069609175s) [noise=true , address=145.79.227.49:4338 , piece_id=BA6QWKQJDAFRXCNQQTEPPTUDOUQRTOV6LJGKUI6PDG54IGIS44KQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.220694191 (807h9m13.091777236s) [noise=true , address=145.79.227.15:4875 , piece_id=66NSAERWEKNI5HJR22SKYZ3UT4ZGIVPHX4GNC7OCWNWUHHYOXQSA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.223448024 (807h9m13.09453107s) [noise=true , address=145.79.227.28:4727 , piece_id=ELCOLD2FC5EZNIVQEIJUX4UXAPWMP2NGKUHYPYSYREH5Y4KUABBA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.226749842 (807h9m13.097832877s) [noise=true , address=145.79.227.45:4230 , piece_id=TIY5UNNH7LQCR4M4TIBI55WDYAAOUY45GGBF5CYAFBETAWH6RXBA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.240985449 (807h9m13.112068484s) [noise=true , address=145.79.227.47:4267 , piece_id=MAPONRBWD2MPQFXIGOXQ5T4QVUKE726LLXUBWA2WAZOOWSTALAJA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.305163744 (807h9m13.176246779s) [noise=true , address=145.79.227.50:4375 , piece_id=UIYTXVHAKZ5JMBKMLLNIXNLSCGOGZCWFW6UOLDRLLXTV6G4H4SBA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.452825885 (807h9m13.32390893s) [noise=true , address=145.79.227.33:4871 , piece_id=OYZWFPL5IJB2CEPUWRK3ENTHYB23ZC65GJ7MRGW222JZRXV4Z4MA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.477222951 (807h9m13.348305996s) [noise=true , address=77.90.31.13:4629 , piece_id=VAZ33WQ2GAX63OREDM2UMJHAYM2V3YHXY72DOK2QGKY42FT44XDQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.487139634 (807h9m13.358222669s) [noise=true , address=77.90.31.16:4736 , piece_id=3OF7GH4SQ2OXWT4AMSH6U542D4RF4T4UBNAKG47QHUGF3AA662VQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.556463607 (807h9m13.427546652s) [noise=true , address=145.79.227.30:4762 , piece_id=FCFXHYISPDWSMLCEN5D64VSGYN3GHCLMBCZCGB3U77OMMYJCBVFQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.627164652 (807h9m13.498247697s) [noise=true , address=45.91.116.14:4191 , piece_id=JMZE2QQYMZJLKRLN7DURUWAHXWYRPA4FRQ7H4ITTQUFDIAMKHM5Q, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.640632367 (807h9m13.511715412s) [noise=true , address=45.91.116.9:4262 , piece_id=HVF4HNEVWBLOPEX6LMJNZZDYRZOLTT3NGFMIGMS46S3TVHDDLNDA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.640653266 (807h9m13.511736301s) [seg_pos={0 0}, upload=705993]: Segment reached optimal threshold of 13 pieces. Jan 8 06:38:13.640904686 (807h9m13.511987721s) [noise=true , address=45.91.116.6:4976 , piece_id=JUCN3Z7B66IVMMHFE7KPJUMMQCXIQ2I2LZHQURVGV3INEW6BNXGA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 8 06:38:13.640913031 (807h9m13.511996067s) [noise=true , address=45.91.116.6:4976 , piece_id=JUCN3Z7B66IVMMHFE7KPJUMMQCXIQ2I2LZHQURVGV3INEW6BNXGA, seg_pos={0 0}, upload=705993]: Long tail cancellation due to optimal threshold Jan 8 06:38:13.640957756 (807h9m13.512040801s) [noise=true , address=45.91.116.10:4082 , piece_id=H3UICIRBXOP5XWROFAS2BBDZSKLRHX6S3E4OTGXGNGUHVJHRLPKQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jan 8 06:38:13.640980901 (807h9m13.512063936s) [noise=true , address=45.91.116.10:4082 , piece_id=H3UICIRBXOP5XWROFAS2BBDZSKLRHX6S3E4OTGXGNGUHVJHRLPKQ, seg_pos={0 0}, upload=705993]: Long tail cancellation due to optimal threshold Jan 8 06:38:13.641034412 (807h9m13.512117447s) [noise=true , address=45.91.116.5:4298 , piece_id=RUKJRTACXQ5XW6RIJ4ZP2DJN5GK6FISSBL73VTQ4QHSTI4KGKEGA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jan 8 06:38:13.641040725 (807h9m13.51212377s) [noise=true , address=45.91.116.5:4298 , piece_id=RUKJRTACXQ5XW6RIJ4ZP2DJN5GK6FISSBL73VTQ4QHSTI4KGKEGA, seg_pos={0 0}, upload=705993]: Long tail cancellation due to optimal threshold Jan 8 06:38:13.641058478 (807h9m13.512141513s) [seg_pos={0 0}, upload=705993]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 8 06:38:13.641146847 (807h9m13.512229892s) [upload=705993]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 8 06:38:13.641152558 (807h9m13.512235593s) [upload=705993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 8 06:38:13.924716336 (807h9m13.795799371s) [upload=705993]: Done uploading segments... Jan 8 06:38:13.924735052 (807h9m13.795818097s) [upload=705993]: Upload finished. err: <nil> Duration: 5.774766918s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X..XX............................................] duration: 4.84788761s trace: https://jaeger.dev.storj.io/trace/233c74a6b9db5cf8 FAIL (2026-01-08 06:41:31 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 53) Duration: 4.847915983s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [XXX.X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000271832s trace: https://jaeger.dev.storj.io/trace/43d4e19cc35a36b2 FAIL (2026-01-08 06:41:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00029628s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [X.X.XXX.X.....X..X.......XXX.XX.XX..X.XXX.XXXX.XXXXXXX..X.XX] duration: 30.001401804s trace: https://jaeger.dev.storj.io/trace/d8176320e5a5cfe FAIL (2026-01-08 06:41:22 UTC) orbiter warning: context deadline exceeded Duration: 30.001422604s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.7859543s trace: https://jaeger.dev.storj.io/trace/3738a06b2bfcba74 FAIL (2026-01-08 06:41:07 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12hkSjuH8cdS4byda1bykpe2QJz42HXaK1uRtkTbCQdynovAKSz, Piece ID: 4GTLFPYP5LTIZPMRSOVJNHNBOMLFZTIF7CD5QHSAWAQT2DQYCAEQ) context deadline exceeded; piecestore: (Node ID: 1bWLCSW1tixYzHThBQsUPvXtTpnf4egF8uGHFUXDQkCeE4BwBU, Piece ID: RX5TQUH56ZXKJADRRFT7DX6J6VV32WB3EFRCM456NZ2TD2WHKM3A) context deadline exceeded; piecestore: (Node ID: 1XxiRqM4j35HGGzWPnmrHHfbWS8jryWrW5aY66SgDuDQKHYopv, Piece ID: GAOHJL4R553KJFKTWIRCGC3NHHM4TBFNH6GJYFREM23JQQS645TQ) context deadline exceeded; piecestore: (Node ID: 15LfiPEGogvaJkFN67RXm8egaXMGxsc9XmPo26GFwT5TZKmg75, Piece ID: HD6GDZKL6A43NRG367DI674KCKPZAN4PP6S5ZRHHYR33EPAQ4KRQ) context deadline exceeded; piecestore: (Node ID: 1m2RK4RXMBjqBpWj8Xof42VU2fyPNygiY1WR3gcsVRxd6jKe4E, Piece ID: SCDRYTITO42A42357QKLW7RUWW6VKZLZLYQ3H2NIPXAKYGOIC5HQ) context deadline exceeded; piecestore: (Node ID: 1vtdYMUQ1aEzqZeSWkWD9RceYHunHdt9QwAzy7CwyYruPtSNRw, Piece ID: SYC3VAYM2YKT5ZM2VAQQC7663JAM47LYYCU62ZFRVH5UJDTTXMRQ) context deadline exceeded; piecestore: (Node ID: 1dstfvWPzYyG7oHnsByWcpgv6C4NVw9RPt8MdxtZ634FaLrSt4, Piece ID: D6G2XNNG5QPU2VSWNAP6XPRA3LM5GREUKMQG3KEWTKXFBNSEPIZA) context deadline exceeded; piecestore: (Node ID: 12Bf3H3JPRVT3GKxieKhLpk7ZfXc3FrdtweLJKRpFo6yNTomKDu, Piece ID: UD3L4QJO2DEW7V3SJ7HHPO5P6VWX4MURJKKEQ5KIQUHKAUZOZVTQ) context deadline exceeded; piecestore: (Node ID: 1ZtoenGAVBrgVoJkmkyxDqire258rdYcqC3avm8xAPuBGiD1bU, Piece ID: 4GAT5EPHUEH7UO3IHA5FU23RMY7NGYLJU2CBCX3RS2BCAPIDP7VA) context deadline exceeded; piecestore: (Node ID: 12vhu4j9fCJhbUW9rxZshXEbh9gbG1j1sPzgPhDdiD7ecVTQaHz, Piece ID: 6N3EAXOD5X75DQR6W4AHVJ54XQ7BGOF2BRN32IDBHLLYOOYI3ILQ) context deadline exceeded; piecestore: (Node ID: 1kLgnR5vmDgUuaf1aWAUyDsXuHn8aEwVXEQiruQrkfKpXqTiQ1, Piece ID: DGJRCP2CNO7DMWSJ5XMYFWV6CF22SLA4LPCVWLTBRKCQ6NV7HS7Q) context deadline exceeded Duration: 5.785983826s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 8 out of 60 is failed (threshold=3) [................XXXXXXXX....................................] duration: 6.820780573s trace: https://jaeger.dev.storj.io/trace/635cbd49e9d724cb FAIL (2026-01-08 06:40:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 21) ======LOGS====== _id=TM5C7SQQA5IAVFWRG2GV2WYRB2IRLWDLVB3MFWK7XILSBDUDIAVQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.977864602 (806h43m10.848947647s) [noise=true , address=185.167.204.206:4059 , piece_id=FKQBFAG3DNQ5OI5DJ7GAZINSWVSVDZED56SJVHBY2ULJO5HUAGNQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.978752623 (806h43m10.849835658s) [noise=true , address=185.167.204.206:4064 , piece_id=3FMVMJWMWGAHNVQX2AIBVII5Z6XTPJUCYABKNXJ4NWRC2LQ6I3FA, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.979487392 (806h43m10.850570427s) [noise=true , address=89.149.201.147:4020 , piece_id=U3G7AY7GMBA527UQXXY6DOLNKJBUR4CZD7T3MOJSQNFFGYJXKR3A, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.980303676 (806h43m10.851386721s) [noise=true , address=185.167.204.206:4061 , piece_id=LJVZ2GBV4S5T6XXBYDEELH6BKTBY2TTOBBXWNNNH2VSE2W5RI5YA, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.980907016 (806h43m10.851990051s) [noise=true , address=89.149.201.147:4011 , piece_id=PXK522IVCJHJDLWXUIP3PQF7PJXVPXLPGM6HE43CY6G2VSLOAWPQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.981714563 (806h43m10.852797608s) [noise=true , address=81.171.14.70:4043 , piece_id=AM727IICNSFNSVCQ3IZ55CAZK6YH3JUFBOJAUHMMYRJ6NEZZREDQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.982411280 (806h43m10.853494325s) [noise=true , address=89.149.201.147:4004 , piece_id=BRZPEKISSTSV7IPNBW7ZJJIIGOTR4SQDJDMQZ3OXOSUEQJZB5CMQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.983960189 (806h43m10.855043224s) [noise=true , address=81.171.14.70:4048 , piece_id=HLP5YVTQMQXLGROPTZJ4RPKFYCCBB3W5ZPHASDVMTJ6ZCTSGMDKQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:11.816868763 (806h43m11.687951808s) [upload=705619]: Next returned nil segment Jan 8 06:12:11.816884092 (806h43m11.687967137s) [upload=705619]: Waiting for error group managing segments... Jan 8 06:12:12.207503101 (806h43m12.078586136s) [noise=true , address=185.167.204.205:4014 , piece_id=42DLCGFRUY5XHUEJPKO6E6BKJSWTUC5IDYRSAE3OXFFVCYA2E67A, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.207636776 (806h43m12.078719821s) [noise=true , address=185.167.204.205:4011 , piece_id=36G2RTRXP6WEHPMQKMEQ3334GYKN6RFC7VYMW6BSCLBQ37OT43YQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.208187303 (806h43m12.079270338s) [noise=true , address=185.167.204.206:4041 , piece_id=CDCLTZOQNX4QLRZ5JT7ZIBQKKBRWVX3SWJSKC46EAIMPSV2SYONA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.208277846 (806h43m12.079360881s) [noise=true , address=185.167.204.205:4003 , piece_id=EGXKQAKMGKB3SDIEFFEVFE6XYGHZ4ZR6A5B6MFDB7XFY7KID4QYA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.211357060 (806h43m12.082440095s) [noise=true , address=185.167.204.206:4062 , piece_id=MJLGFBLTJEG7UHJMFLI72MHHH6FWFLP3O4VNTUAS5Z4KCH27EHOQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.215506782 (806h43m12.086589817s) [noise=true , address=185.167.204.206:4068 , piece_id=L6DRFJFP5OFG2UEGHUGNCRWVGJ7XQFGJJUXNT7WGSEJXS5TMKNRQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.221151259 (806h43m12.092234294s) [noise=true , address=185.167.204.206:4059 , piece_id=FKQBFAG3DNQ5OI5DJ7GAZINSWVSVDZED56SJVHBY2ULJO5HUAGNQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.247553292 (806h43m12.118636337s) [noise=true , address=185.167.204.206:4064 , piece_id=3FMVMJWMWGAHNVQX2AIBVII5Z6XTPJUCYABKNXJ4NWRC2LQ6I3FA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.268061592 (806h43m12.139144637s) [noise=true , address=185.167.204.206:4061 , piece_id=LJVZ2GBV4S5T6XXBYDEELH6BKTBY2TTOBBXWNNNH2VSE2W5RI5YA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.268319703 (806h43m12.139402739s) [noise=true , address=185.167.204.205:4017 , piece_id=ACCHFJNRCMWJWXPZJDXALXSUVPLETHOSDAGKPKDAKRMWJ2JI6RMQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.270984638 (806h43m12.142067684s) [noise=true , address=185.167.204.206:4037 , piece_id=HAKQW2RQFJJGBIG26EV4B5TJJBZRM2YMXVU5PZQIKNA5YYEAZVHA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.272396927 (806h43m12.143479962s) [noise=true , address=185.167.204.205:4034 , piece_id=EMHALXT4VKLMJNTQV4XMRGLVOPBRF7O45YKF5Q3RYUBF6DGHPTJQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:13.719906169 (806h43m13.590989215s) [noise=true , address=89.149.201.147:4007 , piece_id=XDKNFX56PLHN6AXAKPM3LSRUHAFHIOZIJF2CAFXOMDLUH7URSS3Q, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:15.774082352 (806h43m15.645165397s) [noise=true , address=89.149.201.147:4005 , piece_id=3WEJWRZJOWY6I7ZNNWHK5QYTF5MULZNKHWVZMBEJJPFNB6WU4UCA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:16.530239032 (806h43m16.401322067s) [noise=true , address=89.149.201.147:4001 , piece_id=LWDR4KCC5QHQDLFVZYVTYJH26GRABV2JFC3525OVIORRG2SRHQMA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:16.928628457 (806h43m16.799711492s) [noise=true , address=81.171.14.70:4041 , piece_id=J3IG4LCWN7UM2NCQAY4FKTPNYH5NRW7TDB6ZGMNPQEI7HY5CZGRQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:17.995225112 (806h43m17.866308157s) [noise=true , address=89.149.201.147:4011 , piece_id=PXK522IVCJHJDLWXUIP3PQF7PJXVPXLPGM6HE43CY6G2VSLOAWPQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:18.045551680 (806h43m17.916634725s) [noise=true , address=81.171.14.70:4027 , piece_id=5P2A7PSIMWZC2I6SAGUEUEJDDW67BD7AKQ34Z3XZHFPJGSHLJLVQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:19.316134332 (806h43m19.187217367s) [noise=true , address=89.149.201.147:4004 , piece_id=BRZPEKISSTSV7IPNBW7ZJJIIGOTR4SQDJDMQZ3OXOSUEQJZB5CMQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:19.607969825 (806h43m19.47905288s) [noise=true , address=81.171.14.70:4035 , piece_id=HAUGVFLDMZCSFVPSOYLS4PBCVHMMMOTPD4ODSDCS3P4NSTX6KPJQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:13:00.649061845 (806h44m0.52014488s) [noise=true , address=89.149.201.147:4010 , piece_id=TM5C7SQQA5IAVFWRG2GV2WYRB2IRLWDLVB3MFWK7XILSBDUDIAVQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:13:10.291882458 (806h44m10.162965513s) [noise=true , address=89.149.201.147:4015 , piece_id=GLHXEDCIEKGM2SPTCNCTJYS2GIEUMIT5LQXCHMDTVNNIY5KLASVQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K, address:89.149.201.147:4015): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.291987709 (806h44m10.163070774s) [noise=true , address=81.171.14.70:4034 , piece_id=VGWIA5ZBLFNA33QXVLF54XTCOCAHSU6TVV3O52DTZ2L3ROIPHNAA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12jcAHt22bETT27CCXJ8ZaaYxxoKBonBYsFcoq6A34DVVjvhQAN, address:81.171.14.70:4034): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.292023307 (806h44m10.163106342s) [noise=true , address=81.171.14.70:4038 , piece_id=QOQ3IYLTGQ27HOLQKQZOOILZXWAPR6PEHLQTHLGLBLXLLQDBA5RA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB, address:81.171.14.70:4038): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.292120161 (806h44m10.163203216s) [noise=true , address=89.149.201.147:4020 , piece_id=U3G7AY7GMBA527UQXXY6DOLNKJBUR4CZD7T3MOJSQNFFGYJXKR3A, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.292271189 (806h44m10.163354245s) [noise=true , address=81.171.14.70:4048 , piece_id=HLP5YVTQMQXLGROPTZJ4RPKFYCCBB3W5ZPHASDVMTJ6ZCTSGMDKQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: read tcp 188.166.225.170:40684->81.171.14.70:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 8 06:13:10.292450230 (806h44m10.163533285s) [noise=true , address=81.171.14.70:4043 , piece_id=AM727IICNSFNSVCQ3IZ55CAZK6YH3JUFBOJAUHMMYRJ6NEZZREDQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko, address:81.171.14.70:4043): protocol: read tcp 188.166.225.170:38496->81.171.14.70:4043: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 8 06:13:10.292509182 (806h44m10.163592237s) [noise=true , address=81.171.14.70:4029 , piece_id=6UO7K2VEO7HMZPJM4S34Z6Z3UWKFMLD3BLATMSIRL4WDIXKVTZCA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:60086->81.171.14.70:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 8 06:13:10.292533999 (806h44m10.163617034s) [seg_pos={0 0}, upload=705619]: Done waiting for segment. successful: 21 optimal: 26 errs: <nil> Jan 8 06:13:10.462867035 (806h44m10.33395007s) [upload=705619]: Done uploading segments... Jan 8 06:13:10.462901039 (806h44m10.333984085s) [upload=705619]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 21) Duration: 6.820812314s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X.............X......XXX..........XX......X...........X....] duration: 5.708564306s trace: https://jaeger.dev.storj.io/trace/11a85404b66ec466 FAIL (2026-01-08 06:40:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: JMZE2QQYMZJLKRLN7DURUWAHXWYRPA4FRQ7H4ITTQUFDIAMKHM5Q) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: HVF4HNEVWBLOPEX6LMJNZZDYRZOLTT3NGFMIGMS46S3TVHDDLNDA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 3OF7GH4SQ2OXWT4AMSH6U542D4RF4T4UBNAKG47QHUGF3AA662VQ) context deadline exceeded ======LOGS====== Jan 8 06:38:10.969187554 (807h9m10.840270599s) [upload=705993]: Starting upload Jan 8 06:38:10.969275200 (807h9m10.840358245s) [upload=705993]: Done starting upload Jan 8 06:38:10.969338862 (807h9m10.840421907s) [upload=705993]: Uploading segments... Jan 8 06:38:10.969657759 (807h9m10.840740804s) [upload=705993]: Got next segment. Inline: false Jan 8 06:38:10.969705860 (807h9m10.840788896s) [upload=705993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 8 06:38:10.969712884 (807h9m10.840795929s) [upload=705993]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 8 06:38:11.214842718 (807h9m11.085925753s) [seg_pos={0 0}, upload=705993]: Begin upload segment... Jan 8 06:38:11.214885970 (807h9m11.085969005s) [seg_pos={0 0}, upload=705993]: Done begin upload segment. Jan 8 06:38:11.214913212 (807h9m11.085996247s) [noise=true , address=45.91.116.9:4262 , piece_id=HVF4HNEVWBLOPEX6LMJNZZDYRZOLTT3NGFMIGMS46S3TVHDDLNDA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.214935674 (807h9m11.086018709s) [noise=true , address=145.79.227.15:4875 , piece_id=66NSAERWEKNI5HJR22SKYZ3UT4ZGIVPHX4GNC7OCWNWUHHYOXQSA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215307472 (807h9m11.086390518s) [noise=true , address=145.79.227.49:4338 , piece_id=BA6QWKQJDAFRXCNQQTEPPTUDOUQRTOV6LJGKUI6PDG54IGIS44KQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215445766 (807h9m11.086528801s) [noise=true , address=145.79.227.47:4267 , piece_id=MAPONRBWD2MPQFXIGOXQ5T4QVUKE726LLXUBWA2WAZOOWSTALAJA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215800942 (807h9m11.086883977s) [noise=true , address=45.91.116.10:4082 , piece_id=H3UICIRBXOP5XWROFAS2BBDZSKLRHX6S3E4OTGXGNGUHVJHRLPKQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215815641 (807h9m11.086898686s) [noise=true , address=145.79.227.46:4602 , piece_id=4DLQBWUICSVM7CDVEDJQJRVHOLA62VDZV55ORHOFUXJKZXNODWRQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215868070 (807h9m11.086951115s) [noise=true , address=77.90.31.16:4736 , piece_id=3OF7GH4SQ2OXWT4AMSH6U542D4RF4T4UBNAKG47QHUGF3AA662VQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216134628 (807h9m11.087217663s) [noise=true , address=45.91.116.14:4191 , piece_id=JMZE2QQYMZJLKRLN7DURUWAHXWYRPA4FRQ7H4ITTQUFDIAMKHM5Q, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216232675 (807h9m11.08731572s) [noise=true , address=145.79.227.50:4375 , piece_id=UIYTXVHAKZ5JMBKMLLNIXNLSCGOGZCWFW6UOLDRLLXTV6G4H4SBA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216462912 (807h9m11.087545947s) [noise=true , address=45.91.116.5:4298 , piece_id=RUKJRTACXQ5XW6RIJ4ZP2DJN5GK6FISSBL73VTQ4QHSTI4KGKEGA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216686669 (807h9m11.087769734s) [noise=true , address=145.79.227.33:4871 , piece_id=OYZWFPL5IJB2CEPUWRK3ENTHYB23ZC65GJ7MRGW222JZRXV4Z4MA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216829160 (807h9m11.087912205s) [noise=true , address=45.91.116.6:4976 , piece_id=JUCN3Z7B66IVMMHFE7KPJUMMQCXIQ2I2LZHQURVGV3INEW6BNXGA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.217197411 (807h9m11.088280446s) [noise=true , address=145.79.227.30:4762 , piece_id=FCFXHYISPDWSMLCEN5D64VSGYN3GHCLMBCZCGB3U77OMMYJCBVFQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.217587214 (807h9m11.088670259s) [noise=true , address=145.79.227.28:4727 , piece_id=ELCOLD2FC5EZNIVQEIJUX4UXAPWMP2NGKUHYPYSYREH5Y4KUABBA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.217682535 (807h9m11.08876558s) [noise=true , address=77.90.31.13:4629 , piece_id=VAZ33WQ2GAX63OREDM2UMJHAYM2V3YHXY72DOK2QGKY42FT44XDQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.218057970 (807h9m11.089141015s) [noise=true , address=145.79.227.45:4230 , piece_id=TIY5UNNH7LQCR4M4TIBI55WDYAAOUY45GGBF5CYAFBETAWH6RXBA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:12.906151024 (807h9m12.777234069s) [upload=705993]: Next returned nil segment Jan 8 06:38:12.906163417 (807h9m12.777246462s) [upload=705993]: Waiting for error group managing segments... Jan 8 06:38:13.191636160 (807h9m13.062719195s) [noise=true , address=145.79.227.46:4602 , piece_id=4DLQBWUICSVM7CDVEDJQJRVHOLA62VDZV55ORHOFUXJKZXNODWRQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.198526130 (807h9m13.069609175s) [noise=true , address=145.79.227.49:4338 , piece_id=BA6QWKQJDAFRXCNQQTEPPTUDOUQRTOV6LJGKUI6PDG54IGIS44KQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.220694191 (807h9m13.091777236s) [noise=true , address=145.79.227.15:4875 , piece_id=66NSAERWEKNI5HJR22SKYZ3UT4ZGIVPHX4GNC7OCWNWUHHYOXQSA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.223448024 (807h9m13.09453107s) [noise=true , address=145.79.227.28:4727 , piece_id=ELCOLD2FC5EZNIVQEIJUX4UXAPWMP2NGKUHYPYSYREH5Y4KUABBA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.226749842 (807h9m13.097832877s) [noise=true , address=145.79.227.45:4230 , piece_id=TIY5UNNH7LQCR4M4TIBI55WDYAAOUY45GGBF5CYAFBETAWH6RXBA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.240985449 (807h9m13.112068484s) [noise=true , address=145.79.227.47:4267 , piece_id=MAPONRBWD2MPQFXIGOXQ5T4QVUKE726LLXUBWA2WAZOOWSTALAJA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.305163744 (807h9m13.176246779s) [noise=true , address=145.79.227.50:4375 , piece_id=UIYTXVHAKZ5JMBKMLLNIXNLSCGOGZCWFW6UOLDRLLXTV6G4H4SBA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.452825885 (807h9m13.32390893s) [noise=true , address=145.79.227.33:4871 , piece_id=OYZWFPL5IJB2CEPUWRK3ENTHYB23ZC65GJ7MRGW222JZRXV4Z4MA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.477222951 (807h9m13.348305996s) [noise=true , address=77.90.31.13:4629 , piece_id=VAZ33WQ2GAX63OREDM2UMJHAYM2V3YHXY72DOK2QGKY42FT44XDQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.487139634 (807h9m13.358222669s) [noise=true , address=77.90.31.16:4736 , piece_id=3OF7GH4SQ2OXWT4AMSH6U542D4RF4T4UBNAKG47QHUGF3AA662VQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.556463607 (807h9m13.427546652s) [noise=true , address=145.79.227.30:4762 , piece_id=FCFXHYISPDWSMLCEN5D64VSGYN3GHCLMBCZCGB3U77OMMYJCBVFQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.627164652 (807h9m13.498247697s) [noise=true , address=45.91.116.14:4191 , piece_id=JMZE2QQYMZJLKRLN7DURUWAHXWYRPA4FRQ7H4ITTQUFDIAMKHM5Q, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.640632367 (807h9m13.511715412s) [noise=true , address=45.91.116.9:4262 , piece_id=HVF4HNEVWBLOPEX6LMJNZZDYRZOLTT3NGFMIGMS46S3TVHDDLNDA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.640653266 (807h9m13.511736301s) [seg_pos={0 0}, upload=705993]: Segment reached optimal threshold of 13 pieces. Jan 8 06:38:13.640904686 (807h9m13.511987721s) [noise=true , address=45.91.116.6:4976 , piece_id=JUCN3Z7B66IVMMHFE7KPJUMMQCXIQ2I2LZHQURVGV3INEW6BNXGA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 8 06:38:13.640913031 (807h9m13.511996067s) [noise=true , address=45.91.116.6:4976 , piece_id=JUCN3Z7B66IVMMHFE7KPJUMMQCXIQ2I2LZHQURVGV3INEW6BNXGA, seg_pos={0 0}, upload=705993]: Long tail cancellation due to optimal threshold Jan 8 06:38:13.640957756 (807h9m13.512040801s) [noise=true , address=45.91.116.10:4082 , piece_id=H3UICIRBXOP5XWROFAS2BBDZSKLRHX6S3E4OTGXGNGUHVJHRLPKQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jan 8 06:38:13.640980901 (807h9m13.512063936s) [noise=true , address=45.91.116.10:4082 , piece_id=H3UICIRBXOP5XWROFAS2BBDZSKLRHX6S3E4OTGXGNGUHVJHRLPKQ, seg_pos={0 0}, upload=705993]: Long tail cancellation due to optimal threshold Jan 8 06:38:13.641034412 (807h9m13.512117447s) [noise=true , address=45.91.116.5:4298 , piece_id=RUKJRTACXQ5XW6RIJ4ZP2DJN5GK6FISSBL73VTQ4QHSTI4KGKEGA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jan 8 06:38:13.641040725 (807h9m13.51212377s) [noise=true , address=45.91.116.5:4298 , piece_id=RUKJRTACXQ5XW6RIJ4ZP2DJN5GK6FISSBL73VTQ4QHSTI4KGKEGA, seg_pos={0 0}, upload=705993]: Long tail cancellation due to optimal threshold Jan 8 06:38:13.641058478 (807h9m13.512141513s) [seg_pos={0 0}, upload=705993]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 8 06:38:13.641146847 (807h9m13.512229892s) [upload=705993]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 8 06:38:13.641152558 (807h9m13.512235593s) [upload=705993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 8 06:38:13.924716336 (807h9m13.795799371s) [upload=705993]: Done uploading segments... Jan 8 06:38:13.924735052 (807h9m13.795818097s) [upload=705993]: Upload finished. err: <nil> Duration: 5.708592089s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X....X....X............] duration: 6.860481779s trace: https://jaeger.dev.storj.io/trace/23102568ffb9ad71 FAIL (2026-01-08 06:40:49 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.8605127s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [X.XXX........X..X...............X....XXX...XX.X......X..X...] duration: 32.885118264s trace: https://jaeger.dev.storj.io/trace/b037bcc8f577a49 FAIL (2026-01-08 06:40:40 UTC) orbiter warning: context deadline exceeded Duration: 32.885146768s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X..XX...........................................] duration: 5.693231891s trace: https://jaeger.dev.storj.io/trace/233c74a6b9db5cf8 FAIL (2026-01-08 06:40:39 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 53) Duration: 5.693260575s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 8 out of 60 is failed (threshold=3) [.................XXXXXXXX...................................] duration: 6.803895211s trace: https://jaeger.dev.storj.io/trace/635cbd49e9d724cb FAIL (2026-01-08 06:40:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 21) ======LOGS====== _id=TM5C7SQQA5IAVFWRG2GV2WYRB2IRLWDLVB3MFWK7XILSBDUDIAVQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.977864602 (806h43m10.848947647s) [noise=true , address=185.167.204.206:4059 , piece_id=FKQBFAG3DNQ5OI5DJ7GAZINSWVSVDZED56SJVHBY2ULJO5HUAGNQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.978752623 (806h43m10.849835658s) [noise=true , address=185.167.204.206:4064 , piece_id=3FMVMJWMWGAHNVQX2AIBVII5Z6XTPJUCYABKNXJ4NWRC2LQ6I3FA, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.979487392 (806h43m10.850570427s) [noise=true , address=89.149.201.147:4020 , piece_id=U3G7AY7GMBA527UQXXY6DOLNKJBUR4CZD7T3MOJSQNFFGYJXKR3A, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.980303676 (806h43m10.851386721s) [noise=true , address=185.167.204.206:4061 , piece_id=LJVZ2GBV4S5T6XXBYDEELH6BKTBY2TTOBBXWNNNH2VSE2W5RI5YA, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.980907016 (806h43m10.851990051s) [noise=true , address=89.149.201.147:4011 , piece_id=PXK522IVCJHJDLWXUIP3PQF7PJXVPXLPGM6HE43CY6G2VSLOAWPQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.981714563 (806h43m10.852797608s) [noise=true , address=81.171.14.70:4043 , piece_id=AM727IICNSFNSVCQ3IZ55CAZK6YH3JUFBOJAUHMMYRJ6NEZZREDQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.982411280 (806h43m10.853494325s) [noise=true , address=89.149.201.147:4004 , piece_id=BRZPEKISSTSV7IPNBW7ZJJIIGOTR4SQDJDMQZ3OXOSUEQJZB5CMQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.983960189 (806h43m10.855043224s) [noise=true , address=81.171.14.70:4048 , piece_id=HLP5YVTQMQXLGROPTZJ4RPKFYCCBB3W5ZPHASDVMTJ6ZCTSGMDKQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:11.816868763 (806h43m11.687951808s) [upload=705619]: Next returned nil segment Jan 8 06:12:11.816884092 (806h43m11.687967137s) [upload=705619]: Waiting for error group managing segments... Jan 8 06:12:12.207503101 (806h43m12.078586136s) [noise=true , address=185.167.204.205:4014 , piece_id=42DLCGFRUY5XHUEJPKO6E6BKJSWTUC5IDYRSAE3OXFFVCYA2E67A, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.207636776 (806h43m12.078719821s) [noise=true , address=185.167.204.205:4011 , piece_id=36G2RTRXP6WEHPMQKMEQ3334GYKN6RFC7VYMW6BSCLBQ37OT43YQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.208187303 (806h43m12.079270338s) [noise=true , address=185.167.204.206:4041 , piece_id=CDCLTZOQNX4QLRZ5JT7ZIBQKKBRWVX3SWJSKC46EAIMPSV2SYONA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.208277846 (806h43m12.079360881s) [noise=true , address=185.167.204.205:4003 , piece_id=EGXKQAKMGKB3SDIEFFEVFE6XYGHZ4ZR6A5B6MFDB7XFY7KID4QYA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.211357060 (806h43m12.082440095s) [noise=true , address=185.167.204.206:4062 , piece_id=MJLGFBLTJEG7UHJMFLI72MHHH6FWFLP3O4VNTUAS5Z4KCH27EHOQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.215506782 (806h43m12.086589817s) [noise=true , address=185.167.204.206:4068 , piece_id=L6DRFJFP5OFG2UEGHUGNCRWVGJ7XQFGJJUXNT7WGSEJXS5TMKNRQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.221151259 (806h43m12.092234294s) [noise=true , address=185.167.204.206:4059 , piece_id=FKQBFAG3DNQ5OI5DJ7GAZINSWVSVDZED56SJVHBY2ULJO5HUAGNQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.247553292 (806h43m12.118636337s) [noise=true , address=185.167.204.206:4064 , piece_id=3FMVMJWMWGAHNVQX2AIBVII5Z6XTPJUCYABKNXJ4NWRC2LQ6I3FA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.268061592 (806h43m12.139144637s) [noise=true , address=185.167.204.206:4061 , piece_id=LJVZ2GBV4S5T6XXBYDEELH6BKTBY2TTOBBXWNNNH2VSE2W5RI5YA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.268319703 (806h43m12.139402739s) [noise=true , address=185.167.204.205:4017 , piece_id=ACCHFJNRCMWJWXPZJDXALXSUVPLETHOSDAGKPKDAKRMWJ2JI6RMQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.270984638 (806h43m12.142067684s) [noise=true , address=185.167.204.206:4037 , piece_id=HAKQW2RQFJJGBIG26EV4B5TJJBZRM2YMXVU5PZQIKNA5YYEAZVHA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.272396927 (806h43m12.143479962s) [noise=true , address=185.167.204.205:4034 , piece_id=EMHALXT4VKLMJNTQV4XMRGLVOPBRF7O45YKF5Q3RYUBF6DGHPTJQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:13.719906169 (806h43m13.590989215s) [noise=true , address=89.149.201.147:4007 , piece_id=XDKNFX56PLHN6AXAKPM3LSRUHAFHIOZIJF2CAFXOMDLUH7URSS3Q, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:15.774082352 (806h43m15.645165397s) [noise=true , address=89.149.201.147:4005 , piece_id=3WEJWRZJOWY6I7ZNNWHK5QYTF5MULZNKHWVZMBEJJPFNB6WU4UCA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:16.530239032 (806h43m16.401322067s) [noise=true , address=89.149.201.147:4001 , piece_id=LWDR4KCC5QHQDLFVZYVTYJH26GRABV2JFC3525OVIORRG2SRHQMA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:16.928628457 (806h43m16.799711492s) [noise=true , address=81.171.14.70:4041 , piece_id=J3IG4LCWN7UM2NCQAY4FKTPNYH5NRW7TDB6ZGMNPQEI7HY5CZGRQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:17.995225112 (806h43m17.866308157s) [noise=true , address=89.149.201.147:4011 , piece_id=PXK522IVCJHJDLWXUIP3PQF7PJXVPXLPGM6HE43CY6G2VSLOAWPQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:18.045551680 (806h43m17.916634725s) [noise=true , address=81.171.14.70:4027 , piece_id=5P2A7PSIMWZC2I6SAGUEUEJDDW67BD7AKQ34Z3XZHFPJGSHLJLVQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:19.316134332 (806h43m19.187217367s) [noise=true , address=89.149.201.147:4004 , piece_id=BRZPEKISSTSV7IPNBW7ZJJIIGOTR4SQDJDMQZ3OXOSUEQJZB5CMQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:19.607969825 (806h43m19.47905288s) [noise=true , address=81.171.14.70:4035 , piece_id=HAUGVFLDMZCSFVPSOYLS4PBCVHMMMOTPD4ODSDCS3P4NSTX6KPJQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:13:00.649061845 (806h44m0.52014488s) [noise=true , address=89.149.201.147:4010 , piece_id=TM5C7SQQA5IAVFWRG2GV2WYRB2IRLWDLVB3MFWK7XILSBDUDIAVQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:13:10.291882458 (806h44m10.162965513s) [noise=true , address=89.149.201.147:4015 , piece_id=GLHXEDCIEKGM2SPTCNCTJYS2GIEUMIT5LQXCHMDTVNNIY5KLASVQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K, address:89.149.201.147:4015): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.291987709 (806h44m10.163070774s) [noise=true , address=81.171.14.70:4034 , piece_id=VGWIA5ZBLFNA33QXVLF54XTCOCAHSU6TVV3O52DTZ2L3ROIPHNAA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12jcAHt22bETT27CCXJ8ZaaYxxoKBonBYsFcoq6A34DVVjvhQAN, address:81.171.14.70:4034): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.292023307 (806h44m10.163106342s) [noise=true , address=81.171.14.70:4038 , piece_id=QOQ3IYLTGQ27HOLQKQZOOILZXWAPR6PEHLQTHLGLBLXLLQDBA5RA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB, address:81.171.14.70:4038): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.292120161 (806h44m10.163203216s) [noise=true , address=89.149.201.147:4020 , piece_id=U3G7AY7GMBA527UQXXY6DOLNKJBUR4CZD7T3MOJSQNFFGYJXKR3A, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.292271189 (806h44m10.163354245s) [noise=true , address=81.171.14.70:4048 , piece_id=HLP5YVTQMQXLGROPTZJ4RPKFYCCBB3W5ZPHASDVMTJ6ZCTSGMDKQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: read tcp 188.166.225.170:40684->81.171.14.70:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 8 06:13:10.292450230 (806h44m10.163533285s) [noise=true , address=81.171.14.70:4043 , piece_id=AM727IICNSFNSVCQ3IZ55CAZK6YH3JUFBOJAUHMMYRJ6NEZZREDQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko, address:81.171.14.70:4043): protocol: read tcp 188.166.225.170:38496->81.171.14.70:4043: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 8 06:13:10.292509182 (806h44m10.163592237s) [noise=true , address=81.171.14.70:4029 , piece_id=6UO7K2VEO7HMZPJM4S34Z6Z3UWKFMLD3BLATMSIRL4WDIXKVTZCA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:60086->81.171.14.70:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 8 06:13:10.292533999 (806h44m10.163617034s) [seg_pos={0 0}, upload=705619]: Done waiting for segment. successful: 21 optimal: 26 errs: <nil> Jan 8 06:13:10.462867035 (806h44m10.33395007s) [upload=705619]: Done uploading segments... Jan 8 06:13:10.462901039 (806h44m10.333984085s) [upload=705619]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 21) Duration: 6.803924337s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 3.441681301s trace: https://jaeger.dev.storj.io/trace/6173eb4c5ef6f546 FAIL (2026-01-08 06:40:31 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 53) Duration: 3.441708071s core uplink ap1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [XXXX.X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.00098016s trace: https://jaeger.dev.storj.io/trace/7edd825706f35921 FAIL (2026-01-08 06:40:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000996862s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [......................................X....X....X...........] duration: 6.602858909s trace: https://jaeger.dev.storj.io/trace/23102568ffb9ad71 FAIL (2026-01-08 06:40:31 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.602886912s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [.X.X.XXX.X.....X..X.......XXX.XX.XX..X.XXX.XXXX.XXXXXXX..X.X] duration: 30.001635272s trace: https://jaeger.dev.storj.io/trace/7407fe8bb660f316 FAIL (2026-01-08 06:40:22 UTC) orbiter warning: context deadline exceeded Duration: 30.001658076s edge gateway us-select-1

1
...
290
291
292
293
294
...
8566