Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [.X.X.XXX.X.....X..X.......XXX.XX.XX..X.XXX.XXXX.XXXXXXX..X.X] duration: 30.001635272s trace: https://jaeger.dev.storj.io/trace/7407fe8bb660f316 FAIL (2026-01-08 06:40:22 UTC) orbiter warning: context deadline exceeded Duration: 30.001658076s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [XXXXX.X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000881842s trace: https://jaeger.dev.storj.io/trace/60bf0726babf4b2f FAIL (2026-01-08 06:40:10 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000910527s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.601715043s trace: https://jaeger.dev.storj.io/trace/3738a06b2bfcba74 FAIL (2026-01-08 06:40:03 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12hkSjuH8cdS4byda1bykpe2QJz42HXaK1uRtkTbCQdynovAKSz, Piece ID: 4GTLFPYP5LTIZPMRSOVJNHNBOMLFZTIF7CD5QHSAWAQT2DQYCAEQ) context deadline exceeded; piecestore: (Node ID: 1bWLCSW1tixYzHThBQsUPvXtTpnf4egF8uGHFUXDQkCeE4BwBU, Piece ID: RX5TQUH56ZXKJADRRFT7DX6J6VV32WB3EFRCM456NZ2TD2WHKM3A) context deadline exceeded; piecestore: (Node ID: 1XxiRqM4j35HGGzWPnmrHHfbWS8jryWrW5aY66SgDuDQKHYopv, Piece ID: GAOHJL4R553KJFKTWIRCGC3NHHM4TBFNH6GJYFREM23JQQS645TQ) context deadline exceeded; piecestore: (Node ID: 15LfiPEGogvaJkFN67RXm8egaXMGxsc9XmPo26GFwT5TZKmg75, Piece ID: HD6GDZKL6A43NRG367DI674KCKPZAN4PP6S5ZRHHYR33EPAQ4KRQ) context deadline exceeded; piecestore: (Node ID: 1m2RK4RXMBjqBpWj8Xof42VU2fyPNygiY1WR3gcsVRxd6jKe4E, Piece ID: SCDRYTITO42A42357QKLW7RUWW6VKZLZLYQ3H2NIPXAKYGOIC5HQ) context deadline exceeded; piecestore: (Node ID: 1vtdYMUQ1aEzqZeSWkWD9RceYHunHdt9QwAzy7CwyYruPtSNRw, Piece ID: SYC3VAYM2YKT5ZM2VAQQC7663JAM47LYYCU62ZFRVH5UJDTTXMRQ) context deadline exceeded; piecestore: (Node ID: 1dstfvWPzYyG7oHnsByWcpgv6C4NVw9RPt8MdxtZ634FaLrSt4, Piece ID: D6G2XNNG5QPU2VSWNAP6XPRA3LM5GREUKMQG3KEWTKXFBNSEPIZA) context deadline exceeded; piecestore: (Node ID: 12Bf3H3JPRVT3GKxieKhLpk7ZfXc3FrdtweLJKRpFo6yNTomKDu, Piece ID: UD3L4QJO2DEW7V3SJ7HHPO5P6VWX4MURJKKEQ5KIQUHKAUZOZVTQ) context deadline exceeded; piecestore: (Node ID: 1ZtoenGAVBrgVoJkmkyxDqire258rdYcqC3avm8xAPuBGiD1bU, Piece ID: 4GAT5EPHUEH7UO3IHA5FU23RMY7NGYLJU2CBCX3RS2BCAPIDP7VA) context deadline exceeded; piecestore: (Node ID: 12vhu4j9fCJhbUW9rxZshXEbh9gbG1j1sPzgPhDdiD7ecVTQaHz, Piece ID: 6N3EAXOD5X75DQR6W4AHVJ54XQ7BGOF2BRN32IDBHLLYOOYI3ILQ) context deadline exceeded; piecestore: (Node ID: 1kLgnR5vmDgUuaf1aWAUyDsXuHn8aEwVXEQiruQrkfKpXqTiQ1, Piece ID: DGJRCP2CNO7DMWSJ5XMYFWV6CF22SLA4LPCVWLTBRKCQ6NV7HS7Q) context deadline exceeded Duration: 6.601736243s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 8 out of 60 is failed (threshold=3) [..................XXXXXXXX..................................] duration: 6.704660611s trace: https://jaeger.dev.storj.io/trace/635cbd49e9d724cb FAIL (2026-01-08 06:40:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 21) ======LOGS====== _id=TM5C7SQQA5IAVFWRG2GV2WYRB2IRLWDLVB3MFWK7XILSBDUDIAVQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.977864602 (806h43m10.848947647s) [noise=true , address=185.167.204.206:4059 , piece_id=FKQBFAG3DNQ5OI5DJ7GAZINSWVSVDZED56SJVHBY2ULJO5HUAGNQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.978752623 (806h43m10.849835658s) [noise=true , address=185.167.204.206:4064 , piece_id=3FMVMJWMWGAHNVQX2AIBVII5Z6XTPJUCYABKNXJ4NWRC2LQ6I3FA, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.979487392 (806h43m10.850570427s) [noise=true , address=89.149.201.147:4020 , piece_id=U3G7AY7GMBA527UQXXY6DOLNKJBUR4CZD7T3MOJSQNFFGYJXKR3A, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.980303676 (806h43m10.851386721s) [noise=true , address=185.167.204.206:4061 , piece_id=LJVZ2GBV4S5T6XXBYDEELH6BKTBY2TTOBBXWNNNH2VSE2W5RI5YA, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.980907016 (806h43m10.851990051s) [noise=true , address=89.149.201.147:4011 , piece_id=PXK522IVCJHJDLWXUIP3PQF7PJXVPXLPGM6HE43CY6G2VSLOAWPQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.981714563 (806h43m10.852797608s) [noise=true , address=81.171.14.70:4043 , piece_id=AM727IICNSFNSVCQ3IZ55CAZK6YH3JUFBOJAUHMMYRJ6NEZZREDQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.982411280 (806h43m10.853494325s) [noise=true , address=89.149.201.147:4004 , piece_id=BRZPEKISSTSV7IPNBW7ZJJIIGOTR4SQDJDMQZ3OXOSUEQJZB5CMQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.983960189 (806h43m10.855043224s) [noise=true , address=81.171.14.70:4048 , piece_id=HLP5YVTQMQXLGROPTZJ4RPKFYCCBB3W5ZPHASDVMTJ6ZCTSGMDKQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:11.816868763 (806h43m11.687951808s) [upload=705619]: Next returned nil segment Jan 8 06:12:11.816884092 (806h43m11.687967137s) [upload=705619]: Waiting for error group managing segments... Jan 8 06:12:12.207503101 (806h43m12.078586136s) [noise=true , address=185.167.204.205:4014 , piece_id=42DLCGFRUY5XHUEJPKO6E6BKJSWTUC5IDYRSAE3OXFFVCYA2E67A, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.207636776 (806h43m12.078719821s) [noise=true , address=185.167.204.205:4011 , piece_id=36G2RTRXP6WEHPMQKMEQ3334GYKN6RFC7VYMW6BSCLBQ37OT43YQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.208187303 (806h43m12.079270338s) [noise=true , address=185.167.204.206:4041 , piece_id=CDCLTZOQNX4QLRZ5JT7ZIBQKKBRWVX3SWJSKC46EAIMPSV2SYONA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.208277846 (806h43m12.079360881s) [noise=true , address=185.167.204.205:4003 , piece_id=EGXKQAKMGKB3SDIEFFEVFE6XYGHZ4ZR6A5B6MFDB7XFY7KID4QYA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.211357060 (806h43m12.082440095s) [noise=true , address=185.167.204.206:4062 , piece_id=MJLGFBLTJEG7UHJMFLI72MHHH6FWFLP3O4VNTUAS5Z4KCH27EHOQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.215506782 (806h43m12.086589817s) [noise=true , address=185.167.204.206:4068 , piece_id=L6DRFJFP5OFG2UEGHUGNCRWVGJ7XQFGJJUXNT7WGSEJXS5TMKNRQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.221151259 (806h43m12.092234294s) [noise=true , address=185.167.204.206:4059 , piece_id=FKQBFAG3DNQ5OI5DJ7GAZINSWVSVDZED56SJVHBY2ULJO5HUAGNQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.247553292 (806h43m12.118636337s) [noise=true , address=185.167.204.206:4064 , piece_id=3FMVMJWMWGAHNVQX2AIBVII5Z6XTPJUCYABKNXJ4NWRC2LQ6I3FA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.268061592 (806h43m12.139144637s) [noise=true , address=185.167.204.206:4061 , piece_id=LJVZ2GBV4S5T6XXBYDEELH6BKTBY2TTOBBXWNNNH2VSE2W5RI5YA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.268319703 (806h43m12.139402739s) [noise=true , address=185.167.204.205:4017 , piece_id=ACCHFJNRCMWJWXPZJDXALXSUVPLETHOSDAGKPKDAKRMWJ2JI6RMQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.270984638 (806h43m12.142067684s) [noise=true , address=185.167.204.206:4037 , piece_id=HAKQW2RQFJJGBIG26EV4B5TJJBZRM2YMXVU5PZQIKNA5YYEAZVHA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.272396927 (806h43m12.143479962s) [noise=true , address=185.167.204.205:4034 , piece_id=EMHALXT4VKLMJNTQV4XMRGLVOPBRF7O45YKF5Q3RYUBF6DGHPTJQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:13.719906169 (806h43m13.590989215s) [noise=true , address=89.149.201.147:4007 , piece_id=XDKNFX56PLHN6AXAKPM3LSRUHAFHIOZIJF2CAFXOMDLUH7URSS3Q, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:15.774082352 (806h43m15.645165397s) [noise=true , address=89.149.201.147:4005 , piece_id=3WEJWRZJOWY6I7ZNNWHK5QYTF5MULZNKHWVZMBEJJPFNB6WU4UCA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:16.530239032 (806h43m16.401322067s) [noise=true , address=89.149.201.147:4001 , piece_id=LWDR4KCC5QHQDLFVZYVTYJH26GRABV2JFC3525OVIORRG2SRHQMA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:16.928628457 (806h43m16.799711492s) [noise=true , address=81.171.14.70:4041 , piece_id=J3IG4LCWN7UM2NCQAY4FKTPNYH5NRW7TDB6ZGMNPQEI7HY5CZGRQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:17.995225112 (806h43m17.866308157s) [noise=true , address=89.149.201.147:4011 , piece_id=PXK522IVCJHJDLWXUIP3PQF7PJXVPXLPGM6HE43CY6G2VSLOAWPQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:18.045551680 (806h43m17.916634725s) [noise=true , address=81.171.14.70:4027 , piece_id=5P2A7PSIMWZC2I6SAGUEUEJDDW67BD7AKQ34Z3XZHFPJGSHLJLVQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:19.316134332 (806h43m19.187217367s) [noise=true , address=89.149.201.147:4004 , piece_id=BRZPEKISSTSV7IPNBW7ZJJIIGOTR4SQDJDMQZ3OXOSUEQJZB5CMQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:19.607969825 (806h43m19.47905288s) [noise=true , address=81.171.14.70:4035 , piece_id=HAUGVFLDMZCSFVPSOYLS4PBCVHMMMOTPD4ODSDCS3P4NSTX6KPJQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:13:00.649061845 (806h44m0.52014488s) [noise=true , address=89.149.201.147:4010 , piece_id=TM5C7SQQA5IAVFWRG2GV2WYRB2IRLWDLVB3MFWK7XILSBDUDIAVQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:13:10.291882458 (806h44m10.162965513s) [noise=true , address=89.149.201.147:4015 , piece_id=GLHXEDCIEKGM2SPTCNCTJYS2GIEUMIT5LQXCHMDTVNNIY5KLASVQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K, address:89.149.201.147:4015): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.291987709 (806h44m10.163070774s) [noise=true , address=81.171.14.70:4034 , piece_id=VGWIA5ZBLFNA33QXVLF54XTCOCAHSU6TVV3O52DTZ2L3ROIPHNAA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12jcAHt22bETT27CCXJ8ZaaYxxoKBonBYsFcoq6A34DVVjvhQAN, address:81.171.14.70:4034): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.292023307 (806h44m10.163106342s) [noise=true , address=81.171.14.70:4038 , piece_id=QOQ3IYLTGQ27HOLQKQZOOILZXWAPR6PEHLQTHLGLBLXLLQDBA5RA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB, address:81.171.14.70:4038): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.292120161 (806h44m10.163203216s) [noise=true , address=89.149.201.147:4020 , piece_id=U3G7AY7GMBA527UQXXY6DOLNKJBUR4CZD7T3MOJSQNFFGYJXKR3A, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.292271189 (806h44m10.163354245s) [noise=true , address=81.171.14.70:4048 , piece_id=HLP5YVTQMQXLGROPTZJ4RPKFYCCBB3W5ZPHASDVMTJ6ZCTSGMDKQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: read tcp 188.166.225.170:40684->81.171.14.70:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 8 06:13:10.292450230 (806h44m10.163533285s) [noise=true , address=81.171.14.70:4043 , piece_id=AM727IICNSFNSVCQ3IZ55CAZK6YH3JUFBOJAUHMMYRJ6NEZZREDQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko, address:81.171.14.70:4043): protocol: read tcp 188.166.225.170:38496->81.171.14.70:4043: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 8 06:13:10.292509182 (806h44m10.163592237s) [noise=true , address=81.171.14.70:4029 , piece_id=6UO7K2VEO7HMZPJM4S34Z6Z3UWKFMLD3BLATMSIRL4WDIXKVTZCA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:60086->81.171.14.70:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 8 06:13:10.292533999 (806h44m10.163617034s) [seg_pos={0 0}, upload=705619]: Done waiting for segment. successful: 21 optimal: 26 errs: <nil> Jan 8 06:13:10.462867035 (806h44m10.33395007s) [upload=705619]: Done uploading segments... Jan 8 06:13:10.462901039 (806h44m10.333984085s) [upload=705619]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 21) Duration: 6.704691009s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X..XX..........................................] duration: 5.079586604s trace: https://jaeger.dev.storj.io/trace/233c74a6b9db5cf8 FAIL (2026-01-08 06:39:58 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 53) Duration: 5.079659782s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X.............X......XXX..........XX......X...........X...] duration: 5.760228611s trace: https://jaeger.dev.storj.io/trace/11a85404b66ec466 FAIL (2026-01-08 06:39:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: JMZE2QQYMZJLKRLN7DURUWAHXWYRPA4FRQ7H4ITTQUFDIAMKHM5Q) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: HVF4HNEVWBLOPEX6LMJNZZDYRZOLTT3NGFMIGMS46S3TVHDDLNDA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 3OF7GH4SQ2OXWT4AMSH6U542D4RF4T4UBNAKG47QHUGF3AA662VQ) context deadline exceeded ======LOGS====== Jan 8 06:38:10.969187554 (807h9m10.840270599s) [upload=705993]: Starting upload Jan 8 06:38:10.969275200 (807h9m10.840358245s) [upload=705993]: Done starting upload Jan 8 06:38:10.969338862 (807h9m10.840421907s) [upload=705993]: Uploading segments... Jan 8 06:38:10.969657759 (807h9m10.840740804s) [upload=705993]: Got next segment. Inline: false Jan 8 06:38:10.969705860 (807h9m10.840788896s) [upload=705993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 8 06:38:10.969712884 (807h9m10.840795929s) [upload=705993]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 8 06:38:11.214842718 (807h9m11.085925753s) [seg_pos={0 0}, upload=705993]: Begin upload segment... Jan 8 06:38:11.214885970 (807h9m11.085969005s) [seg_pos={0 0}, upload=705993]: Done begin upload segment. Jan 8 06:38:11.214913212 (807h9m11.085996247s) [noise=true , address=45.91.116.9:4262 , piece_id=HVF4HNEVWBLOPEX6LMJNZZDYRZOLTT3NGFMIGMS46S3TVHDDLNDA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.214935674 (807h9m11.086018709s) [noise=true , address=145.79.227.15:4875 , piece_id=66NSAERWEKNI5HJR22SKYZ3UT4ZGIVPHX4GNC7OCWNWUHHYOXQSA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215307472 (807h9m11.086390518s) [noise=true , address=145.79.227.49:4338 , piece_id=BA6QWKQJDAFRXCNQQTEPPTUDOUQRTOV6LJGKUI6PDG54IGIS44KQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215445766 (807h9m11.086528801s) [noise=true , address=145.79.227.47:4267 , piece_id=MAPONRBWD2MPQFXIGOXQ5T4QVUKE726LLXUBWA2WAZOOWSTALAJA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215800942 (807h9m11.086883977s) [noise=true , address=45.91.116.10:4082 , piece_id=H3UICIRBXOP5XWROFAS2BBDZSKLRHX6S3E4OTGXGNGUHVJHRLPKQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215815641 (807h9m11.086898686s) [noise=true , address=145.79.227.46:4602 , piece_id=4DLQBWUICSVM7CDVEDJQJRVHOLA62VDZV55ORHOFUXJKZXNODWRQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215868070 (807h9m11.086951115s) [noise=true , address=77.90.31.16:4736 , piece_id=3OF7GH4SQ2OXWT4AMSH6U542D4RF4T4UBNAKG47QHUGF3AA662VQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216134628 (807h9m11.087217663s) [noise=true , address=45.91.116.14:4191 , piece_id=JMZE2QQYMZJLKRLN7DURUWAHXWYRPA4FRQ7H4ITTQUFDIAMKHM5Q, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216232675 (807h9m11.08731572s) [noise=true , address=145.79.227.50:4375 , piece_id=UIYTXVHAKZ5JMBKMLLNIXNLSCGOGZCWFW6UOLDRLLXTV6G4H4SBA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216462912 (807h9m11.087545947s) [noise=true , address=45.91.116.5:4298 , piece_id=RUKJRTACXQ5XW6RIJ4ZP2DJN5GK6FISSBL73VTQ4QHSTI4KGKEGA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216686669 (807h9m11.087769734s) [noise=true , address=145.79.227.33:4871 , piece_id=OYZWFPL5IJB2CEPUWRK3ENTHYB23ZC65GJ7MRGW222JZRXV4Z4MA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216829160 (807h9m11.087912205s) [noise=true , address=45.91.116.6:4976 , piece_id=JUCN3Z7B66IVMMHFE7KPJUMMQCXIQ2I2LZHQURVGV3INEW6BNXGA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.217197411 (807h9m11.088280446s) [noise=true , address=145.79.227.30:4762 , piece_id=FCFXHYISPDWSMLCEN5D64VSGYN3GHCLMBCZCGB3U77OMMYJCBVFQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.217587214 (807h9m11.088670259s) [noise=true , address=145.79.227.28:4727 , piece_id=ELCOLD2FC5EZNIVQEIJUX4UXAPWMP2NGKUHYPYSYREH5Y4KUABBA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.217682535 (807h9m11.08876558s) [noise=true , address=77.90.31.13:4629 , piece_id=VAZ33WQ2GAX63OREDM2UMJHAYM2V3YHXY72DOK2QGKY42FT44XDQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.218057970 (807h9m11.089141015s) [noise=true , address=145.79.227.45:4230 , piece_id=TIY5UNNH7LQCR4M4TIBI55WDYAAOUY45GGBF5CYAFBETAWH6RXBA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:12.906151024 (807h9m12.777234069s) [upload=705993]: Next returned nil segment Jan 8 06:38:12.906163417 (807h9m12.777246462s) [upload=705993]: Waiting for error group managing segments... Jan 8 06:38:13.191636160 (807h9m13.062719195s) [noise=true , address=145.79.227.46:4602 , piece_id=4DLQBWUICSVM7CDVEDJQJRVHOLA62VDZV55ORHOFUXJKZXNODWRQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.198526130 (807h9m13.069609175s) [noise=true , address=145.79.227.49:4338 , piece_id=BA6QWKQJDAFRXCNQQTEPPTUDOUQRTOV6LJGKUI6PDG54IGIS44KQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.220694191 (807h9m13.091777236s) [noise=true , address=145.79.227.15:4875 , piece_id=66NSAERWEKNI5HJR22SKYZ3UT4ZGIVPHX4GNC7OCWNWUHHYOXQSA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.223448024 (807h9m13.09453107s) [noise=true , address=145.79.227.28:4727 , piece_id=ELCOLD2FC5EZNIVQEIJUX4UXAPWMP2NGKUHYPYSYREH5Y4KUABBA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.226749842 (807h9m13.097832877s) [noise=true , address=145.79.227.45:4230 , piece_id=TIY5UNNH7LQCR4M4TIBI55WDYAAOUY45GGBF5CYAFBETAWH6RXBA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.240985449 (807h9m13.112068484s) [noise=true , address=145.79.227.47:4267 , piece_id=MAPONRBWD2MPQFXIGOXQ5T4QVUKE726LLXUBWA2WAZOOWSTALAJA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.305163744 (807h9m13.176246779s) [noise=true , address=145.79.227.50:4375 , piece_id=UIYTXVHAKZ5JMBKMLLNIXNLSCGOGZCWFW6UOLDRLLXTV6G4H4SBA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.452825885 (807h9m13.32390893s) [noise=true , address=145.79.227.33:4871 , piece_id=OYZWFPL5IJB2CEPUWRK3ENTHYB23ZC65GJ7MRGW222JZRXV4Z4MA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.477222951 (807h9m13.348305996s) [noise=true , address=77.90.31.13:4629 , piece_id=VAZ33WQ2GAX63OREDM2UMJHAYM2V3YHXY72DOK2QGKY42FT44XDQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.487139634 (807h9m13.358222669s) [noise=true , address=77.90.31.16:4736 , piece_id=3OF7GH4SQ2OXWT4AMSH6U542D4RF4T4UBNAKG47QHUGF3AA662VQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.556463607 (807h9m13.427546652s) [noise=true , address=145.79.227.30:4762 , piece_id=FCFXHYISPDWSMLCEN5D64VSGYN3GHCLMBCZCGB3U77OMMYJCBVFQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.627164652 (807h9m13.498247697s) [noise=true , address=45.91.116.14:4191 , piece_id=JMZE2QQYMZJLKRLN7DURUWAHXWYRPA4FRQ7H4ITTQUFDIAMKHM5Q, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.640632367 (807h9m13.511715412s) [noise=true , address=45.91.116.9:4262 , piece_id=HVF4HNEVWBLOPEX6LMJNZZDYRZOLTT3NGFMIGMS46S3TVHDDLNDA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.640653266 (807h9m13.511736301s) [seg_pos={0 0}, upload=705993]: Segment reached optimal threshold of 13 pieces. Jan 8 06:38:13.640904686 (807h9m13.511987721s) [noise=true , address=45.91.116.6:4976 , piece_id=JUCN3Z7B66IVMMHFE7KPJUMMQCXIQ2I2LZHQURVGV3INEW6BNXGA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 8 06:38:13.640913031 (807h9m13.511996067s) [noise=true , address=45.91.116.6:4976 , piece_id=JUCN3Z7B66IVMMHFE7KPJUMMQCXIQ2I2LZHQURVGV3INEW6BNXGA, seg_pos={0 0}, upload=705993]: Long tail cancellation due to optimal threshold Jan 8 06:38:13.640957756 (807h9m13.512040801s) [noise=true , address=45.91.116.10:4082 , piece_id=H3UICIRBXOP5XWROFAS2BBDZSKLRHX6S3E4OTGXGNGUHVJHRLPKQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jan 8 06:38:13.640980901 (807h9m13.512063936s) [noise=true , address=45.91.116.10:4082 , piece_id=H3UICIRBXOP5XWROFAS2BBDZSKLRHX6S3E4OTGXGNGUHVJHRLPKQ, seg_pos={0 0}, upload=705993]: Long tail cancellation due to optimal threshold Jan 8 06:38:13.641034412 (807h9m13.512117447s) [noise=true , address=45.91.116.5:4298 , piece_id=RUKJRTACXQ5XW6RIJ4ZP2DJN5GK6FISSBL73VTQ4QHSTI4KGKEGA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jan 8 06:38:13.641040725 (807h9m13.51212377s) [noise=true , address=45.91.116.5:4298 , piece_id=RUKJRTACXQ5XW6RIJ4ZP2DJN5GK6FISSBL73VTQ4QHSTI4KGKEGA, seg_pos={0 0}, upload=705993]: Long tail cancellation due to optimal threshold Jan 8 06:38:13.641058478 (807h9m13.512141513s) [seg_pos={0 0}, upload=705993]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 8 06:38:13.641146847 (807h9m13.512229892s) [upload=705993]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 8 06:38:13.641152558 (807h9m13.512235593s) [upload=705993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 8 06:38:13.924716336 (807h9m13.795799371s) [upload=705993]: Done uploading segments... Jan 8 06:38:13.924735052 (807h9m13.795818097s) [upload=705993]: Upload finished. err: <nil> Duration: 5.760252827s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 3.3467155s trace: https://jaeger.dev.storj.io/trace/6173eb4c5ef6f546 FAIL (2026-01-08 06:39:52 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 53) Duration: 3.346744677s core uplink ap1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.X.XXX........X..X...............X....XXX...XX.X......X..X..] duration: 26.556656448s trace: https://jaeger.dev.storj.io/trace/b037bcc8f577a49 FAIL (2026-01-08 06:39:49 UTC) orbiter warning: context deadline exceeded Duration: 26.556677998s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X..XX.........................................] duration: 5.402824119s trace: https://jaeger.dev.storj.io/trace/233c74a6b9db5cf8 FAIL (2026-01-08 06:39:39 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 53) Duration: 5.402853034s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X.............X......XXX..........XX......X...........X..] duration: 5.792991309s trace: https://jaeger.dev.storj.io/trace/11a85404b66ec466 FAIL (2026-01-08 06:39:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: JMZE2QQYMZJLKRLN7DURUWAHXWYRPA4FRQ7H4ITTQUFDIAMKHM5Q) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: HVF4HNEVWBLOPEX6LMJNZZDYRZOLTT3NGFMIGMS46S3TVHDDLNDA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 3OF7GH4SQ2OXWT4AMSH6U542D4RF4T4UBNAKG47QHUGF3AA662VQ) context deadline exceeded ======LOGS====== Jan 8 06:38:10.969187554 (807h9m10.840270599s) [upload=705993]: Starting upload Jan 8 06:38:10.969275200 (807h9m10.840358245s) [upload=705993]: Done starting upload Jan 8 06:38:10.969338862 (807h9m10.840421907s) [upload=705993]: Uploading segments... Jan 8 06:38:10.969657759 (807h9m10.840740804s) [upload=705993]: Got next segment. Inline: false Jan 8 06:38:10.969705860 (807h9m10.840788896s) [upload=705993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 8 06:38:10.969712884 (807h9m10.840795929s) [upload=705993]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 8 06:38:11.214842718 (807h9m11.085925753s) [seg_pos={0 0}, upload=705993]: Begin upload segment... Jan 8 06:38:11.214885970 (807h9m11.085969005s) [seg_pos={0 0}, upload=705993]: Done begin upload segment. Jan 8 06:38:11.214913212 (807h9m11.085996247s) [noise=true , address=45.91.116.9:4262 , piece_id=HVF4HNEVWBLOPEX6LMJNZZDYRZOLTT3NGFMIGMS46S3TVHDDLNDA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.214935674 (807h9m11.086018709s) [noise=true , address=145.79.227.15:4875 , piece_id=66NSAERWEKNI5HJR22SKYZ3UT4ZGIVPHX4GNC7OCWNWUHHYOXQSA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215307472 (807h9m11.086390518s) [noise=true , address=145.79.227.49:4338 , piece_id=BA6QWKQJDAFRXCNQQTEPPTUDOUQRTOV6LJGKUI6PDG54IGIS44KQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215445766 (807h9m11.086528801s) [noise=true , address=145.79.227.47:4267 , piece_id=MAPONRBWD2MPQFXIGOXQ5T4QVUKE726LLXUBWA2WAZOOWSTALAJA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215800942 (807h9m11.086883977s) [noise=true , address=45.91.116.10:4082 , piece_id=H3UICIRBXOP5XWROFAS2BBDZSKLRHX6S3E4OTGXGNGUHVJHRLPKQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215815641 (807h9m11.086898686s) [noise=true , address=145.79.227.46:4602 , piece_id=4DLQBWUICSVM7CDVEDJQJRVHOLA62VDZV55ORHOFUXJKZXNODWRQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215868070 (807h9m11.086951115s) [noise=true , address=77.90.31.16:4736 , piece_id=3OF7GH4SQ2OXWT4AMSH6U542D4RF4T4UBNAKG47QHUGF3AA662VQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216134628 (807h9m11.087217663s) [noise=true , address=45.91.116.14:4191 , piece_id=JMZE2QQYMZJLKRLN7DURUWAHXWYRPA4FRQ7H4ITTQUFDIAMKHM5Q, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216232675 (807h9m11.08731572s) [noise=true , address=145.79.227.50:4375 , piece_id=UIYTXVHAKZ5JMBKMLLNIXNLSCGOGZCWFW6UOLDRLLXTV6G4H4SBA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216462912 (807h9m11.087545947s) [noise=true , address=45.91.116.5:4298 , piece_id=RUKJRTACXQ5XW6RIJ4ZP2DJN5GK6FISSBL73VTQ4QHSTI4KGKEGA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216686669 (807h9m11.087769734s) [noise=true , address=145.79.227.33:4871 , piece_id=OYZWFPL5IJB2CEPUWRK3ENTHYB23ZC65GJ7MRGW222JZRXV4Z4MA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216829160 (807h9m11.087912205s) [noise=true , address=45.91.116.6:4976 , piece_id=JUCN3Z7B66IVMMHFE7KPJUMMQCXIQ2I2LZHQURVGV3INEW6BNXGA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.217197411 (807h9m11.088280446s) [noise=true , address=145.79.227.30:4762 , piece_id=FCFXHYISPDWSMLCEN5D64VSGYN3GHCLMBCZCGB3U77OMMYJCBVFQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.217587214 (807h9m11.088670259s) [noise=true , address=145.79.227.28:4727 , piece_id=ELCOLD2FC5EZNIVQEIJUX4UXAPWMP2NGKUHYPYSYREH5Y4KUABBA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.217682535 (807h9m11.08876558s) [noise=true , address=77.90.31.13:4629 , piece_id=VAZ33WQ2GAX63OREDM2UMJHAYM2V3YHXY72DOK2QGKY42FT44XDQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.218057970 (807h9m11.089141015s) [noise=true , address=145.79.227.45:4230 , piece_id=TIY5UNNH7LQCR4M4TIBI55WDYAAOUY45GGBF5CYAFBETAWH6RXBA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:12.906151024 (807h9m12.777234069s) [upload=705993]: Next returned nil segment Jan 8 06:38:12.906163417 (807h9m12.777246462s) [upload=705993]: Waiting for error group managing segments... Jan 8 06:38:13.191636160 (807h9m13.062719195s) [noise=true , address=145.79.227.46:4602 , piece_id=4DLQBWUICSVM7CDVEDJQJRVHOLA62VDZV55ORHOFUXJKZXNODWRQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.198526130 (807h9m13.069609175s) [noise=true , address=145.79.227.49:4338 , piece_id=BA6QWKQJDAFRXCNQQTEPPTUDOUQRTOV6LJGKUI6PDG54IGIS44KQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.220694191 (807h9m13.091777236s) [noise=true , address=145.79.227.15:4875 , piece_id=66NSAERWEKNI5HJR22SKYZ3UT4ZGIVPHX4GNC7OCWNWUHHYOXQSA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.223448024 (807h9m13.09453107s) [noise=true , address=145.79.227.28:4727 , piece_id=ELCOLD2FC5EZNIVQEIJUX4UXAPWMP2NGKUHYPYSYREH5Y4KUABBA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.226749842 (807h9m13.097832877s) [noise=true , address=145.79.227.45:4230 , piece_id=TIY5UNNH7LQCR4M4TIBI55WDYAAOUY45GGBF5CYAFBETAWH6RXBA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.240985449 (807h9m13.112068484s) [noise=true , address=145.79.227.47:4267 , piece_id=MAPONRBWD2MPQFXIGOXQ5T4QVUKE726LLXUBWA2WAZOOWSTALAJA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.305163744 (807h9m13.176246779s) [noise=true , address=145.79.227.50:4375 , piece_id=UIYTXVHAKZ5JMBKMLLNIXNLSCGOGZCWFW6UOLDRLLXTV6G4H4SBA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.452825885 (807h9m13.32390893s) [noise=true , address=145.79.227.33:4871 , piece_id=OYZWFPL5IJB2CEPUWRK3ENTHYB23ZC65GJ7MRGW222JZRXV4Z4MA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.477222951 (807h9m13.348305996s) [noise=true , address=77.90.31.13:4629 , piece_id=VAZ33WQ2GAX63OREDM2UMJHAYM2V3YHXY72DOK2QGKY42FT44XDQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.487139634 (807h9m13.358222669s) [noise=true , address=77.90.31.16:4736 , piece_id=3OF7GH4SQ2OXWT4AMSH6U542D4RF4T4UBNAKG47QHUGF3AA662VQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.556463607 (807h9m13.427546652s) [noise=true , address=145.79.227.30:4762 , piece_id=FCFXHYISPDWSMLCEN5D64VSGYN3GHCLMBCZCGB3U77OMMYJCBVFQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.627164652 (807h9m13.498247697s) [noise=true , address=45.91.116.14:4191 , piece_id=JMZE2QQYMZJLKRLN7DURUWAHXWYRPA4FRQ7H4ITTQUFDIAMKHM5Q, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.640632367 (807h9m13.511715412s) [noise=true , address=45.91.116.9:4262 , piece_id=HVF4HNEVWBLOPEX6LMJNZZDYRZOLTT3NGFMIGMS46S3TVHDDLNDA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.640653266 (807h9m13.511736301s) [seg_pos={0 0}, upload=705993]: Segment reached optimal threshold of 13 pieces. Jan 8 06:38:13.640904686 (807h9m13.511987721s) [noise=true , address=45.91.116.6:4976 , piece_id=JUCN3Z7B66IVMMHFE7KPJUMMQCXIQ2I2LZHQURVGV3INEW6BNXGA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 8 06:38:13.640913031 (807h9m13.511996067s) [noise=true , address=45.91.116.6:4976 , piece_id=JUCN3Z7B66IVMMHFE7KPJUMMQCXIQ2I2LZHQURVGV3INEW6BNXGA, seg_pos={0 0}, upload=705993]: Long tail cancellation due to optimal threshold Jan 8 06:38:13.640957756 (807h9m13.512040801s) [noise=true , address=45.91.116.10:4082 , piece_id=H3UICIRBXOP5XWROFAS2BBDZSKLRHX6S3E4OTGXGNGUHVJHRLPKQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jan 8 06:38:13.640980901 (807h9m13.512063936s) [noise=true , address=45.91.116.10:4082 , piece_id=H3UICIRBXOP5XWROFAS2BBDZSKLRHX6S3E4OTGXGNGUHVJHRLPKQ, seg_pos={0 0}, upload=705993]: Long tail cancellation due to optimal threshold Jan 8 06:38:13.641034412 (807h9m13.512117447s) [noise=true , address=45.91.116.5:4298 , piece_id=RUKJRTACXQ5XW6RIJ4ZP2DJN5GK6FISSBL73VTQ4QHSTI4KGKEGA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jan 8 06:38:13.641040725 (807h9m13.51212377s) [noise=true , address=45.91.116.5:4298 , piece_id=RUKJRTACXQ5XW6RIJ4ZP2DJN5GK6FISSBL73VTQ4QHSTI4KGKEGA, seg_pos={0 0}, upload=705993]: Long tail cancellation due to optimal threshold Jan 8 06:38:13.641058478 (807h9m13.512141513s) [seg_pos={0 0}, upload=705993]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 8 06:38:13.641146847 (807h9m13.512229892s) [upload=705993]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 8 06:38:13.641152558 (807h9m13.512235593s) [upload=705993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 8 06:38:13.924716336 (807h9m13.795799371s) [upload=705993]: Done uploading segments... Jan 8 06:38:13.924735052 (807h9m13.795818097s) [upload=705993]: Upload finished. err: <nil> Duration: 5.793022178s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.......................................X....X....X..........] duration: 7.412415158s trace: https://jaeger.dev.storj.io/trace/23102568ffb9ad71 FAIL (2026-01-08 06:39:27 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 7.412430558s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [..X.X.XXX.X.....X..X.......XXX.XX.XX..X.XXX.XXXX.XXXXXXX..X.] duration: 17.255178885s trace: https://jaeger.dev.storj.io/trace/5184a2ebffa413a5 FAIL (2026-01-08 06:39:24 UTC) orbiter warning: context deadline exceeded Duration: 17.255208232s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.132799066s trace: https://jaeger.dev.storj.io/trace/3738a06b2bfcba74 FAIL (2026-01-08 06:39:23 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12hkSjuH8cdS4byda1bykpe2QJz42HXaK1uRtkTbCQdynovAKSz, Piece ID: 4GTLFPYP5LTIZPMRSOVJNHNBOMLFZTIF7CD5QHSAWAQT2DQYCAEQ) context deadline exceeded; piecestore: (Node ID: 1bWLCSW1tixYzHThBQsUPvXtTpnf4egF8uGHFUXDQkCeE4BwBU, Piece ID: RX5TQUH56ZXKJADRRFT7DX6J6VV32WB3EFRCM456NZ2TD2WHKM3A) context deadline exceeded; piecestore: (Node ID: 1XxiRqM4j35HGGzWPnmrHHfbWS8jryWrW5aY66SgDuDQKHYopv, Piece ID: GAOHJL4R553KJFKTWIRCGC3NHHM4TBFNH6GJYFREM23JQQS645TQ) context deadline exceeded; piecestore: (Node ID: 15LfiPEGogvaJkFN67RXm8egaXMGxsc9XmPo26GFwT5TZKmg75, Piece ID: HD6GDZKL6A43NRG367DI674KCKPZAN4PP6S5ZRHHYR33EPAQ4KRQ) context deadline exceeded; piecestore: (Node ID: 1m2RK4RXMBjqBpWj8Xof42VU2fyPNygiY1WR3gcsVRxd6jKe4E, Piece ID: SCDRYTITO42A42357QKLW7RUWW6VKZLZLYQ3H2NIPXAKYGOIC5HQ) context deadline exceeded; piecestore: (Node ID: 1vtdYMUQ1aEzqZeSWkWD9RceYHunHdt9QwAzy7CwyYruPtSNRw, Piece ID: SYC3VAYM2YKT5ZM2VAQQC7663JAM47LYYCU62ZFRVH5UJDTTXMRQ) context deadline exceeded; piecestore: (Node ID: 1dstfvWPzYyG7oHnsByWcpgv6C4NVw9RPt8MdxtZ634FaLrSt4, Piece ID: D6G2XNNG5QPU2VSWNAP6XPRA3LM5GREUKMQG3KEWTKXFBNSEPIZA) context deadline exceeded; piecestore: (Node ID: 12Bf3H3JPRVT3GKxieKhLpk7ZfXc3FrdtweLJKRpFo6yNTomKDu, Piece ID: UD3L4QJO2DEW7V3SJ7HHPO5P6VWX4MURJKKEQ5KIQUHKAUZOZVTQ) context deadline exceeded; piecestore: (Node ID: 1ZtoenGAVBrgVoJkmkyxDqire258rdYcqC3avm8xAPuBGiD1bU, Piece ID: 4GAT5EPHUEH7UO3IHA5FU23RMY7NGYLJU2CBCX3RS2BCAPIDP7VA) context deadline exceeded; piecestore: (Node ID: 12vhu4j9fCJhbUW9rxZshXEbh9gbG1j1sPzgPhDdiD7ecVTQaHz, Piece ID: 6N3EAXOD5X75DQR6W4AHVJ54XQ7BGOF2BRN32IDBHLLYOOYI3ILQ) context deadline exceeded; piecestore: (Node ID: 1kLgnR5vmDgUuaf1aWAUyDsXuHn8aEwVXEQiruQrkfKpXqTiQ1, Piece ID: DGJRCP2CNO7DMWSJ5XMYFWV6CF22SLA4LPCVWLTBRKCQ6NV7HS7Q) context deadline exceeded Duration: 6.132827269s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X..XX........................................] duration: 5.241955123s trace: https://jaeger.dev.storj.io/trace/233c74a6b9db5cf8 FAIL (2026-01-08 06:39:21 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 53) Duration: 5.241981996s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [..X.XXX........X..X...............X....XXX...XX.X......X..X.] duration: 19.273010732s trace: https://jaeger.dev.storj.io/trace/b037bcc8f577a49 FAIL (2026-01-08 06:39:14 UTC) orbiter warning: context deadline exceeded Duration: 19.273027563s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 3.270231413s trace: https://jaeger.dev.storj.io/trace/6173eb4c5ef6f546 FAIL (2026-01-08 06:39:13 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 53) Duration: 3.27026127s core uplink ap1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [XXXXXX.X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000490248s trace: https://jaeger.dev.storj.io/trace/4b60ebfab6dc0d68 FAIL (2026-01-08 06:39:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000517539s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [........................................X....X....X.........] duration: 20.495542513s trace: https://jaeger.dev.storj.io/trace/23102568ffb9ad71 FAIL (2026-01-08 06:39:02 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 20.495573883s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [....X.............X......XXX..........XX......X...........X.] duration: 6.089645332s trace: https://jaeger.dev.storj.io/trace/11a85404b66ec466 FAIL (2026-01-08 06:38:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: JMZE2QQYMZJLKRLN7DURUWAHXWYRPA4FRQ7H4ITTQUFDIAMKHM5Q) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: HVF4HNEVWBLOPEX6LMJNZZDYRZOLTT3NGFMIGMS46S3TVHDDLNDA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 3OF7GH4SQ2OXWT4AMSH6U542D4RF4T4UBNAKG47QHUGF3AA662VQ) context deadline exceeded ======LOGS====== Jan 8 06:38:10.969187554 (807h9m10.840270599s) [upload=705993]: Starting upload Jan 8 06:38:10.969275200 (807h9m10.840358245s) [upload=705993]: Done starting upload Jan 8 06:38:10.969338862 (807h9m10.840421907s) [upload=705993]: Uploading segments... Jan 8 06:38:10.969657759 (807h9m10.840740804s) [upload=705993]: Got next segment. Inline: false Jan 8 06:38:10.969705860 (807h9m10.840788896s) [upload=705993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 8 06:38:10.969712884 (807h9m10.840795929s) [upload=705993]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 8 06:38:11.214842718 (807h9m11.085925753s) [seg_pos={0 0}, upload=705993]: Begin upload segment... Jan 8 06:38:11.214885970 (807h9m11.085969005s) [seg_pos={0 0}, upload=705993]: Done begin upload segment. Jan 8 06:38:11.214913212 (807h9m11.085996247s) [noise=true , address=45.91.116.9:4262 , piece_id=HVF4HNEVWBLOPEX6LMJNZZDYRZOLTT3NGFMIGMS46S3TVHDDLNDA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.214935674 (807h9m11.086018709s) [noise=true , address=145.79.227.15:4875 , piece_id=66NSAERWEKNI5HJR22SKYZ3UT4ZGIVPHX4GNC7OCWNWUHHYOXQSA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215307472 (807h9m11.086390518s) [noise=true , address=145.79.227.49:4338 , piece_id=BA6QWKQJDAFRXCNQQTEPPTUDOUQRTOV6LJGKUI6PDG54IGIS44KQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215445766 (807h9m11.086528801s) [noise=true , address=145.79.227.47:4267 , piece_id=MAPONRBWD2MPQFXIGOXQ5T4QVUKE726LLXUBWA2WAZOOWSTALAJA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215800942 (807h9m11.086883977s) [noise=true , address=45.91.116.10:4082 , piece_id=H3UICIRBXOP5XWROFAS2BBDZSKLRHX6S3E4OTGXGNGUHVJHRLPKQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215815641 (807h9m11.086898686s) [noise=true , address=145.79.227.46:4602 , piece_id=4DLQBWUICSVM7CDVEDJQJRVHOLA62VDZV55ORHOFUXJKZXNODWRQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215868070 (807h9m11.086951115s) [noise=true , address=77.90.31.16:4736 , piece_id=3OF7GH4SQ2OXWT4AMSH6U542D4RF4T4UBNAKG47QHUGF3AA662VQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216134628 (807h9m11.087217663s) [noise=true , address=45.91.116.14:4191 , piece_id=JMZE2QQYMZJLKRLN7DURUWAHXWYRPA4FRQ7H4ITTQUFDIAMKHM5Q, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216232675 (807h9m11.08731572s) [noise=true , address=145.79.227.50:4375 , piece_id=UIYTXVHAKZ5JMBKMLLNIXNLSCGOGZCWFW6UOLDRLLXTV6G4H4SBA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216462912 (807h9m11.087545947s) [noise=true , address=45.91.116.5:4298 , piece_id=RUKJRTACXQ5XW6RIJ4ZP2DJN5GK6FISSBL73VTQ4QHSTI4KGKEGA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216686669 (807h9m11.087769734s) [noise=true , address=145.79.227.33:4871 , piece_id=OYZWFPL5IJB2CEPUWRK3ENTHYB23ZC65GJ7MRGW222JZRXV4Z4MA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216829160 (807h9m11.087912205s) [noise=true , address=45.91.116.6:4976 , piece_id=JUCN3Z7B66IVMMHFE7KPJUMMQCXIQ2I2LZHQURVGV3INEW6BNXGA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.217197411 (807h9m11.088280446s) [noise=true , address=145.79.227.30:4762 , piece_id=FCFXHYISPDWSMLCEN5D64VSGYN3GHCLMBCZCGB3U77OMMYJCBVFQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.217587214 (807h9m11.088670259s) [noise=true , address=145.79.227.28:4727 , piece_id=ELCOLD2FC5EZNIVQEIJUX4UXAPWMP2NGKUHYPYSYREH5Y4KUABBA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.217682535 (807h9m11.08876558s) [noise=true , address=77.90.31.13:4629 , piece_id=VAZ33WQ2GAX63OREDM2UMJHAYM2V3YHXY72DOK2QGKY42FT44XDQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.218057970 (807h9m11.089141015s) [noise=true , address=145.79.227.45:4230 , piece_id=TIY5UNNH7LQCR4M4TIBI55WDYAAOUY45GGBF5CYAFBETAWH6RXBA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:12.906151024 (807h9m12.777234069s) [upload=705993]: Next returned nil segment Jan 8 06:38:12.906163417 (807h9m12.777246462s) [upload=705993]: Waiting for error group managing segments... Jan 8 06:38:13.191636160 (807h9m13.062719195s) [noise=true , address=145.79.227.46:4602 , piece_id=4DLQBWUICSVM7CDVEDJQJRVHOLA62VDZV55ORHOFUXJKZXNODWRQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.198526130 (807h9m13.069609175s) [noise=true , address=145.79.227.49:4338 , piece_id=BA6QWKQJDAFRXCNQQTEPPTUDOUQRTOV6LJGKUI6PDG54IGIS44KQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.220694191 (807h9m13.091777236s) [noise=true , address=145.79.227.15:4875 , piece_id=66NSAERWEKNI5HJR22SKYZ3UT4ZGIVPHX4GNC7OCWNWUHHYOXQSA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.223448024 (807h9m13.09453107s) [noise=true , address=145.79.227.28:4727 , piece_id=ELCOLD2FC5EZNIVQEIJUX4UXAPWMP2NGKUHYPYSYREH5Y4KUABBA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.226749842 (807h9m13.097832877s) [noise=true , address=145.79.227.45:4230 , piece_id=TIY5UNNH7LQCR4M4TIBI55WDYAAOUY45GGBF5CYAFBETAWH6RXBA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.240985449 (807h9m13.112068484s) [noise=true , address=145.79.227.47:4267 , piece_id=MAPONRBWD2MPQFXIGOXQ5T4QVUKE726LLXUBWA2WAZOOWSTALAJA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.305163744 (807h9m13.176246779s) [noise=true , address=145.79.227.50:4375 , piece_id=UIYTXVHAKZ5JMBKMLLNIXNLSCGOGZCWFW6UOLDRLLXTV6G4H4SBA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.452825885 (807h9m13.32390893s) [noise=true , address=145.79.227.33:4871 , piece_id=OYZWFPL5IJB2CEPUWRK3ENTHYB23ZC65GJ7MRGW222JZRXV4Z4MA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.477222951 (807h9m13.348305996s) [noise=true , address=77.90.31.13:4629 , piece_id=VAZ33WQ2GAX63OREDM2UMJHAYM2V3YHXY72DOK2QGKY42FT44XDQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.487139634 (807h9m13.358222669s) [noise=true , address=77.90.31.16:4736 , piece_id=3OF7GH4SQ2OXWT4AMSH6U542D4RF4T4UBNAKG47QHUGF3AA662VQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.556463607 (807h9m13.427546652s) [noise=true , address=145.79.227.30:4762 , piece_id=FCFXHYISPDWSMLCEN5D64VSGYN3GHCLMBCZCGB3U77OMMYJCBVFQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.627164652 (807h9m13.498247697s) [noise=true , address=45.91.116.14:4191 , piece_id=JMZE2QQYMZJLKRLN7DURUWAHXWYRPA4FRQ7H4ITTQUFDIAMKHM5Q, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.640632367 (807h9m13.511715412s) [noise=true , address=45.91.116.9:4262 , piece_id=HVF4HNEVWBLOPEX6LMJNZZDYRZOLTT3NGFMIGMS46S3TVHDDLNDA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.640653266 (807h9m13.511736301s) [seg_pos={0 0}, upload=705993]: Segment reached optimal threshold of 13 pieces. Jan 8 06:38:13.640904686 (807h9m13.511987721s) [noise=true , address=45.91.116.6:4976 , piece_id=JUCN3Z7B66IVMMHFE7KPJUMMQCXIQ2I2LZHQURVGV3INEW6BNXGA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 8 06:38:13.640913031 (807h9m13.511996067s) [noise=true , address=45.91.116.6:4976 , piece_id=JUCN3Z7B66IVMMHFE7KPJUMMQCXIQ2I2LZHQURVGV3INEW6BNXGA, seg_pos={0 0}, upload=705993]: Long tail cancellation due to optimal threshold Jan 8 06:38:13.640957756 (807h9m13.512040801s) [noise=true , address=45.91.116.10:4082 , piece_id=H3UICIRBXOP5XWROFAS2BBDZSKLRHX6S3E4OTGXGNGUHVJHRLPKQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jan 8 06:38:13.640980901 (807h9m13.512063936s) [noise=true , address=45.91.116.10:4082 , piece_id=H3UICIRBXOP5XWROFAS2BBDZSKLRHX6S3E4OTGXGNGUHVJHRLPKQ, seg_pos={0 0}, upload=705993]: Long tail cancellation due to optimal threshold Jan 8 06:38:13.641034412 (807h9m13.512117447s) [noise=true , address=45.91.116.5:4298 , piece_id=RUKJRTACXQ5XW6RIJ4ZP2DJN5GK6FISSBL73VTQ4QHSTI4KGKEGA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jan 8 06:38:13.641040725 (807h9m13.51212377s) [noise=true , address=45.91.116.5:4298 , piece_id=RUKJRTACXQ5XW6RIJ4ZP2DJN5GK6FISSBL73VTQ4QHSTI4KGKEGA, seg_pos={0 0}, upload=705993]: Long tail cancellation due to optimal threshold Jan 8 06:38:13.641058478 (807h9m13.512141513s) [seg_pos={0 0}, upload=705993]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 8 06:38:13.641146847 (807h9m13.512229892s) [upload=705993]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 8 06:38:13.641152558 (807h9m13.512235593s) [upload=705993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 8 06:38:13.924716336 (807h9m13.795799371s) [upload=705993]: Done uploading segments... Jan 8 06:38:13.924735052 (807h9m13.795818097s) [upload=705993]: Upload finished. err: <nil> Duration: 6.089671383s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 9 out of 60 is failed (threshold=3) [X..................XXXXXXXX.................................] duration: 6.398172384s trace: https://jaeger.dev.storj.io/trace/635cbd49e9d724cb FAIL (2026-01-08 06:38:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 21) ======LOGS====== _id=TM5C7SQQA5IAVFWRG2GV2WYRB2IRLWDLVB3MFWK7XILSBDUDIAVQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.977864602 (806h43m10.848947647s) [noise=true , address=185.167.204.206:4059 , piece_id=FKQBFAG3DNQ5OI5DJ7GAZINSWVSVDZED56SJVHBY2ULJO5HUAGNQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.978752623 (806h43m10.849835658s) [noise=true , address=185.167.204.206:4064 , piece_id=3FMVMJWMWGAHNVQX2AIBVII5Z6XTPJUCYABKNXJ4NWRC2LQ6I3FA, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.979487392 (806h43m10.850570427s) [noise=true , address=89.149.201.147:4020 , piece_id=U3G7AY7GMBA527UQXXY6DOLNKJBUR4CZD7T3MOJSQNFFGYJXKR3A, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.980303676 (806h43m10.851386721s) [noise=true , address=185.167.204.206:4061 , piece_id=LJVZ2GBV4S5T6XXBYDEELH6BKTBY2TTOBBXWNNNH2VSE2W5RI5YA, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.980907016 (806h43m10.851990051s) [noise=true , address=89.149.201.147:4011 , piece_id=PXK522IVCJHJDLWXUIP3PQF7PJXVPXLPGM6HE43CY6G2VSLOAWPQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.981714563 (806h43m10.852797608s) [noise=true , address=81.171.14.70:4043 , piece_id=AM727IICNSFNSVCQ3IZ55CAZK6YH3JUFBOJAUHMMYRJ6NEZZREDQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.982411280 (806h43m10.853494325s) [noise=true , address=89.149.201.147:4004 , piece_id=BRZPEKISSTSV7IPNBW7ZJJIIGOTR4SQDJDMQZ3OXOSUEQJZB5CMQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.983960189 (806h43m10.855043224s) [noise=true , address=81.171.14.70:4048 , piece_id=HLP5YVTQMQXLGROPTZJ4RPKFYCCBB3W5ZPHASDVMTJ6ZCTSGMDKQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:11.816868763 (806h43m11.687951808s) [upload=705619]: Next returned nil segment Jan 8 06:12:11.816884092 (806h43m11.687967137s) [upload=705619]: Waiting for error group managing segments... Jan 8 06:12:12.207503101 (806h43m12.078586136s) [noise=true , address=185.167.204.205:4014 , piece_id=42DLCGFRUY5XHUEJPKO6E6BKJSWTUC5IDYRSAE3OXFFVCYA2E67A, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.207636776 (806h43m12.078719821s) [noise=true , address=185.167.204.205:4011 , piece_id=36G2RTRXP6WEHPMQKMEQ3334GYKN6RFC7VYMW6BSCLBQ37OT43YQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.208187303 (806h43m12.079270338s) [noise=true , address=185.167.204.206:4041 , piece_id=CDCLTZOQNX4QLRZ5JT7ZIBQKKBRWVX3SWJSKC46EAIMPSV2SYONA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.208277846 (806h43m12.079360881s) [noise=true , address=185.167.204.205:4003 , piece_id=EGXKQAKMGKB3SDIEFFEVFE6XYGHZ4ZR6A5B6MFDB7XFY7KID4QYA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.211357060 (806h43m12.082440095s) [noise=true , address=185.167.204.206:4062 , piece_id=MJLGFBLTJEG7UHJMFLI72MHHH6FWFLP3O4VNTUAS5Z4KCH27EHOQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.215506782 (806h43m12.086589817s) [noise=true , address=185.167.204.206:4068 , piece_id=L6DRFJFP5OFG2UEGHUGNCRWVGJ7XQFGJJUXNT7WGSEJXS5TMKNRQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.221151259 (806h43m12.092234294s) [noise=true , address=185.167.204.206:4059 , piece_id=FKQBFAG3DNQ5OI5DJ7GAZINSWVSVDZED56SJVHBY2ULJO5HUAGNQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.247553292 (806h43m12.118636337s) [noise=true , address=185.167.204.206:4064 , piece_id=3FMVMJWMWGAHNVQX2AIBVII5Z6XTPJUCYABKNXJ4NWRC2LQ6I3FA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.268061592 (806h43m12.139144637s) [noise=true , address=185.167.204.206:4061 , piece_id=LJVZ2GBV4S5T6XXBYDEELH6BKTBY2TTOBBXWNNNH2VSE2W5RI5YA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.268319703 (806h43m12.139402739s) [noise=true , address=185.167.204.205:4017 , piece_id=ACCHFJNRCMWJWXPZJDXALXSUVPLETHOSDAGKPKDAKRMWJ2JI6RMQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.270984638 (806h43m12.142067684s) [noise=true , address=185.167.204.206:4037 , piece_id=HAKQW2RQFJJGBIG26EV4B5TJJBZRM2YMXVU5PZQIKNA5YYEAZVHA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.272396927 (806h43m12.143479962s) [noise=true , address=185.167.204.205:4034 , piece_id=EMHALXT4VKLMJNTQV4XMRGLVOPBRF7O45YKF5Q3RYUBF6DGHPTJQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:13.719906169 (806h43m13.590989215s) [noise=true , address=89.149.201.147:4007 , piece_id=XDKNFX56PLHN6AXAKPM3LSRUHAFHIOZIJF2CAFXOMDLUH7URSS3Q, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:15.774082352 (806h43m15.645165397s) [noise=true , address=89.149.201.147:4005 , piece_id=3WEJWRZJOWY6I7ZNNWHK5QYTF5MULZNKHWVZMBEJJPFNB6WU4UCA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:16.530239032 (806h43m16.401322067s) [noise=true , address=89.149.201.147:4001 , piece_id=LWDR4KCC5QHQDLFVZYVTYJH26GRABV2JFC3525OVIORRG2SRHQMA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:16.928628457 (806h43m16.799711492s) [noise=true , address=81.171.14.70:4041 , piece_id=J3IG4LCWN7UM2NCQAY4FKTPNYH5NRW7TDB6ZGMNPQEI7HY5CZGRQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:17.995225112 (806h43m17.866308157s) [noise=true , address=89.149.201.147:4011 , piece_id=PXK522IVCJHJDLWXUIP3PQF7PJXVPXLPGM6HE43CY6G2VSLOAWPQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:18.045551680 (806h43m17.916634725s) [noise=true , address=81.171.14.70:4027 , piece_id=5P2A7PSIMWZC2I6SAGUEUEJDDW67BD7AKQ34Z3XZHFPJGSHLJLVQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:19.316134332 (806h43m19.187217367s) [noise=true , address=89.149.201.147:4004 , piece_id=BRZPEKISSTSV7IPNBW7ZJJIIGOTR4SQDJDMQZ3OXOSUEQJZB5CMQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:19.607969825 (806h43m19.47905288s) [noise=true , address=81.171.14.70:4035 , piece_id=HAUGVFLDMZCSFVPSOYLS4PBCVHMMMOTPD4ODSDCS3P4NSTX6KPJQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:13:00.649061845 (806h44m0.52014488s) [noise=true , address=89.149.201.147:4010 , piece_id=TM5C7SQQA5IAVFWRG2GV2WYRB2IRLWDLVB3MFWK7XILSBDUDIAVQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:13:10.291882458 (806h44m10.162965513s) [noise=true , address=89.149.201.147:4015 , piece_id=GLHXEDCIEKGM2SPTCNCTJYS2GIEUMIT5LQXCHMDTVNNIY5KLASVQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K, address:89.149.201.147:4015): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.291987709 (806h44m10.163070774s) [noise=true , address=81.171.14.70:4034 , piece_id=VGWIA5ZBLFNA33QXVLF54XTCOCAHSU6TVV3O52DTZ2L3ROIPHNAA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12jcAHt22bETT27CCXJ8ZaaYxxoKBonBYsFcoq6A34DVVjvhQAN, address:81.171.14.70:4034): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.292023307 (806h44m10.163106342s) [noise=true , address=81.171.14.70:4038 , piece_id=QOQ3IYLTGQ27HOLQKQZOOILZXWAPR6PEHLQTHLGLBLXLLQDBA5RA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB, address:81.171.14.70:4038): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.292120161 (806h44m10.163203216s) [noise=true , address=89.149.201.147:4020 , piece_id=U3G7AY7GMBA527UQXXY6DOLNKJBUR4CZD7T3MOJSQNFFGYJXKR3A, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.292271189 (806h44m10.163354245s) [noise=true , address=81.171.14.70:4048 , piece_id=HLP5YVTQMQXLGROPTZJ4RPKFYCCBB3W5ZPHASDVMTJ6ZCTSGMDKQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: read tcp 188.166.225.170:40684->81.171.14.70:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 8 06:13:10.292450230 (806h44m10.163533285s) [noise=true , address=81.171.14.70:4043 , piece_id=AM727IICNSFNSVCQ3IZ55CAZK6YH3JUFBOJAUHMMYRJ6NEZZREDQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko, address:81.171.14.70:4043): protocol: read tcp 188.166.225.170:38496->81.171.14.70:4043: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 8 06:13:10.292509182 (806h44m10.163592237s) [noise=true , address=81.171.14.70:4029 , piece_id=6UO7K2VEO7HMZPJM4S34Z6Z3UWKFMLD3BLATMSIRL4WDIXKVTZCA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:60086->81.171.14.70:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 8 06:13:10.292533999 (806h44m10.163617034s) [seg_pos={0 0}, upload=705619]: Done waiting for segment. successful: 21 optimal: 26 errs: <nil> Jan 8 06:13:10.462867035 (806h44m10.33395007s) [upload=705619]: Done uploading segments... Jan 8 06:13:10.462901039 (806h44m10.333984085s) [upload=705619]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 21) Duration: 6.398195818s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [...X.XXX........X..X...............X....XXX...XX.X......X..X] duration: 1m0.001767911s trace: https://jaeger.dev.storj.io/trace/b037bcc8f577a49 FAIL (2026-01-08 06:38:55 UTC) orbiter warning: context deadline exceeded Duration: 1m0.001807687s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X..XX.......................................] duration: 5.062797468s trace: https://jaeger.dev.storj.io/trace/233c74a6b9db5cf8 FAIL (2026-01-08 06:38:50 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 53) Duration: 5.062824551s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....X.............X......XXX..........XX......X...........X] duration: 30.002869742s trace: https://jaeger.dev.storj.io/trace/11a85404b66ec466 FAIL (2026-01-08 06:38:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: JMZE2QQYMZJLKRLN7DURUWAHXWYRPA4FRQ7H4ITTQUFDIAMKHM5Q) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: HVF4HNEVWBLOPEX6LMJNZZDYRZOLTT3NGFMIGMS46S3TVHDDLNDA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: 3OF7GH4SQ2OXWT4AMSH6U542D4RF4T4UBNAKG47QHUGF3AA662VQ) context deadline exceeded ======LOGS====== Jan 8 06:38:10.969187554 (807h9m10.840270599s) [upload=705993]: Starting upload Jan 8 06:38:10.969275200 (807h9m10.840358245s) [upload=705993]: Done starting upload Jan 8 06:38:10.969338862 (807h9m10.840421907s) [upload=705993]: Uploading segments... Jan 8 06:38:10.969657759 (807h9m10.840740804s) [upload=705993]: Got next segment. Inline: false Jan 8 06:38:10.969705860 (807h9m10.840788896s) [upload=705993]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 8 06:38:10.969712884 (807h9m10.840795929s) [upload=705993]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 8 06:38:11.214842718 (807h9m11.085925753s) [seg_pos={0 0}, upload=705993]: Begin upload segment... Jan 8 06:38:11.214885970 (807h9m11.085969005s) [seg_pos={0 0}, upload=705993]: Done begin upload segment. Jan 8 06:38:11.214913212 (807h9m11.085996247s) [noise=true , address=45.91.116.9:4262 , piece_id=HVF4HNEVWBLOPEX6LMJNZZDYRZOLTT3NGFMIGMS46S3TVHDDLNDA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.214935674 (807h9m11.086018709s) [noise=true , address=145.79.227.15:4875 , piece_id=66NSAERWEKNI5HJR22SKYZ3UT4ZGIVPHX4GNC7OCWNWUHHYOXQSA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215307472 (807h9m11.086390518s) [noise=true , address=145.79.227.49:4338 , piece_id=BA6QWKQJDAFRXCNQQTEPPTUDOUQRTOV6LJGKUI6PDG54IGIS44KQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215445766 (807h9m11.086528801s) [noise=true , address=145.79.227.47:4267 , piece_id=MAPONRBWD2MPQFXIGOXQ5T4QVUKE726LLXUBWA2WAZOOWSTALAJA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215800942 (807h9m11.086883977s) [noise=true , address=45.91.116.10:4082 , piece_id=H3UICIRBXOP5XWROFAS2BBDZSKLRHX6S3E4OTGXGNGUHVJHRLPKQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215815641 (807h9m11.086898686s) [noise=true , address=145.79.227.46:4602 , piece_id=4DLQBWUICSVM7CDVEDJQJRVHOLA62VDZV55ORHOFUXJKZXNODWRQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.215868070 (807h9m11.086951115s) [noise=true , address=77.90.31.16:4736 , piece_id=3OF7GH4SQ2OXWT4AMSH6U542D4RF4T4UBNAKG47QHUGF3AA662VQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216134628 (807h9m11.087217663s) [noise=true , address=45.91.116.14:4191 , piece_id=JMZE2QQYMZJLKRLN7DURUWAHXWYRPA4FRQ7H4ITTQUFDIAMKHM5Q, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216232675 (807h9m11.08731572s) [noise=true , address=145.79.227.50:4375 , piece_id=UIYTXVHAKZ5JMBKMLLNIXNLSCGOGZCWFW6UOLDRLLXTV6G4H4SBA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216462912 (807h9m11.087545947s) [noise=true , address=45.91.116.5:4298 , piece_id=RUKJRTACXQ5XW6RIJ4ZP2DJN5GK6FISSBL73VTQ4QHSTI4KGKEGA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216686669 (807h9m11.087769734s) [noise=true , address=145.79.227.33:4871 , piece_id=OYZWFPL5IJB2CEPUWRK3ENTHYB23ZC65GJ7MRGW222JZRXV4Z4MA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.216829160 (807h9m11.087912205s) [noise=true , address=45.91.116.6:4976 , piece_id=JUCN3Z7B66IVMMHFE7KPJUMMQCXIQ2I2LZHQURVGV3INEW6BNXGA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.217197411 (807h9m11.088280446s) [noise=true , address=145.79.227.30:4762 , piece_id=FCFXHYISPDWSMLCEN5D64VSGYN3GHCLMBCZCGB3U77OMMYJCBVFQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.217587214 (807h9m11.088670259s) [noise=true , address=145.79.227.28:4727 , piece_id=ELCOLD2FC5EZNIVQEIJUX4UXAPWMP2NGKUHYPYSYREH5Y4KUABBA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.217682535 (807h9m11.08876558s) [noise=true , address=77.90.31.13:4629 , piece_id=VAZ33WQ2GAX63OREDM2UMJHAYM2V3YHXY72DOK2QGKY42FT44XDQ, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:11.218057970 (807h9m11.089141015s) [noise=true , address=145.79.227.45:4230 , piece_id=TIY5UNNH7LQCR4M4TIBI55WDYAAOUY45GGBF5CYAFBETAWH6RXBA, seg_pos={0 0}, upload=705993]: Uploading piece... Jan 8 06:38:12.906151024 (807h9m12.777234069s) [upload=705993]: Next returned nil segment Jan 8 06:38:12.906163417 (807h9m12.777246462s) [upload=705993]: Waiting for error group managing segments... Jan 8 06:38:13.191636160 (807h9m13.062719195s) [noise=true , address=145.79.227.46:4602 , piece_id=4DLQBWUICSVM7CDVEDJQJRVHOLA62VDZV55ORHOFUXJKZXNODWRQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.198526130 (807h9m13.069609175s) [noise=true , address=145.79.227.49:4338 , piece_id=BA6QWKQJDAFRXCNQQTEPPTUDOUQRTOV6LJGKUI6PDG54IGIS44KQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.220694191 (807h9m13.091777236s) [noise=true , address=145.79.227.15:4875 , piece_id=66NSAERWEKNI5HJR22SKYZ3UT4ZGIVPHX4GNC7OCWNWUHHYOXQSA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.223448024 (807h9m13.09453107s) [noise=true , address=145.79.227.28:4727 , piece_id=ELCOLD2FC5EZNIVQEIJUX4UXAPWMP2NGKUHYPYSYREH5Y4KUABBA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.226749842 (807h9m13.097832877s) [noise=true , address=145.79.227.45:4230 , piece_id=TIY5UNNH7LQCR4M4TIBI55WDYAAOUY45GGBF5CYAFBETAWH6RXBA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.240985449 (807h9m13.112068484s) [noise=true , address=145.79.227.47:4267 , piece_id=MAPONRBWD2MPQFXIGOXQ5T4QVUKE726LLXUBWA2WAZOOWSTALAJA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.305163744 (807h9m13.176246779s) [noise=true , address=145.79.227.50:4375 , piece_id=UIYTXVHAKZ5JMBKMLLNIXNLSCGOGZCWFW6UOLDRLLXTV6G4H4SBA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.452825885 (807h9m13.32390893s) [noise=true , address=145.79.227.33:4871 , piece_id=OYZWFPL5IJB2CEPUWRK3ENTHYB23ZC65GJ7MRGW222JZRXV4Z4MA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.477222951 (807h9m13.348305996s) [noise=true , address=77.90.31.13:4629 , piece_id=VAZ33WQ2GAX63OREDM2UMJHAYM2V3YHXY72DOK2QGKY42FT44XDQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.487139634 (807h9m13.358222669s) [noise=true , address=77.90.31.16:4736 , piece_id=3OF7GH4SQ2OXWT4AMSH6U542D4RF4T4UBNAKG47QHUGF3AA662VQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.556463607 (807h9m13.427546652s) [noise=true , address=145.79.227.30:4762 , piece_id=FCFXHYISPDWSMLCEN5D64VSGYN3GHCLMBCZCGB3U77OMMYJCBVFQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.627164652 (807h9m13.498247697s) [noise=true , address=45.91.116.14:4191 , piece_id=JMZE2QQYMZJLKRLN7DURUWAHXWYRPA4FRQ7H4ITTQUFDIAMKHM5Q, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.640632367 (807h9m13.511715412s) [noise=true , address=45.91.116.9:4262 , piece_id=HVF4HNEVWBLOPEX6LMJNZZDYRZOLTT3NGFMIGMS46S3TVHDDLNDA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: <nil> Jan 8 06:38:13.640653266 (807h9m13.511736301s) [seg_pos={0 0}, upload=705993]: Segment reached optimal threshold of 13 pieces. Jan 8 06:38:13.640904686 (807h9m13.511987721s) [noise=true , address=45.91.116.6:4976 , piece_id=JUCN3Z7B66IVMMHFE7KPJUMMQCXIQ2I2LZHQURVGV3INEW6BNXGA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 8 06:38:13.640913031 (807h9m13.511996067s) [noise=true , address=45.91.116.6:4976 , piece_id=JUCN3Z7B66IVMMHFE7KPJUMMQCXIQ2I2LZHQURVGV3INEW6BNXGA, seg_pos={0 0}, upload=705993]: Long tail cancellation due to optimal threshold Jan 8 06:38:13.640957756 (807h9m13.512040801s) [noise=true , address=45.91.116.10:4082 , piece_id=H3UICIRBXOP5XWROFAS2BBDZSKLRHX6S3E4OTGXGNGUHVJHRLPKQ, seg_pos={0 0}, upload=705993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jan 8 06:38:13.640980901 (807h9m13.512063936s) [noise=true , address=45.91.116.10:4082 , piece_id=H3UICIRBXOP5XWROFAS2BBDZSKLRHX6S3E4OTGXGNGUHVJHRLPKQ, seg_pos={0 0}, upload=705993]: Long tail cancellation due to optimal threshold Jan 8 06:38:13.641034412 (807h9m13.512117447s) [noise=true , address=45.91.116.5:4298 , piece_id=RUKJRTACXQ5XW6RIJ4ZP2DJN5GK6FISSBL73VTQ4QHSTI4KGKEGA, seg_pos={0 0}, upload=705993]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Jan 8 06:38:13.641040725 (807h9m13.51212377s) [noise=true , address=45.91.116.5:4298 , piece_id=RUKJRTACXQ5XW6RIJ4ZP2DJN5GK6FISSBL73VTQ4QHSTI4KGKEGA, seg_pos={0 0}, upload=705993]: Long tail cancellation due to optimal threshold Jan 8 06:38:13.641058478 (807h9m13.512141513s) [seg_pos={0 0}, upload=705993]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 8 06:38:13.641146847 (807h9m13.512229892s) [upload=705993]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 8 06:38:13.641152558 (807h9m13.512235593s) [upload=705993]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 8 06:38:13.924716336 (807h9m13.795799371s) [upload=705993]: Done uploading segments... Jan 8 06:38:13.924735052 (807h9m13.795818097s) [upload=705993]: Upload finished. err: <nil> Duration: 30.002891253s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [...X.X.XXX.X.....X..X.......XXX.XX.XX..X.XXX.XXXX.XXXXXXX..X] duration: 30.001528739s trace: https://jaeger.dev.storj.io/trace/5184a2ebffa413a5 FAIL (2026-01-08 06:38:31 UTC) orbiter warning: context deadline exceeded Duration: 30.001543246s edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 3.381650031s trace: https://jaeger.dev.storj.io/trace/6173eb4c5ef6f546 FAIL (2026-01-08 06:38:17 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 53) Duration: 3.381677395s core uplink ap1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [XXXXXXX.X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001780313s trace: https://jaeger.dev.storj.io/trace/758ca9122f4063c6 FAIL (2026-01-08 06:38:14 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00180481s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.138075438s trace: https://jaeger.dev.storj.io/trace/3738a06b2bfcba74 FAIL (2026-01-08 06:38:13 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12hkSjuH8cdS4byda1bykpe2QJz42HXaK1uRtkTbCQdynovAKSz, Piece ID: 4GTLFPYP5LTIZPMRSOVJNHNBOMLFZTIF7CD5QHSAWAQT2DQYCAEQ) context deadline exceeded; piecestore: (Node ID: 1bWLCSW1tixYzHThBQsUPvXtTpnf4egF8uGHFUXDQkCeE4BwBU, Piece ID: RX5TQUH56ZXKJADRRFT7DX6J6VV32WB3EFRCM456NZ2TD2WHKM3A) context deadline exceeded; piecestore: (Node ID: 1XxiRqM4j35HGGzWPnmrHHfbWS8jryWrW5aY66SgDuDQKHYopv, Piece ID: GAOHJL4R553KJFKTWIRCGC3NHHM4TBFNH6GJYFREM23JQQS645TQ) context deadline exceeded; piecestore: (Node ID: 15LfiPEGogvaJkFN67RXm8egaXMGxsc9XmPo26GFwT5TZKmg75, Piece ID: HD6GDZKL6A43NRG367DI674KCKPZAN4PP6S5ZRHHYR33EPAQ4KRQ) context deadline exceeded; piecestore: (Node ID: 1m2RK4RXMBjqBpWj8Xof42VU2fyPNygiY1WR3gcsVRxd6jKe4E, Piece ID: SCDRYTITO42A42357QKLW7RUWW6VKZLZLYQ3H2NIPXAKYGOIC5HQ) context deadline exceeded; piecestore: (Node ID: 1vtdYMUQ1aEzqZeSWkWD9RceYHunHdt9QwAzy7CwyYruPtSNRw, Piece ID: SYC3VAYM2YKT5ZM2VAQQC7663JAM47LYYCU62ZFRVH5UJDTTXMRQ) context deadline exceeded; piecestore: (Node ID: 1dstfvWPzYyG7oHnsByWcpgv6C4NVw9RPt8MdxtZ634FaLrSt4, Piece ID: D6G2XNNG5QPU2VSWNAP6XPRA3LM5GREUKMQG3KEWTKXFBNSEPIZA) context deadline exceeded; piecestore: (Node ID: 12Bf3H3JPRVT3GKxieKhLpk7ZfXc3FrdtweLJKRpFo6yNTomKDu, Piece ID: UD3L4QJO2DEW7V3SJ7HHPO5P6VWX4MURJKKEQ5KIQUHKAUZOZVTQ) context deadline exceeded; piecestore: (Node ID: 1ZtoenGAVBrgVoJkmkyxDqire258rdYcqC3avm8xAPuBGiD1bU, Piece ID: 4GAT5EPHUEH7UO3IHA5FU23RMY7NGYLJU2CBCX3RS2BCAPIDP7VA) context deadline exceeded; piecestore: (Node ID: 12vhu4j9fCJhbUW9rxZshXEbh9gbG1j1sPzgPhDdiD7ecVTQaHz, Piece ID: 6N3EAXOD5X75DQR6W4AHVJ54XQ7BGOF2BRN32IDBHLLYOOYI3ILQ) context deadline exceeded; piecestore: (Node ID: 1kLgnR5vmDgUuaf1aWAUyDsXuHn8aEwVXEQiruQrkfKpXqTiQ1, Piece ID: DGJRCP2CNO7DMWSJ5XMYFWV6CF22SLA4LPCVWLTBRKCQ6NV7HS7Q) context deadline exceeded Duration: 6.138108872s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [....X.X.XXX.X.....X..X.......XXX.XX.XX..X.XXX.XXXX.XXXXXXX..] duration: 30.000920679s trace: https://jaeger.dev.storj.io/trace/581b340c4a171e60 FAIL (2026-01-08 06:38:01 UTC) orbiter warning: context deadline exceeded Duration: 30.000947511s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.........................................X....X....X........] duration: 6.782001872s trace: https://jaeger.dev.storj.io/trace/23102568ffb9ad71 FAIL (2026-01-08 06:37:54 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.782083337s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [....X.XXX........X..X...............X....XXX...XX.X......X..] duration: 21.961292669s trace: https://jaeger.dev.storj.io/trace/12aa5f7c87ea6d98 FAIL (2026-01-08 06:37:53 UTC) orbiter warning: context deadline exceeded Duration: 21.961307257s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X..XX......................................] duration: 5.321370392s trace: https://jaeger.dev.storj.io/trace/233c74a6b9db5cf8 FAIL (2026-01-08 06:37:52 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 53) Duration: 5.321397234s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 9 out of 60 is failed (threshold=3) [.X..................XXXXXXXX................................] duration: 11.143475537s trace: https://jaeger.dev.storj.io/trace/635cbd49e9d724cb FAIL (2026-01-08 06:37:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 21) ======LOGS====== _id=TM5C7SQQA5IAVFWRG2GV2WYRB2IRLWDLVB3MFWK7XILSBDUDIAVQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.977864602 (806h43m10.848947647s) [noise=true , address=185.167.204.206:4059 , piece_id=FKQBFAG3DNQ5OI5DJ7GAZINSWVSVDZED56SJVHBY2ULJO5HUAGNQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.978752623 (806h43m10.849835658s) [noise=true , address=185.167.204.206:4064 , piece_id=3FMVMJWMWGAHNVQX2AIBVII5Z6XTPJUCYABKNXJ4NWRC2LQ6I3FA, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.979487392 (806h43m10.850570427s) [noise=true , address=89.149.201.147:4020 , piece_id=U3G7AY7GMBA527UQXXY6DOLNKJBUR4CZD7T3MOJSQNFFGYJXKR3A, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.980303676 (806h43m10.851386721s) [noise=true , address=185.167.204.206:4061 , piece_id=LJVZ2GBV4S5T6XXBYDEELH6BKTBY2TTOBBXWNNNH2VSE2W5RI5YA, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.980907016 (806h43m10.851990051s) [noise=true , address=89.149.201.147:4011 , piece_id=PXK522IVCJHJDLWXUIP3PQF7PJXVPXLPGM6HE43CY6G2VSLOAWPQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.981714563 (806h43m10.852797608s) [noise=true , address=81.171.14.70:4043 , piece_id=AM727IICNSFNSVCQ3IZ55CAZK6YH3JUFBOJAUHMMYRJ6NEZZREDQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.982411280 (806h43m10.853494325s) [noise=true , address=89.149.201.147:4004 , piece_id=BRZPEKISSTSV7IPNBW7ZJJIIGOTR4SQDJDMQZ3OXOSUEQJZB5CMQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.983960189 (806h43m10.855043224s) [noise=true , address=81.171.14.70:4048 , piece_id=HLP5YVTQMQXLGROPTZJ4RPKFYCCBB3W5ZPHASDVMTJ6ZCTSGMDKQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:11.816868763 (806h43m11.687951808s) [upload=705619]: Next returned nil segment Jan 8 06:12:11.816884092 (806h43m11.687967137s) [upload=705619]: Waiting for error group managing segments... Jan 8 06:12:12.207503101 (806h43m12.078586136s) [noise=true , address=185.167.204.205:4014 , piece_id=42DLCGFRUY5XHUEJPKO6E6BKJSWTUC5IDYRSAE3OXFFVCYA2E67A, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.207636776 (806h43m12.078719821s) [noise=true , address=185.167.204.205:4011 , piece_id=36G2RTRXP6WEHPMQKMEQ3334GYKN6RFC7VYMW6BSCLBQ37OT43YQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.208187303 (806h43m12.079270338s) [noise=true , address=185.167.204.206:4041 , piece_id=CDCLTZOQNX4QLRZ5JT7ZIBQKKBRWVX3SWJSKC46EAIMPSV2SYONA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.208277846 (806h43m12.079360881s) [noise=true , address=185.167.204.205:4003 , piece_id=EGXKQAKMGKB3SDIEFFEVFE6XYGHZ4ZR6A5B6MFDB7XFY7KID4QYA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.211357060 (806h43m12.082440095s) [noise=true , address=185.167.204.206:4062 , piece_id=MJLGFBLTJEG7UHJMFLI72MHHH6FWFLP3O4VNTUAS5Z4KCH27EHOQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.215506782 (806h43m12.086589817s) [noise=true , address=185.167.204.206:4068 , piece_id=L6DRFJFP5OFG2UEGHUGNCRWVGJ7XQFGJJUXNT7WGSEJXS5TMKNRQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.221151259 (806h43m12.092234294s) [noise=true , address=185.167.204.206:4059 , piece_id=FKQBFAG3DNQ5OI5DJ7GAZINSWVSVDZED56SJVHBY2ULJO5HUAGNQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.247553292 (806h43m12.118636337s) [noise=true , address=185.167.204.206:4064 , piece_id=3FMVMJWMWGAHNVQX2AIBVII5Z6XTPJUCYABKNXJ4NWRC2LQ6I3FA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.268061592 (806h43m12.139144637s) [noise=true , address=185.167.204.206:4061 , piece_id=LJVZ2GBV4S5T6XXBYDEELH6BKTBY2TTOBBXWNNNH2VSE2W5RI5YA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.268319703 (806h43m12.139402739s) [noise=true , address=185.167.204.205:4017 , piece_id=ACCHFJNRCMWJWXPZJDXALXSUVPLETHOSDAGKPKDAKRMWJ2JI6RMQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.270984638 (806h43m12.142067684s) [noise=true , address=185.167.204.206:4037 , piece_id=HAKQW2RQFJJGBIG26EV4B5TJJBZRM2YMXVU5PZQIKNA5YYEAZVHA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.272396927 (806h43m12.143479962s) [noise=true , address=185.167.204.205:4034 , piece_id=EMHALXT4VKLMJNTQV4XMRGLVOPBRF7O45YKF5Q3RYUBF6DGHPTJQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:13.719906169 (806h43m13.590989215s) [noise=true , address=89.149.201.147:4007 , piece_id=XDKNFX56PLHN6AXAKPM3LSRUHAFHIOZIJF2CAFXOMDLUH7URSS3Q, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:15.774082352 (806h43m15.645165397s) [noise=true , address=89.149.201.147:4005 , piece_id=3WEJWRZJOWY6I7ZNNWHK5QYTF5MULZNKHWVZMBEJJPFNB6WU4UCA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:16.530239032 (806h43m16.401322067s) [noise=true , address=89.149.201.147:4001 , piece_id=LWDR4KCC5QHQDLFVZYVTYJH26GRABV2JFC3525OVIORRG2SRHQMA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:16.928628457 (806h43m16.799711492s) [noise=true , address=81.171.14.70:4041 , piece_id=J3IG4LCWN7UM2NCQAY4FKTPNYH5NRW7TDB6ZGMNPQEI7HY5CZGRQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:17.995225112 (806h43m17.866308157s) [noise=true , address=89.149.201.147:4011 , piece_id=PXK522IVCJHJDLWXUIP3PQF7PJXVPXLPGM6HE43CY6G2VSLOAWPQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:18.045551680 (806h43m17.916634725s) [noise=true , address=81.171.14.70:4027 , piece_id=5P2A7PSIMWZC2I6SAGUEUEJDDW67BD7AKQ34Z3XZHFPJGSHLJLVQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:19.316134332 (806h43m19.187217367s) [noise=true , address=89.149.201.147:4004 , piece_id=BRZPEKISSTSV7IPNBW7ZJJIIGOTR4SQDJDMQZ3OXOSUEQJZB5CMQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:19.607969825 (806h43m19.47905288s) [noise=true , address=81.171.14.70:4035 , piece_id=HAUGVFLDMZCSFVPSOYLS4PBCVHMMMOTPD4ODSDCS3P4NSTX6KPJQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:13:00.649061845 (806h44m0.52014488s) [noise=true , address=89.149.201.147:4010 , piece_id=TM5C7SQQA5IAVFWRG2GV2WYRB2IRLWDLVB3MFWK7XILSBDUDIAVQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:13:10.291882458 (806h44m10.162965513s) [noise=true , address=89.149.201.147:4015 , piece_id=GLHXEDCIEKGM2SPTCNCTJYS2GIEUMIT5LQXCHMDTVNNIY5KLASVQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K, address:89.149.201.147:4015): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.291987709 (806h44m10.163070774s) [noise=true , address=81.171.14.70:4034 , piece_id=VGWIA5ZBLFNA33QXVLF54XTCOCAHSU6TVV3O52DTZ2L3ROIPHNAA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12jcAHt22bETT27CCXJ8ZaaYxxoKBonBYsFcoq6A34DVVjvhQAN, address:81.171.14.70:4034): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.292023307 (806h44m10.163106342s) [noise=true , address=81.171.14.70:4038 , piece_id=QOQ3IYLTGQ27HOLQKQZOOILZXWAPR6PEHLQTHLGLBLXLLQDBA5RA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB, address:81.171.14.70:4038): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.292120161 (806h44m10.163203216s) [noise=true , address=89.149.201.147:4020 , piece_id=U3G7AY7GMBA527UQXXY6DOLNKJBUR4CZD7T3MOJSQNFFGYJXKR3A, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.292271189 (806h44m10.163354245s) [noise=true , address=81.171.14.70:4048 , piece_id=HLP5YVTQMQXLGROPTZJ4RPKFYCCBB3W5ZPHASDVMTJ6ZCTSGMDKQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: read tcp 188.166.225.170:40684->81.171.14.70:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 8 06:13:10.292450230 (806h44m10.163533285s) [noise=true , address=81.171.14.70:4043 , piece_id=AM727IICNSFNSVCQ3IZ55CAZK6YH3JUFBOJAUHMMYRJ6NEZZREDQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko, address:81.171.14.70:4043): protocol: read tcp 188.166.225.170:38496->81.171.14.70:4043: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 8 06:13:10.292509182 (806h44m10.163592237s) [noise=true , address=81.171.14.70:4029 , piece_id=6UO7K2VEO7HMZPJM4S34Z6Z3UWKFMLD3BLATMSIRL4WDIXKVTZCA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:60086->81.171.14.70:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 8 06:13:10.292533999 (806h44m10.163617034s) [seg_pos={0 0}, upload=705619]: Done waiting for segment. successful: 21 optimal: 26 errs: <nil> Jan 8 06:13:10.462867035 (806h44m10.33395007s) [upload=705619]: Done uploading segments... Jan 8 06:13:10.462901039 (806h44m10.333984085s) [upload=705619]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 21) Duration: 11.143507879s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 4.316263526s trace: https://jaeger.dev.storj.io/trace/6173eb4c5ef6f546 FAIL (2026-01-08 06:37:40 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 53) Duration: 4.316286019s core uplink ap1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [X....X.XXX........X..X...............X....XXX...XX.X......X.] duration: 33.108792394s trace: https://jaeger.dev.storj.io/trace/12aa5f7c87ea6d98 FAIL (2026-01-08 06:37:31 UTC) orbiter warning: context deadline exceeded Duration: 33.108808555s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......X.............X......XXX..........XX......X...........] duration: 5.896621527s trace: https://jaeger.dev.storj.io/trace/2f4a5d1589141c78 FAIL (2026-01-08 06:37:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 8 06:29:57.497836890 (807h0m57.368919925s) [upload=705875]: Starting upload Jan 8 06:29:57.497942772 (807h0m57.369025817s) [upload=705875]: Done starting upload Jan 8 06:29:57.498032733 (807h0m57.369115798s) [upload=705875]: Uploading segments... Jan 8 06:29:57.498226622 (807h0m57.369309657s) [upload=705875]: Got next segment. Inline: false Jan 8 06:29:57.498268923 (807h0m57.369351958s) [upload=705875]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 8 06:29:57.498275035 (807h0m57.36935808s) [upload=705875]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 8 06:29:57.742290737 (807h0m57.613373772s) [seg_pos={0 0}, upload=705875]: Begin upload segment... Jan 8 06:29:57.742427608 (807h0m57.613510643s) [seg_pos={0 0}, upload=705875]: Done begin upload segment. Jan 8 06:29:57.742467413 (807h0m57.613550448s) [noise=true , address=45.91.116.12:4225 , piece_id=QXWSWIUFYPRXCH5AAZ6PEJ37HCHDJOX4CO3IY4ICIZKXUAI6V6KQ, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:57.742505766 (807h0m57.613588801s) [noise=true , address=145.79.227.29:4942 , piece_id=J4N4AWCQBMH4VTOLKVBLGGGR4LTOXC34KVZHAV6JZAUO5LXQ2MLA, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:57.742989087 (807h0m57.614072122s) [noise=true , address=145.79.227.51:4411 , piece_id=YNJETDNTDH2WQG7HFKCENQRI5LY76RUKDAY3UMUIVJVYOBEUVURA, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:57.743391483 (807h0m57.614474569s) [noise=true , address=77.90.31.19:4842 , piece_id=OS6NWVUWS7QS74BAUTBEJ4DD5DFZMGTSYZNXK47ELZDHTIC5UIZQ, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:57.743853232 (807h0m57.614936277s) [noise=true , address=145.79.227.33:4871 , piece_id=CLJAXD7KEUKQPFRK3EWJ7BLUSYRSIVBYDB4SORXPU4WVEJTQSTGA, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:57.744402128 (807h0m57.615485163s) [noise=true , address=145.79.227.15:4873 , piece_id=IF45DGQ6N4O2WF77I5S7WOJ45647XEWFDCNYWJBP5ZBCKZHKU6GQ, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:57.744782803 (807h0m57.615865838s) [noise=true , address=45.91.116.6:4974 , piece_id=3QMCHIZMMV3YG5PDMC3FBJ76T6Z2GNHBEPOPZPXO6GTHPRUJGEHQ, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:57.745100147 (807h0m57.616183182s) [noise=true , address=145.79.227.8:4330 , piece_id=F5SS22IRGVNXLL6CMFJ32PASAS5UKLVZR2RETHRKASOJEJB5YGXA, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:57.745400149 (807h0m57.616483184s) [noise=true , address=145.79.227.47:4266 , piece_id=NVZRQ4E7SKDL42JSEKV2OANGS7KO43JZIWTIGAMIBYHMZDZ7YJ6A, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:57.745939385 (807h0m57.61702242s) [noise=true , address=145.79.227.41:4086 , piece_id=J252QHNFHJK5DRA5KHUOOOQJORJCTD5UAD3EHVJE7NSUCDSA5CLQ, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:57.746237683 (807h0m57.617320718s) [noise=true , address=77.90.31.15:4700 , piece_id=OM6ZQMPLZYDEQWUY7L3FE5FD3XS2PBMOZUNTFGO5YOEO2IVFETFA, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:57.746601846 (807h0m57.617684881s) [noise=true , address=145.79.227.9:4365 , piece_id=5EZKRCKWT3HHOBBAWRNPPPHZGYA4Y5G5MO5NWDGXLZBYVOY42LQA, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:57.746923509 (807h0m57.618006554s) [noise=true , address=77.90.31.21:4877 , piece_id=2VJESI7ESS3XU5RHUN3OGTD2EN5FY235OFQBE6I3HKOXSLFKXIGQ, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:57.747212198 (807h0m57.618295233s) [noise=true , address=45.91.116.4:4939 , piece_id=ERL4UXWSP7TYDR55BTYD2WFO4MX5PSTZTRXRB635NVUQZSYC6PHA, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:57.747518251 (807h0m57.618601286s) [noise=true , address=145.79.227.3:4149 , piece_id=VXRXX2BG6R7DPNU3RUWEGQMDHL5UHERSFE5Q2236OCZ2K3D37MCA, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:57.747788676 (807h0m57.618871711s) [noise=true , address=145.79.227.37:4567 , piece_id=RKI4JEZ2D4GPGBL4J3GJZURGV62WAOZFPWXBCAVGHQSV2ULNB3CQ, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:59.451705521 (807h0m59.322788566s) [upload=705875]: Next returned nil segment Jan 8 06:29:59.451724296 (807h0m59.322807341s) [upload=705875]: Waiting for error group managing segments... Jan 8 06:29:59.731703830 (807h0m59.602786875s) [noise=true , address=145.79.227.8:4330 , piece_id=F5SS22IRGVNXLL6CMFJ32PASAS5UKLVZR2RETHRKASOJEJB5YGXA, seg_pos={0 0}, upload=705875]: Done uploading piece. err: <nil> Jan 8 06:29:59.739107869 (807h0m59.610190904s) [noise=true , address=145.79.227.3:4149 , piece_id=VXRXX2BG6R7DPNU3RUWEGQMDHL5UHERSFE5Q2236OCZ2K3D37MCA, seg_pos={0 0}, upload=705875]: Done uploading piece. err: <nil> Jan 8 06:29:59.746157061 (807h0m59.617240096s) [noise=true , address=145.79.227.47:4266 , piece_id=NVZRQ4E7SKDL42JSEKV2OANGS7KO43JZIWTIGAMIBYHMZDZ7YJ6A, seg_pos={0 0}, upload=705875]: Done uploading piece. err: <nil> Jan 8 06:29:59.750791986 (807h0m59.621875022s) [noise=true , address=145.79.227.29:4942 , piece_id=J4N4AWCQBMH4VTOLKVBLGGGR4LTOXC34KVZHAV6JZAUO5LXQ2MLA, seg_pos={0 0}, upload=705875]: Done uploading piece. err: <nil> Jan 8 06:29:59.754874059 (807h0m59.625957094s) [noise=true , address=145.79.227.41:4086 , piece_id=J252QHNFHJK5DRA5KHUOOOQJORJCTD5UAD3EHVJE7NSUCDSA5CLQ, seg_pos={0 0}, upload=705875]: Done uploading piece. err: <nil> Jan 8 06:29:59.755794261 (807h0m59.626877296s) [noise=true , address=145.79.227.51:4411 , piece_id=YNJETDNTDH2WQG7HFKCENQRI5LY76RUKDAY3UMUIVJVYOBEUVURA, seg_pos={0 0}, upload=705875]: Done uploading piece. err: <nil> Jan 8 06:29:59.757967229 (807h0m59.629050264s) [noise=true , address=145.79.227.9:4365 , piece_id=5EZKRCKWT3HHOBBAWRNPPPHZGYA4Y5G5MO5NWDGXLZBYVOY42LQA, seg_pos={0 0}, upload=705875]: Done uploading piece. err: <nil> Jan 8 06:29:59.760120860 (807h0m59.631203905s) [noise=true , address=145.79.227.33:4871 , piece_id=CLJAXD7KEUKQPFRK3EWJ7BLUSYRSIVBYDB4SORXPU4WVEJTQSTGA, seg_pos={0 0}, upload=705875]: Done uploading piece. err: <nil> Jan 8 06:29:59.760370145 (807h0m59.63145318s) [noise=true , address=145.79.227.15:4873 , piece_id=IF45DGQ6N4O2WF77I5S7WOJ45647XEWFDCNYWJBP5ZBCKZHKU6GQ, seg_pos={0 0}, upload=705875]: Done uploading piece. err: <nil> Jan 8 06:29:59.843714465 (807h0m59.7147975s) [noise=true , address=145.79.227.37:4567 , piece_id=RKI4JEZ2D4GPGBL4J3GJZURGV62WAOZFPWXBCAVGHQSV2ULNB3CQ, seg_pos={0 0}, upload=705875]: Done uploading piece. err: <nil> Jan 8 06:30:00.014889799 (807h0m59.885972834s) [noise=true , address=77.90.31.15:4700 , piece_id=OM6ZQMPLZYDEQWUY7L3FE5FD3XS2PBMOZUNTFGO5YOEO2IVFETFA, seg_pos={0 0}, upload=705875]: Done uploading piece. err: <nil> Jan 8 06:30:00.026178233 (807h0m59.897261279s) [noise=true , address=77.90.31.21:4877 , piece_id=2VJESI7ESS3XU5RHUN3OGTD2EN5FY235OFQBE6I3HKOXSLFKXIGQ, seg_pos={0 0}, upload=705875]: Done uploading piece. err: <nil> Jan 8 06:30:00.032585304 (807h0m59.903668349s) [noise=true , address=77.90.31.19:4842 , piece_id=OS6NWVUWS7QS74BAUTBEJ4DD5DFZMGTSYZNXK47ELZDHTIC5UIZQ, seg_pos={0 0}, upload=705875]: Done uploading piece. err: <nil> Jan 8 06:30:00.032611754 (807h0m59.903694789s) [seg_pos={0 0}, upload=705875]: Segment reached optimal threshold of 13 pieces. Jan 8 06:30:00.032854577 (807h0m59.903937622s) [noise=true , address=45.91.116.4:4939 , piece_id=ERL4UXWSP7TYDR55BTYD2WFO4MX5PSTZTRXRB635NVUQZSYC6PHA, seg_pos={0 0}, upload=705875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jan 8 06:30:00.032871659 (807h0m59.903954694s) [noise=true , address=45.91.116.4:4939 , piece_id=ERL4UXWSP7TYDR55BTYD2WFO4MX5PSTZTRXRB635NVUQZSYC6PHA, seg_pos={0 0}, upload=705875]: Long tail cancellation due to optimal threshold Jan 8 06:30:00.033056821 (807h0m59.904139856s) [noise=true , address=45.91.116.12:4225 , piece_id=QXWSWIUFYPRXCH5AAZ6PEJ37HCHDJOX4CO3IY4ICIZKXUAI6V6KQ, seg_pos={0 0}, upload=705875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Jan 8 06:30:00.033066360 (807h0m59.904149395s) [noise=true , address=45.91.116.12:4225 , piece_id=QXWSWIUFYPRXCH5AAZ6PEJ37HCHDJOX4CO3IY4ICIZKXUAI6V6KQ, seg_pos={0 0}, upload=705875]: Long tail cancellation due to optimal threshold Jan 8 06:30:00.033089764 (807h0m59.904172799s) [noise=true , address=45.91.116.6:4974 , piece_id=3QMCHIZMMV3YG5PDMC3FBJ76T6Z2GNHBEPOPZPXO6GTHPRUJGEHQ, seg_pos={0 0}, upload=705875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:51184->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:51184->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51184->45.91.116.6:4974: use of closed network connection Jan 8 06:30:00.033097680 (807h0m59.904180715s) [noise=true , address=45.91.116.6:4974 , piece_id=3QMCHIZMMV3YG5PDMC3FBJ76T6Z2GNHBEPOPZPXO6GTHPRUJGEHQ, seg_pos={0 0}, upload=705875]: Long tail cancellation due to optimal threshold Jan 8 06:30:00.033137836 (807h0m59.904220881s) [seg_pos={0 0}, upload=705875]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 8 06:30:00.033241193 (807h0m59.904324238s) [upload=705875]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 8 06:30:00.033248697 (807h0m59.904331732s) [upload=705875]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 8 06:30:00.320262835 (807h1m0.19134588s) [upload=705875]: Done uploading segments... Jan 8 06:30:00.320287482 (807h1m0.191370538s) [upload=705875]: Upload finished. err: <nil> Duration: 5.896650554s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 4.13212565s trace: https://jaeger.dev.storj.io/trace/6173eb4c5ef6f546 FAIL (2026-01-08 06:37:13 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 53) Duration: 4.132149015s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.77041926s trace: https://jaeger.dev.storj.io/trace/3738a06b2bfcba74 FAIL (2026-01-08 06:37:09 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12hkSjuH8cdS4byda1bykpe2QJz42HXaK1uRtkTbCQdynovAKSz, Piece ID: 4GTLFPYP5LTIZPMRSOVJNHNBOMLFZTIF7CD5QHSAWAQT2DQYCAEQ) context deadline exceeded; piecestore: (Node ID: 1bWLCSW1tixYzHThBQsUPvXtTpnf4egF8uGHFUXDQkCeE4BwBU, Piece ID: RX5TQUH56ZXKJADRRFT7DX6J6VV32WB3EFRCM456NZ2TD2WHKM3A) context deadline exceeded; piecestore: (Node ID: 1XxiRqM4j35HGGzWPnmrHHfbWS8jryWrW5aY66SgDuDQKHYopv, Piece ID: GAOHJL4R553KJFKTWIRCGC3NHHM4TBFNH6GJYFREM23JQQS645TQ) context deadline exceeded; piecestore: (Node ID: 15LfiPEGogvaJkFN67RXm8egaXMGxsc9XmPo26GFwT5TZKmg75, Piece ID: HD6GDZKL6A43NRG367DI674KCKPZAN4PP6S5ZRHHYR33EPAQ4KRQ) context deadline exceeded; piecestore: (Node ID: 1m2RK4RXMBjqBpWj8Xof42VU2fyPNygiY1WR3gcsVRxd6jKe4E, Piece ID: SCDRYTITO42A42357QKLW7RUWW6VKZLZLYQ3H2NIPXAKYGOIC5HQ) context deadline exceeded; piecestore: (Node ID: 1vtdYMUQ1aEzqZeSWkWD9RceYHunHdt9QwAzy7CwyYruPtSNRw, Piece ID: SYC3VAYM2YKT5ZM2VAQQC7663JAM47LYYCU62ZFRVH5UJDTTXMRQ) context deadline exceeded; piecestore: (Node ID: 1dstfvWPzYyG7oHnsByWcpgv6C4NVw9RPt8MdxtZ634FaLrSt4, Piece ID: D6G2XNNG5QPU2VSWNAP6XPRA3LM5GREUKMQG3KEWTKXFBNSEPIZA) context deadline exceeded; piecestore: (Node ID: 12Bf3H3JPRVT3GKxieKhLpk7ZfXc3FrdtweLJKRpFo6yNTomKDu, Piece ID: UD3L4QJO2DEW7V3SJ7HHPO5P6VWX4MURJKKEQ5KIQUHKAUZOZVTQ) context deadline exceeded; piecestore: (Node ID: 1ZtoenGAVBrgVoJkmkyxDqire258rdYcqC3avm8xAPuBGiD1bU, Piece ID: 4GAT5EPHUEH7UO3IHA5FU23RMY7NGYLJU2CBCX3RS2BCAPIDP7VA) context deadline exceeded; piecestore: (Node ID: 12vhu4j9fCJhbUW9rxZshXEbh9gbG1j1sPzgPhDdiD7ecVTQaHz, Piece ID: 6N3EAXOD5X75DQR6W4AHVJ54XQ7BGOF2BRN32IDBHLLYOOYI3ILQ) context deadline exceeded; piecestore: (Node ID: 1kLgnR5vmDgUuaf1aWAUyDsXuHn8aEwVXEQiruQrkfKpXqTiQ1, Piece ID: DGJRCP2CNO7DMWSJ5XMYFWV6CF22SLA4LPCVWLTBRKCQ6NV7HS7Q) context deadline exceeded Duration: 5.770529278s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [.....X.X.XXX.X.....X..X.......XXX.XX.XX..X.XXX.XXXX.XXXXXXX.] duration: 24.119488032s trace: https://jaeger.dev.storj.io/trace/581b340c4a171e60 FAIL (2026-01-08 06:37:05 UTC) orbiter warning: context deadline exceeded Duration: 24.119503932s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 57 out of 60 is failed (threshold=6) [XXXXXXXX.X..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000780231s trace: https://jaeger.dev.storj.io/trace/617cabfcb0ce9a1d FAIL (2026-01-08 06:37:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000794738s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X..XX.....................................] duration: 5.078701441s trace: https://jaeger.dev.storj.io/trace/233c74a6b9db5cf8 FAIL (2026-01-08 06:36:58 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 53) Duration: 5.078733373s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 9 out of 60 is failed (threshold=3) [..X..................XXXXXXXX...............................] duration: 11.587195888s trace: https://jaeger.dev.storj.io/trace/635cbd49e9d724cb FAIL (2026-01-08 06:36:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 21) ======LOGS====== _id=TM5C7SQQA5IAVFWRG2GV2WYRB2IRLWDLVB3MFWK7XILSBDUDIAVQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.977864602 (806h43m10.848947647s) [noise=true , address=185.167.204.206:4059 , piece_id=FKQBFAG3DNQ5OI5DJ7GAZINSWVSVDZED56SJVHBY2ULJO5HUAGNQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.978752623 (806h43m10.849835658s) [noise=true , address=185.167.204.206:4064 , piece_id=3FMVMJWMWGAHNVQX2AIBVII5Z6XTPJUCYABKNXJ4NWRC2LQ6I3FA, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.979487392 (806h43m10.850570427s) [noise=true , address=89.149.201.147:4020 , piece_id=U3G7AY7GMBA527UQXXY6DOLNKJBUR4CZD7T3MOJSQNFFGYJXKR3A, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.980303676 (806h43m10.851386721s) [noise=true , address=185.167.204.206:4061 , piece_id=LJVZ2GBV4S5T6XXBYDEELH6BKTBY2TTOBBXWNNNH2VSE2W5RI5YA, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.980907016 (806h43m10.851990051s) [noise=true , address=89.149.201.147:4011 , piece_id=PXK522IVCJHJDLWXUIP3PQF7PJXVPXLPGM6HE43CY6G2VSLOAWPQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.981714563 (806h43m10.852797608s) [noise=true , address=81.171.14.70:4043 , piece_id=AM727IICNSFNSVCQ3IZ55CAZK6YH3JUFBOJAUHMMYRJ6NEZZREDQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.982411280 (806h43m10.853494325s) [noise=true , address=89.149.201.147:4004 , piece_id=BRZPEKISSTSV7IPNBW7ZJJIIGOTR4SQDJDMQZ3OXOSUEQJZB5CMQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:10.983960189 (806h43m10.855043224s) [noise=true , address=81.171.14.70:4048 , piece_id=HLP5YVTQMQXLGROPTZJ4RPKFYCCBB3W5ZPHASDVMTJ6ZCTSGMDKQ, seg_pos={0 0}, upload=705619]: Uploading piece... Jan 8 06:12:11.816868763 (806h43m11.687951808s) [upload=705619]: Next returned nil segment Jan 8 06:12:11.816884092 (806h43m11.687967137s) [upload=705619]: Waiting for error group managing segments... Jan 8 06:12:12.207503101 (806h43m12.078586136s) [noise=true , address=185.167.204.205:4014 , piece_id=42DLCGFRUY5XHUEJPKO6E6BKJSWTUC5IDYRSAE3OXFFVCYA2E67A, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.207636776 (806h43m12.078719821s) [noise=true , address=185.167.204.205:4011 , piece_id=36G2RTRXP6WEHPMQKMEQ3334GYKN6RFC7VYMW6BSCLBQ37OT43YQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.208187303 (806h43m12.079270338s) [noise=true , address=185.167.204.206:4041 , piece_id=CDCLTZOQNX4QLRZ5JT7ZIBQKKBRWVX3SWJSKC46EAIMPSV2SYONA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.208277846 (806h43m12.079360881s) [noise=true , address=185.167.204.205:4003 , piece_id=EGXKQAKMGKB3SDIEFFEVFE6XYGHZ4ZR6A5B6MFDB7XFY7KID4QYA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.211357060 (806h43m12.082440095s) [noise=true , address=185.167.204.206:4062 , piece_id=MJLGFBLTJEG7UHJMFLI72MHHH6FWFLP3O4VNTUAS5Z4KCH27EHOQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.215506782 (806h43m12.086589817s) [noise=true , address=185.167.204.206:4068 , piece_id=L6DRFJFP5OFG2UEGHUGNCRWVGJ7XQFGJJUXNT7WGSEJXS5TMKNRQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.221151259 (806h43m12.092234294s) [noise=true , address=185.167.204.206:4059 , piece_id=FKQBFAG3DNQ5OI5DJ7GAZINSWVSVDZED56SJVHBY2ULJO5HUAGNQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.247553292 (806h43m12.118636337s) [noise=true , address=185.167.204.206:4064 , piece_id=3FMVMJWMWGAHNVQX2AIBVII5Z6XTPJUCYABKNXJ4NWRC2LQ6I3FA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.268061592 (806h43m12.139144637s) [noise=true , address=185.167.204.206:4061 , piece_id=LJVZ2GBV4S5T6XXBYDEELH6BKTBY2TTOBBXWNNNH2VSE2W5RI5YA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.268319703 (806h43m12.139402739s) [noise=true , address=185.167.204.205:4017 , piece_id=ACCHFJNRCMWJWXPZJDXALXSUVPLETHOSDAGKPKDAKRMWJ2JI6RMQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.270984638 (806h43m12.142067684s) [noise=true , address=185.167.204.206:4037 , piece_id=HAKQW2RQFJJGBIG26EV4B5TJJBZRM2YMXVU5PZQIKNA5YYEAZVHA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:12.272396927 (806h43m12.143479962s) [noise=true , address=185.167.204.205:4034 , piece_id=EMHALXT4VKLMJNTQV4XMRGLVOPBRF7O45YKF5Q3RYUBF6DGHPTJQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:13.719906169 (806h43m13.590989215s) [noise=true , address=89.149.201.147:4007 , piece_id=XDKNFX56PLHN6AXAKPM3LSRUHAFHIOZIJF2CAFXOMDLUH7URSS3Q, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:15.774082352 (806h43m15.645165397s) [noise=true , address=89.149.201.147:4005 , piece_id=3WEJWRZJOWY6I7ZNNWHK5QYTF5MULZNKHWVZMBEJJPFNB6WU4UCA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:16.530239032 (806h43m16.401322067s) [noise=true , address=89.149.201.147:4001 , piece_id=LWDR4KCC5QHQDLFVZYVTYJH26GRABV2JFC3525OVIORRG2SRHQMA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:16.928628457 (806h43m16.799711492s) [noise=true , address=81.171.14.70:4041 , piece_id=J3IG4LCWN7UM2NCQAY4FKTPNYH5NRW7TDB6ZGMNPQEI7HY5CZGRQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:17.995225112 (806h43m17.866308157s) [noise=true , address=89.149.201.147:4011 , piece_id=PXK522IVCJHJDLWXUIP3PQF7PJXVPXLPGM6HE43CY6G2VSLOAWPQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:18.045551680 (806h43m17.916634725s) [noise=true , address=81.171.14.70:4027 , piece_id=5P2A7PSIMWZC2I6SAGUEUEJDDW67BD7AKQ34Z3XZHFPJGSHLJLVQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:19.316134332 (806h43m19.187217367s) [noise=true , address=89.149.201.147:4004 , piece_id=BRZPEKISSTSV7IPNBW7ZJJIIGOTR4SQDJDMQZ3OXOSUEQJZB5CMQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:12:19.607969825 (806h43m19.47905288s) [noise=true , address=81.171.14.70:4035 , piece_id=HAUGVFLDMZCSFVPSOYLS4PBCVHMMMOTPD4ODSDCS3P4NSTX6KPJQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:13:00.649061845 (806h44m0.52014488s) [noise=true , address=89.149.201.147:4010 , piece_id=TM5C7SQQA5IAVFWRG2GV2WYRB2IRLWDLVB3MFWK7XILSBDUDIAVQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: <nil> Jan 8 06:13:10.291882458 (806h44m10.162965513s) [noise=true , address=89.149.201.147:4015 , piece_id=GLHXEDCIEKGM2SPTCNCTJYS2GIEUMIT5LQXCHMDTVNNIY5KLASVQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K, address:89.149.201.147:4015): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.291987709 (806h44m10.163070774s) [noise=true , address=81.171.14.70:4034 , piece_id=VGWIA5ZBLFNA33QXVLF54XTCOCAHSU6TVV3O52DTZ2L3ROIPHNAA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12jcAHt22bETT27CCXJ8ZaaYxxoKBonBYsFcoq6A34DVVjvhQAN, address:81.171.14.70:4034): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.292023307 (806h44m10.163106342s) [noise=true , address=81.171.14.70:4038 , piece_id=QOQ3IYLTGQ27HOLQKQZOOILZXWAPR6PEHLQTHLGLBLXLLQDBA5RA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:15im1ewAFZd3JR1JxHcRxcrSkGaM7VwporX63EoYSfPmmj5ZzB, address:81.171.14.70:4038): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.292120161 (806h44m10.163203216s) [noise=true , address=89.149.201.147:4020 , piece_id=U3G7AY7GMBA527UQXXY6DOLNKJBUR4CZD7T3MOJSQNFFGYJXKR3A, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic, address:89.149.201.147:4020): protocol: expected piece hash; context deadline exceeded; EOF Jan 8 06:13:10.292271189 (806h44m10.163354245s) [noise=true , address=81.171.14.70:4048 , piece_id=HLP5YVTQMQXLGROPTZJ4RPKFYCCBB3W5ZPHASDVMTJ6ZCTSGMDKQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12bbN67ZebXjF4iSyaWu4TDsG1XugTm1YdNHqRdRCbZ2WWRunRG, address:81.171.14.70:4048): protocol: read tcp 188.166.225.170:40684->81.171.14.70:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 8 06:13:10.292450230 (806h44m10.163533285s) [noise=true , address=81.171.14.70:4043 , piece_id=AM727IICNSFNSVCQ3IZ55CAZK6YH3JUFBOJAUHMMYRJ6NEZZREDQ, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko, address:81.171.14.70:4043): protocol: read tcp 188.166.225.170:38496->81.171.14.70:4043: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 8 06:13:10.292509182 (806h44m10.163592237s) [noise=true , address=81.171.14.70:4029 , piece_id=6UO7K2VEO7HMZPJM4S34Z6Z3UWKFMLD3BLATMSIRL4WDIXKVTZCA, seg_pos={0 0}, upload=705619]: Done uploading piece. err: ecclient: upload failed (node:18yEfU2X4jSrom1JDtQ22cUEvomcg91ohvay2Nsr1Vr8juo9to, address:81.171.14.70:4029): protocol: read tcp 188.166.225.170:60086->81.171.14.70:4029: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 8 06:13:10.292533999 (806h44m10.163617034s) [seg_pos={0 0}, upload=705619]: Done waiting for segment. successful: 21 optimal: 26 errs: <nil> Jan 8 06:13:10.462867035 (806h44m10.33395007s) [upload=705619]: Done uploading segments... Jan 8 06:13:10.462901039 (806h44m10.333984085s) [upload=705619]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 21) Duration: 11.587222358s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.X....X.XXX........X..X...............X....XXX...XX.X......X] duration: 1m0.000790666s trace: https://jaeger.dev.storj.io/trace/12aa5f7c87ea6d98 FAIL (2026-01-08 06:36:57 UTC) orbiter warning: context deadline exceeded Duration: 1m0.000809651s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [..........................................X....X....X.......] duration: 12.113082326s trace: https://jaeger.dev.storj.io/trace/23102568ffb9ad71 FAIL (2026-01-08 06:36:56 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 12.113110199s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.908451634s trace: https://jaeger.dev.storj.io/trace/3738a06b2bfcba74 FAIL (2026-01-08 06:36:52 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12hkSjuH8cdS4byda1bykpe2QJz42HXaK1uRtkTbCQdynovAKSz, Piece ID: 4GTLFPYP5LTIZPMRSOVJNHNBOMLFZTIF7CD5QHSAWAQT2DQYCAEQ) context deadline exceeded; piecestore: (Node ID: 1bWLCSW1tixYzHThBQsUPvXtTpnf4egF8uGHFUXDQkCeE4BwBU, Piece ID: RX5TQUH56ZXKJADRRFT7DX6J6VV32WB3EFRCM456NZ2TD2WHKM3A) context deadline exceeded; piecestore: (Node ID: 1XxiRqM4j35HGGzWPnmrHHfbWS8jryWrW5aY66SgDuDQKHYopv, Piece ID: GAOHJL4R553KJFKTWIRCGC3NHHM4TBFNH6GJYFREM23JQQS645TQ) context deadline exceeded; piecestore: (Node ID: 15LfiPEGogvaJkFN67RXm8egaXMGxsc9XmPo26GFwT5TZKmg75, Piece ID: HD6GDZKL6A43NRG367DI674KCKPZAN4PP6S5ZRHHYR33EPAQ4KRQ) context deadline exceeded; piecestore: (Node ID: 1m2RK4RXMBjqBpWj8Xof42VU2fyPNygiY1WR3gcsVRxd6jKe4E, Piece ID: SCDRYTITO42A42357QKLW7RUWW6VKZLZLYQ3H2NIPXAKYGOIC5HQ) context deadline exceeded; piecestore: (Node ID: 1vtdYMUQ1aEzqZeSWkWD9RceYHunHdt9QwAzy7CwyYruPtSNRw, Piece ID: SYC3VAYM2YKT5ZM2VAQQC7663JAM47LYYCU62ZFRVH5UJDTTXMRQ) context deadline exceeded; piecestore: (Node ID: 1dstfvWPzYyG7oHnsByWcpgv6C4NVw9RPt8MdxtZ634FaLrSt4, Piece ID: D6G2XNNG5QPU2VSWNAP6XPRA3LM5GREUKMQG3KEWTKXFBNSEPIZA) context deadline exceeded; piecestore: (Node ID: 12Bf3H3JPRVT3GKxieKhLpk7ZfXc3FrdtweLJKRpFo6yNTomKDu, Piece ID: UD3L4QJO2DEW7V3SJ7HHPO5P6VWX4MURJKKEQ5KIQUHKAUZOZVTQ) context deadline exceeded; piecestore: (Node ID: 1ZtoenGAVBrgVoJkmkyxDqire258rdYcqC3avm8xAPuBGiD1bU, Piece ID: 4GAT5EPHUEH7UO3IHA5FU23RMY7NGYLJU2CBCX3RS2BCAPIDP7VA) context deadline exceeded; piecestore: (Node ID: 12vhu4j9fCJhbUW9rxZshXEbh9gbG1j1sPzgPhDdiD7ecVTQaHz, Piece ID: 6N3EAXOD5X75DQR6W4AHVJ54XQ7BGOF2BRN32IDBHLLYOOYI3ILQ) context deadline exceeded; piecestore: (Node ID: 1kLgnR5vmDgUuaf1aWAUyDsXuHn8aEwVXEQiruQrkfKpXqTiQ1, Piece ID: DGJRCP2CNO7DMWSJ5XMYFWV6CF22SLA4LPCVWLTBRKCQ6NV7HS7Q) context deadline exceeded Duration: 5.908487713s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [......X.X.XXX.X.....X..X.......XXX.XX.XX..X.XXX.XXXX.XXXXXXX] duration: 30.001540042s trace: https://jaeger.dev.storj.io/trace/581b340c4a171e60 FAIL (2026-01-08 06:36:40 UTC) orbiter warning: context deadline exceeded Duration: 30.00156941s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......X.............X......XXX..........XX......X..........] duration: 5.760806264s trace: https://jaeger.dev.storj.io/trace/2f4a5d1589141c78 FAIL (2026-01-08 06:36:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 8 06:29:57.497836890 (807h0m57.368919925s) [upload=705875]: Starting upload Jan 8 06:29:57.497942772 (807h0m57.369025817s) [upload=705875]: Done starting upload Jan 8 06:29:57.498032733 (807h0m57.369115798s) [upload=705875]: Uploading segments... Jan 8 06:29:57.498226622 (807h0m57.369309657s) [upload=705875]: Got next segment. Inline: false Jan 8 06:29:57.498268923 (807h0m57.369351958s) [upload=705875]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 8 06:29:57.498275035 (807h0m57.36935808s) [upload=705875]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 8 06:29:57.742290737 (807h0m57.613373772s) [seg_pos={0 0}, upload=705875]: Begin upload segment... Jan 8 06:29:57.742427608 (807h0m57.613510643s) [seg_pos={0 0}, upload=705875]: Done begin upload segment. Jan 8 06:29:57.742467413 (807h0m57.613550448s) [noise=true , address=45.91.116.12:4225 , piece_id=QXWSWIUFYPRXCH5AAZ6PEJ37HCHDJOX4CO3IY4ICIZKXUAI6V6KQ, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:57.742505766 (807h0m57.613588801s) [noise=true , address=145.79.227.29:4942 , piece_id=J4N4AWCQBMH4VTOLKVBLGGGR4LTOXC34KVZHAV6JZAUO5LXQ2MLA, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:57.742989087 (807h0m57.614072122s) [noise=true , address=145.79.227.51:4411 , piece_id=YNJETDNTDH2WQG7HFKCENQRI5LY76RUKDAY3UMUIVJVYOBEUVURA, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:57.743391483 (807h0m57.614474569s) [noise=true , address=77.90.31.19:4842 , piece_id=OS6NWVUWS7QS74BAUTBEJ4DD5DFZMGTSYZNXK47ELZDHTIC5UIZQ, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:57.743853232 (807h0m57.614936277s) [noise=true , address=145.79.227.33:4871 , piece_id=CLJAXD7KEUKQPFRK3EWJ7BLUSYRSIVBYDB4SORXPU4WVEJTQSTGA, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:57.744402128 (807h0m57.615485163s) [noise=true , address=145.79.227.15:4873 , piece_id=IF45DGQ6N4O2WF77I5S7WOJ45647XEWFDCNYWJBP5ZBCKZHKU6GQ, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:57.744782803 (807h0m57.615865838s) [noise=true , address=45.91.116.6:4974 , piece_id=3QMCHIZMMV3YG5PDMC3FBJ76T6Z2GNHBEPOPZPXO6GTHPRUJGEHQ, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:57.745100147 (807h0m57.616183182s) [noise=true , address=145.79.227.8:4330 , piece_id=F5SS22IRGVNXLL6CMFJ32PASAS5UKLVZR2RETHRKASOJEJB5YGXA, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:57.745400149 (807h0m57.616483184s) [noise=true , address=145.79.227.47:4266 , piece_id=NVZRQ4E7SKDL42JSEKV2OANGS7KO43JZIWTIGAMIBYHMZDZ7YJ6A, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:57.745939385 (807h0m57.61702242s) [noise=true , address=145.79.227.41:4086 , piece_id=J252QHNFHJK5DRA5KHUOOOQJORJCTD5UAD3EHVJE7NSUCDSA5CLQ, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:57.746237683 (807h0m57.617320718s) [noise=true , address=77.90.31.15:4700 , piece_id=OM6ZQMPLZYDEQWUY7L3FE5FD3XS2PBMOZUNTFGO5YOEO2IVFETFA, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:57.746601846 (807h0m57.617684881s) [noise=true , address=145.79.227.9:4365 , piece_id=5EZKRCKWT3HHOBBAWRNPPPHZGYA4Y5G5MO5NWDGXLZBYVOY42LQA, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:57.746923509 (807h0m57.618006554s) [noise=true , address=77.90.31.21:4877 , piece_id=2VJESI7ESS3XU5RHUN3OGTD2EN5FY235OFQBE6I3HKOXSLFKXIGQ, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:57.747212198 (807h0m57.618295233s) [noise=true , address=45.91.116.4:4939 , piece_id=ERL4UXWSP7TYDR55BTYD2WFO4MX5PSTZTRXRB635NVUQZSYC6PHA, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:57.747518251 (807h0m57.618601286s) [noise=true , address=145.79.227.3:4149 , piece_id=VXRXX2BG6R7DPNU3RUWEGQMDHL5UHERSFE5Q2236OCZ2K3D37MCA, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:57.747788676 (807h0m57.618871711s) [noise=true , address=145.79.227.37:4567 , piece_id=RKI4JEZ2D4GPGBL4J3GJZURGV62WAOZFPWXBCAVGHQSV2ULNB3CQ, seg_pos={0 0}, upload=705875]: Uploading piece... Jan 8 06:29:59.451705521 (807h0m59.322788566s) [upload=705875]: Next returned nil segment Jan 8 06:29:59.451724296 (807h0m59.322807341s) [upload=705875]: Waiting for error group managing segments... Jan 8 06:29:59.731703830 (807h0m59.602786875s) [noise=true , address=145.79.227.8:4330 , piece_id=F5SS22IRGVNXLL6CMFJ32PASAS5UKLVZR2RETHRKASOJEJB5YGXA, seg_pos={0 0}, upload=705875]: Done uploading piece. err: <nil> Jan 8 06:29:59.739107869 (807h0m59.610190904s) [noise=true , address=145.79.227.3:4149 , piece_id=VXRXX2BG6R7DPNU3RUWEGQMDHL5UHERSFE5Q2236OCZ2K3D37MCA, seg_pos={0 0}, upload=705875]: Done uploading piece. err: <nil> Jan 8 06:29:59.746157061 (807h0m59.617240096s) [noise=true , address=145.79.227.47:4266 , piece_id=NVZRQ4E7SKDL42JSEKV2OANGS7KO43JZIWTIGAMIBYHMZDZ7YJ6A, seg_pos={0 0}, upload=705875]: Done uploading piece. err: <nil> Jan 8 06:29:59.750791986 (807h0m59.621875022s) [noise=true , address=145.79.227.29:4942 , piece_id=J4N4AWCQBMH4VTOLKVBLGGGR4LTOXC34KVZHAV6JZAUO5LXQ2MLA, seg_pos={0 0}, upload=705875]: Done uploading piece. err: <nil> Jan 8 06:29:59.754874059 (807h0m59.625957094s) [noise=true , address=145.79.227.41:4086 , piece_id=J252QHNFHJK5DRA5KHUOOOQJORJCTD5UAD3EHVJE7NSUCDSA5CLQ, seg_pos={0 0}, upload=705875]: Done uploading piece. err: <nil> Jan 8 06:29:59.755794261 (807h0m59.626877296s) [noise=true , address=145.79.227.51:4411 , piece_id=YNJETDNTDH2WQG7HFKCENQRI5LY76RUKDAY3UMUIVJVYOBEUVURA, seg_pos={0 0}, upload=705875]: Done uploading piece. err: <nil> Jan 8 06:29:59.757967229 (807h0m59.629050264s) [noise=true , address=145.79.227.9:4365 , piece_id=5EZKRCKWT3HHOBBAWRNPPPHZGYA4Y5G5MO5NWDGXLZBYVOY42LQA, seg_pos={0 0}, upload=705875]: Done uploading piece. err: <nil> Jan 8 06:29:59.760120860 (807h0m59.631203905s) [noise=true , address=145.79.227.33:4871 , piece_id=CLJAXD7KEUKQPFRK3EWJ7BLUSYRSIVBYDB4SORXPU4WVEJTQSTGA, seg_pos={0 0}, upload=705875]: Done uploading piece. err: <nil> Jan 8 06:29:59.760370145 (807h0m59.63145318s) [noise=true , address=145.79.227.15:4873 , piece_id=IF45DGQ6N4O2WF77I5S7WOJ45647XEWFDCNYWJBP5ZBCKZHKU6GQ, seg_pos={0 0}, upload=705875]: Done uploading piece. err: <nil> Jan 8 06:29:59.843714465 (807h0m59.7147975s) [noise=true , address=145.79.227.37:4567 , piece_id=RKI4JEZ2D4GPGBL4J3GJZURGV62WAOZFPWXBCAVGHQSV2ULNB3CQ, seg_pos={0 0}, upload=705875]: Done uploading piece. err: <nil> Jan 8 06:30:00.014889799 (807h0m59.885972834s) [noise=true , address=77.90.31.15:4700 , piece_id=OM6ZQMPLZYDEQWUY7L3FE5FD3XS2PBMOZUNTFGO5YOEO2IVFETFA, seg_pos={0 0}, upload=705875]: Done uploading piece. err: <nil> Jan 8 06:30:00.026178233 (807h0m59.897261279s) [noise=true , address=77.90.31.21:4877 , piece_id=2VJESI7ESS3XU5RHUN3OGTD2EN5FY235OFQBE6I3HKOXSLFKXIGQ, seg_pos={0 0}, upload=705875]: Done uploading piece. err: <nil> Jan 8 06:30:00.032585304 (807h0m59.903668349s) [noise=true , address=77.90.31.19:4842 , piece_id=OS6NWVUWS7QS74BAUTBEJ4DD5DFZMGTSYZNXK47ELZDHTIC5UIZQ, seg_pos={0 0}, upload=705875]: Done uploading piece. err: <nil> Jan 8 06:30:00.032611754 (807h0m59.903694789s) [seg_pos={0 0}, upload=705875]: Segment reached optimal threshold of 13 pieces. Jan 8 06:30:00.032854577 (807h0m59.903937622s) [noise=true , address=45.91.116.4:4939 , piece_id=ERL4UXWSP7TYDR55BTYD2WFO4MX5PSTZTRXRB635NVUQZSYC6PHA, seg_pos={0 0}, upload=705875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Jan 8 06:30:00.032871659 (807h0m59.903954694s) [noise=true , address=45.91.116.4:4939 , piece_id=ERL4UXWSP7TYDR55BTYD2WFO4MX5PSTZTRXRB635NVUQZSYC6PHA, seg_pos={0 0}, upload=705875]: Long tail cancellation due to optimal threshold Jan 8 06:30:00.033056821 (807h0m59.904139856s) [noise=true , address=45.91.116.12:4225 , piece_id=QXWSWIUFYPRXCH5AAZ6PEJ37HCHDJOX4CO3IY4ICIZKXUAI6V6KQ, seg_pos={0 0}, upload=705875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Jan 8 06:30:00.033066360 (807h0m59.904149395s) [noise=true , address=45.91.116.12:4225 , piece_id=QXWSWIUFYPRXCH5AAZ6PEJ37HCHDJOX4CO3IY4ICIZKXUAI6V6KQ, seg_pos={0 0}, upload=705875]: Long tail cancellation due to optimal threshold Jan 8 06:30:00.033089764 (807h0m59.904172799s) [noise=true , address=45.91.116.6:4974 , piece_id=3QMCHIZMMV3YG5PDMC3FBJ76T6Z2GNHBEPOPZPXO6GTHPRUJGEHQ, seg_pos={0 0}, upload=705875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:51184->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:51184->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51184->45.91.116.6:4974: use of closed network connection Jan 8 06:30:00.033097680 (807h0m59.904180715s) [noise=true , address=45.91.116.6:4974 , piece_id=3QMCHIZMMV3YG5PDMC3FBJ76T6Z2GNHBEPOPZPXO6GTHPRUJGEHQ, seg_pos={0 0}, upload=705875]: Long tail cancellation due to optimal threshold Jan 8 06:30:00.033137836 (807h0m59.904220881s) [seg_pos={0 0}, upload=705875]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 8 06:30:00.033241193 (807h0m59.904324238s) [upload=705875]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 8 06:30:00.033248697 (807h0m59.904331732s) [upload=705875]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 8 06:30:00.320262835 (807h1m0.19134588s) [upload=705875]: Done uploading segments... Jan 8 06:30:00.320287482 (807h1m0.191370538s) [upload=705875]: Upload finished. err: <nil> Duration: 5.760837424s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 3.690236747s trace: https://jaeger.dev.storj.io/trace/6173eb4c5ef6f546 FAIL (2026-01-08 06:36:26 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 53) Duration: 3.690262405s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.95264236s trace: https://jaeger.dev.storj.io/trace/3738a06b2bfcba74 FAIL (2026-01-08 06:36:22 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12hkSjuH8cdS4byda1bykpe2QJz42HXaK1uRtkTbCQdynovAKSz, Piece ID: 4GTLFPYP5LTIZPMRSOVJNHNBOMLFZTIF7CD5QHSAWAQT2DQYCAEQ) context deadline exceeded; piecestore: (Node ID: 1bWLCSW1tixYzHThBQsUPvXtTpnf4egF8uGHFUXDQkCeE4BwBU, Piece ID: RX5TQUH56ZXKJADRRFT7DX6J6VV32WB3EFRCM456NZ2TD2WHKM3A) context deadline exceeded; piecestore: (Node ID: 1XxiRqM4j35HGGzWPnmrHHfbWS8jryWrW5aY66SgDuDQKHYopv, Piece ID: GAOHJL4R553KJFKTWIRCGC3NHHM4TBFNH6GJYFREM23JQQS645TQ) context deadline exceeded; piecestore: (Node ID: 15LfiPEGogvaJkFN67RXm8egaXMGxsc9XmPo26GFwT5TZKmg75, Piece ID: HD6GDZKL6A43NRG367DI674KCKPZAN4PP6S5ZRHHYR33EPAQ4KRQ) context deadline exceeded; piecestore: (Node ID: 1m2RK4RXMBjqBpWj8Xof42VU2fyPNygiY1WR3gcsVRxd6jKe4E, Piece ID: SCDRYTITO42A42357QKLW7RUWW6VKZLZLYQ3H2NIPXAKYGOIC5HQ) context deadline exceeded; piecestore: (Node ID: 1vtdYMUQ1aEzqZeSWkWD9RceYHunHdt9QwAzy7CwyYruPtSNRw, Piece ID: SYC3VAYM2YKT5ZM2VAQQC7663JAM47LYYCU62ZFRVH5UJDTTXMRQ) context deadline exceeded; piecestore: (Node ID: 1dstfvWPzYyG7oHnsByWcpgv6C4NVw9RPt8MdxtZ634FaLrSt4, Piece ID: D6G2XNNG5QPU2VSWNAP6XPRA3LM5GREUKMQG3KEWTKXFBNSEPIZA) context deadline exceeded; piecestore: (Node ID: 12Bf3H3JPRVT3GKxieKhLpk7ZfXc3FrdtweLJKRpFo6yNTomKDu, Piece ID: UD3L4QJO2DEW7V3SJ7HHPO5P6VWX4MURJKKEQ5KIQUHKAUZOZVTQ) context deadline exceeded; piecestore: (Node ID: 1ZtoenGAVBrgVoJkmkyxDqire258rdYcqC3avm8xAPuBGiD1bU, Piece ID: 4GAT5EPHUEH7UO3IHA5FU23RMY7NGYLJU2CBCX3RS2BCAPIDP7VA) context deadline exceeded; piecestore: (Node ID: 12vhu4j9fCJhbUW9rxZshXEbh9gbG1j1sPzgPhDdiD7ecVTQaHz, Piece ID: 6N3EAXOD5X75DQR6W4AHVJ54XQ7BGOF2BRN32IDBHLLYOOYI3ILQ) context deadline exceeded; piecestore: (Node ID: 1kLgnR5vmDgUuaf1aWAUyDsXuHn8aEwVXEQiruQrkfKpXqTiQ1, Piece ID: DGJRCP2CNO7DMWSJ5XMYFWV6CF22SLA4LPCVWLTBRKCQ6NV7HS7Q) context deadline exceeded Duration: 5.952671767s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.......X.X.XXX.X.....X..X.......XXX.XX.XX..X.XXX.XXXX.XXXXXX] duration: 30.001289314s trace: https://jaeger.dev.storj.io/trace/6049d41e25142448 FAIL (2026-01-08 06:36:10 UTC) orbiter warning: context deadline exceeded Duration: 30.001313851s edge gateway us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 3.100545707s trace: https://jaeger.dev.storj.io/trace/6173eb4c5ef6f546 FAIL (2026-01-08 06:36:08 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 53) Duration: 3.100573218s core uplink ap1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)