Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X...XXXXXXX..X.X..X.X.....X...X......X.X..X..XXX...........] duration: 5.728411198s trace: https://jaeger.dev.storj.io/trace/1509babc40b8daba FAIL (2025-12-27 15:43:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: 3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA) context deadline exceeded ======LOGS====== Dec 27 15:35:36.834619921 (528h6m36.705702997s) [upload=463210]: Starting upload Dec 27 15:35:36.834705574 (528h6m36.705788649s) [upload=463210]: Done starting upload Dec 27 15:35:36.834811706 (528h6m36.705894761s) [upload=463210]: Uploading segments... Dec 27 15:35:36.834963957 (528h6m36.706047002s) [upload=463210]: Got next segment. Inline: false Dec 27 15:35:36.835004414 (528h6m36.706087449s) [upload=463210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:35:36.835010325 (528h6m36.70609337s) [upload=463210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:35:37.088013461 (528h6m36.959096506s) [seg_pos={0 0}, upload=463210]: Begin upload segment... Dec 27 15:35:37.088072674 (528h6m36.959155709s) [seg_pos={0 0}, upload=463210]: Done begin upload segment. Dec 27 15:35:37.088134742 (528h6m36.959217787s) [noise=true , address=45.91.116.6:4973 , piece_id=YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.088160852 (528h6m36.959243887s) [noise=true , address=145.79.227.12:4767 , piece_id=UYPFM2ZW6CVDQXHADJ2GJBC6KX35TBHBHSKG5BBIHZPGYDAFNEZA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.089097435 (528h6m36.96018048s) [noise=true , address=145.79.227.52:4445 , piece_id=DKSWMR77ALMRSOW47G44IFI7W3HZYQWHVV4JXB5QZZEUOVPZ7QDA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.089544757 (528h6m36.960627802s) [noise=true , address=45.91.116.11:4118 , piece_id=3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090044959 (528h6m36.961128004s) [noise=true , address=145.79.227.45:4231 , piece_id=5E2OU3ID7U7JKH7MB2EHTAQZX5UZ2I2NTD6VACRSRQM2SCOV723Q, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090414994 (528h6m36.961498039s) [noise=true , address=145.79.227.49:4338 , piece_id=EP7IUXGYAF2LHM5BDTEK6HGN4O4OTPWTVYHWTYRWQ7CBLK26MZSQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090784278 (528h6m36.961867323s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.091246458 (528h6m36.962329503s) [noise=true , address=145.79.227.19:4078 , piece_id=UDFCQU6FOVIICU25EQOBN57JPCXNONYAW4Z2SQH7RBH2XSEFD2HA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.091740087 (528h6m36.962823122s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.093138921 (528h6m36.964221966s) [noise=true , address=145.79.227.10:4402 , piece_id=45HK3GSRJBMWALEJK3T4RZME4OZOJSU57JP4KXP6CHJZMSKK3Y4Q, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.093160833 (528h6m36.964243878s) [noise=true , address=145.79.227.48:4303 , piece_id=EKTFWLS3BYKVOL7NTVN7VIGWT3ET3MAPKM46RIFZK73ZQFST4J4A, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.094811165 (528h6m36.96589421s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.095010345 (528h6m36.96609338s) [noise=true , address=145.79.227.39:4015 , piece_id=B5G2XXRPZTWKSSA37SBWNV5BX3POFELY2UBRVQIPNS5LDGBH47PQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.096583099 (528h6m36.967666144s) [noise=true , address=145.79.227.3:4151 , piece_id=NTDEUAYE6X2QKXWO543TPLFMUDSHNXPFYZ42SDYNMIR7LQBIHIGQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.098185200 (528h6m36.969268235s) [noise=true , address=145.79.227.26:4655 , piece_id=CEBD7AHETY7SJ2563LQ57ZX6ALLYBTFIDSLTFOLOSRLPOBIS3CAA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.099692360 (528h6m36.970775406s) [noise=true , address=145.79.227.2:4113 , piece_id=VS5QG2XZA3YTDRHEO6ILLOH7H3GPSPGCL2NOP73CVA7VCBXMUDGA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:38.706744778 (528h6m38.577827823s) [upload=463210]: Next returned nil segment Dec 27 15:35:38.706800425 (528h6m38.57788346s) [upload=463210]: Waiting for error group managing segments... Dec 27 15:35:39.091009801 (528h6m38.962092847s) [noise=true , address=145.79.227.49:4338 , piece_id=EP7IUXGYAF2LHM5BDTEK6HGN4O4OTPWTVYHWTYRWQ7CBLK26MZSQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.091660060 (528h6m38.962743105s) [noise=true , address=145.79.227.12:4767 , piece_id=UYPFM2ZW6CVDQXHADJ2GJBC6KX35TBHBHSKG5BBIHZPGYDAFNEZA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.101083924 (528h6m38.972166959s) [noise=true , address=145.79.227.48:4303 , piece_id=EKTFWLS3BYKVOL7NTVN7VIGWT3ET3MAPKM46RIFZK73ZQFST4J4A, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.106377303 (528h6m38.977460338s) [noise=true , address=145.79.227.2:4113 , piece_id=VS5QG2XZA3YTDRHEO6ILLOH7H3GPSPGCL2NOP73CVA7VCBXMUDGA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.109339505 (528h6m38.98042254s) [noise=true , address=145.79.227.39:4015 , piece_id=B5G2XXRPZTWKSSA37SBWNV5BX3POFELY2UBRVQIPNS5LDGBH47PQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.109995724 (528h6m38.981078759s) [noise=true , address=145.79.227.10:4402 , piece_id=45HK3GSRJBMWALEJK3T4RZME4OZOJSU57JP4KXP6CHJZMSKK3Y4Q, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.112087207 (528h6m38.983170252s) [noise=true , address=145.79.227.26:4655 , piece_id=CEBD7AHETY7SJ2563LQ57ZX6ALLYBTFIDSLTFOLOSRLPOBIS3CAA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.124988133 (528h6m38.996071178s) [noise=true , address=145.79.227.45:4231 , piece_id=5E2OU3ID7U7JKH7MB2EHTAQZX5UZ2I2NTD6VACRSRQM2SCOV723Q, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.130435766 (528h6m39.001518801s) [noise=true , address=145.79.227.3:4151 , piece_id=NTDEUAYE6X2QKXWO543TPLFMUDSHNXPFYZ42SDYNMIR7LQBIHIGQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.133234746 (528h6m39.004317781s) [noise=true , address=145.79.227.52:4445 , piece_id=DKSWMR77ALMRSOW47G44IFI7W3HZYQWHVV4JXB5QZZEUOVPZ7QDA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.139036994 (528h6m39.010120039s) [noise=true , address=145.79.227.19:4078 , piece_id=UDFCQU6FOVIICU25EQOBN57JPCXNONYAW4Z2SQH7RBH2XSEFD2HA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.728671419 (528h6m39.599754454s) [noise=true , address=45.91.116.11:4118 , piece_id=3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.801455083 (528h6m39.672538128s) [noise=true , address=45.91.116.6:4973 , piece_id=YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.801481042 (528h6m39.672564087s) [seg_pos={0 0}, upload=463210]: Segment reached optimal threshold of 13 pieces. Dec 27 15:35:39.801753310 (528h6m39.672836345s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection Dec 27 15:35:39.801763870 (528h6m39.672846915s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.801890311 (528h6m39.672973356s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 15:35:39.801899779 (528h6m39.672982814s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.801963491 (528h6m39.673046526s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 27 15:35:39.801983249 (528h6m39.673066284s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.802063842 (528h6m39.673146877s) [seg_pos={0 0}, upload=463210]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:35:39.802150638 (528h6m39.673233683s) [upload=463210]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:35:39.802158653 (528h6m39.673241708s) [upload=463210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:35:40.131666538 (528h6m40.002749573s) [upload=463210]: Done uploading segments... Dec 27 15:35:40.131691745 (528h6m40.00277478s) [upload=463210]: Upload finished. err: <nil> Duration: 5.728442447s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [...XXX.X.XXXX.X.XXXX...XXXX.XXXXX...X.....X...XX.XXXX....XXX] duration: 5.00159329s trace: https://jaeger.dev.storj.io/trace/7d0cd5ff760f6a93 FAIL (2025-12-27 15:43:06 UTC) orbiter warning: context deadline exceeded Duration: 5.0016194s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..X...XXXXXXX..X.X..X.X.....X...X......X.X..X..XXX..........] duration: 5.75028504s trace: https://jaeger.dev.storj.io/trace/1509babc40b8daba FAIL (2025-12-27 15:42:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: 3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA) context deadline exceeded ======LOGS====== Dec 27 15:35:36.834619921 (528h6m36.705702997s) [upload=463210]: Starting upload Dec 27 15:35:36.834705574 (528h6m36.705788649s) [upload=463210]: Done starting upload Dec 27 15:35:36.834811706 (528h6m36.705894761s) [upload=463210]: Uploading segments... Dec 27 15:35:36.834963957 (528h6m36.706047002s) [upload=463210]: Got next segment. Inline: false Dec 27 15:35:36.835004414 (528h6m36.706087449s) [upload=463210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:35:36.835010325 (528h6m36.70609337s) [upload=463210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:35:37.088013461 (528h6m36.959096506s) [seg_pos={0 0}, upload=463210]: Begin upload segment... Dec 27 15:35:37.088072674 (528h6m36.959155709s) [seg_pos={0 0}, upload=463210]: Done begin upload segment. Dec 27 15:35:37.088134742 (528h6m36.959217787s) [noise=true , address=45.91.116.6:4973 , piece_id=YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.088160852 (528h6m36.959243887s) [noise=true , address=145.79.227.12:4767 , piece_id=UYPFM2ZW6CVDQXHADJ2GJBC6KX35TBHBHSKG5BBIHZPGYDAFNEZA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.089097435 (528h6m36.96018048s) [noise=true , address=145.79.227.52:4445 , piece_id=DKSWMR77ALMRSOW47G44IFI7W3HZYQWHVV4JXB5QZZEUOVPZ7QDA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.089544757 (528h6m36.960627802s) [noise=true , address=45.91.116.11:4118 , piece_id=3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090044959 (528h6m36.961128004s) [noise=true , address=145.79.227.45:4231 , piece_id=5E2OU3ID7U7JKH7MB2EHTAQZX5UZ2I2NTD6VACRSRQM2SCOV723Q, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090414994 (528h6m36.961498039s) [noise=true , address=145.79.227.49:4338 , piece_id=EP7IUXGYAF2LHM5BDTEK6HGN4O4OTPWTVYHWTYRWQ7CBLK26MZSQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090784278 (528h6m36.961867323s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.091246458 (528h6m36.962329503s) [noise=true , address=145.79.227.19:4078 , piece_id=UDFCQU6FOVIICU25EQOBN57JPCXNONYAW4Z2SQH7RBH2XSEFD2HA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.091740087 (528h6m36.962823122s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.093138921 (528h6m36.964221966s) [noise=true , address=145.79.227.10:4402 , piece_id=45HK3GSRJBMWALEJK3T4RZME4OZOJSU57JP4KXP6CHJZMSKK3Y4Q, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.093160833 (528h6m36.964243878s) [noise=true , address=145.79.227.48:4303 , piece_id=EKTFWLS3BYKVOL7NTVN7VIGWT3ET3MAPKM46RIFZK73ZQFST4J4A, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.094811165 (528h6m36.96589421s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.095010345 (528h6m36.96609338s) [noise=true , address=145.79.227.39:4015 , piece_id=B5G2XXRPZTWKSSA37SBWNV5BX3POFELY2UBRVQIPNS5LDGBH47PQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.096583099 (528h6m36.967666144s) [noise=true , address=145.79.227.3:4151 , piece_id=NTDEUAYE6X2QKXWO543TPLFMUDSHNXPFYZ42SDYNMIR7LQBIHIGQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.098185200 (528h6m36.969268235s) [noise=true , address=145.79.227.26:4655 , piece_id=CEBD7AHETY7SJ2563LQ57ZX6ALLYBTFIDSLTFOLOSRLPOBIS3CAA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.099692360 (528h6m36.970775406s) [noise=true , address=145.79.227.2:4113 , piece_id=VS5QG2XZA3YTDRHEO6ILLOH7H3GPSPGCL2NOP73CVA7VCBXMUDGA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:38.706744778 (528h6m38.577827823s) [upload=463210]: Next returned nil segment Dec 27 15:35:38.706800425 (528h6m38.57788346s) [upload=463210]: Waiting for error group managing segments... Dec 27 15:35:39.091009801 (528h6m38.962092847s) [noise=true , address=145.79.227.49:4338 , piece_id=EP7IUXGYAF2LHM5BDTEK6HGN4O4OTPWTVYHWTYRWQ7CBLK26MZSQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.091660060 (528h6m38.962743105s) [noise=true , address=145.79.227.12:4767 , piece_id=UYPFM2ZW6CVDQXHADJ2GJBC6KX35TBHBHSKG5BBIHZPGYDAFNEZA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.101083924 (528h6m38.972166959s) [noise=true , address=145.79.227.48:4303 , piece_id=EKTFWLS3BYKVOL7NTVN7VIGWT3ET3MAPKM46RIFZK73ZQFST4J4A, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.106377303 (528h6m38.977460338s) [noise=true , address=145.79.227.2:4113 , piece_id=VS5QG2XZA3YTDRHEO6ILLOH7H3GPSPGCL2NOP73CVA7VCBXMUDGA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.109339505 (528h6m38.98042254s) [noise=true , address=145.79.227.39:4015 , piece_id=B5G2XXRPZTWKSSA37SBWNV5BX3POFELY2UBRVQIPNS5LDGBH47PQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.109995724 (528h6m38.981078759s) [noise=true , address=145.79.227.10:4402 , piece_id=45HK3GSRJBMWALEJK3T4RZME4OZOJSU57JP4KXP6CHJZMSKK3Y4Q, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.112087207 (528h6m38.983170252s) [noise=true , address=145.79.227.26:4655 , piece_id=CEBD7AHETY7SJ2563LQ57ZX6ALLYBTFIDSLTFOLOSRLPOBIS3CAA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.124988133 (528h6m38.996071178s) [noise=true , address=145.79.227.45:4231 , piece_id=5E2OU3ID7U7JKH7MB2EHTAQZX5UZ2I2NTD6VACRSRQM2SCOV723Q, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.130435766 (528h6m39.001518801s) [noise=true , address=145.79.227.3:4151 , piece_id=NTDEUAYE6X2QKXWO543TPLFMUDSHNXPFYZ42SDYNMIR7LQBIHIGQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.133234746 (528h6m39.004317781s) [noise=true , address=145.79.227.52:4445 , piece_id=DKSWMR77ALMRSOW47G44IFI7W3HZYQWHVV4JXB5QZZEUOVPZ7QDA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.139036994 (528h6m39.010120039s) [noise=true , address=145.79.227.19:4078 , piece_id=UDFCQU6FOVIICU25EQOBN57JPCXNONYAW4Z2SQH7RBH2XSEFD2HA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.728671419 (528h6m39.599754454s) [noise=true , address=45.91.116.11:4118 , piece_id=3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.801455083 (528h6m39.672538128s) [noise=true , address=45.91.116.6:4973 , piece_id=YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.801481042 (528h6m39.672564087s) [seg_pos={0 0}, upload=463210]: Segment reached optimal threshold of 13 pieces. Dec 27 15:35:39.801753310 (528h6m39.672836345s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection Dec 27 15:35:39.801763870 (528h6m39.672846915s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.801890311 (528h6m39.672973356s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 15:35:39.801899779 (528h6m39.672982814s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.801963491 (528h6m39.673046526s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 27 15:35:39.801983249 (528h6m39.673066284s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.802063842 (528h6m39.673146877s) [seg_pos={0 0}, upload=463210]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:35:39.802150638 (528h6m39.673233683s) [upload=463210]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:35:39.802158653 (528h6m39.673241708s) [upload=463210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:35:40.131666538 (528h6m40.002749573s) [upload=463210]: Done uploading segments... Dec 27 15:35:40.131691745 (528h6m40.00277478s) [upload=463210]: Upload finished. err: <nil> Duration: 5.750303495s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [....XXX.X.XXXX.X.XXXX...XXXX.XXXXX...X.....X...XX.XXXX....XX] duration: 5.001302295s trace: https://jaeger.dev.storj.io/trace/1e0f5c742f61a3f3 FAIL (2025-12-27 15:42:34 UTC) orbiter warning: context deadline exceeded Duration: 5.001325209s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X....X...................X.....XX......X..............] duration: 53.649004539s trace: https://jaeger.dev.storj.io/trace/24120a988203c524 FAIL (2025-12-27 15:42:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 191MKpEnoarS7sK4Gs63ZM8zGc5jevFX8VeqViP82duW33nJJz, Piece ID: 6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A) context deadline exceeded; piecestore: (Node ID: 1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, Piece ID: JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA) context deadline exceeded; piecestore: (Node ID: 12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, Piece ID: L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA) context deadline exceeded ======LOGS====== GEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.918056404 (528h0m9.789139439s) [noise=true , address=77.90.31.13:4645 , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.918708286 (528h0m9.789791331s) [noise=true , address=77.90.31.12:4593 , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.919456690 (528h0m9.790539736s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920039571 (528h0m9.791122616s) [noise=true , address=145.79.227.35:4512 , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920276141 (528h0m9.791359186s) [noise=true , address=145.79.227.34:4926 , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920474529 (528h0m9.791557564s) [noise=true , address=145.79.227.13:4816 , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.921046528 (528h0m9.792129563s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.921603759 (528h0m9.792686794s) [noise=true , address=77.90.31.21:4883 , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.922032846 (528h0m9.793115881s) [noise=true , address=145.79.227.44:4210 , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.923593568 (528h0m9.794676613s) [noise=true , address=45.91.116.10:4103 , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.924048133 (528h0m9.795131169s) [noise=true , address=77.90.31.18:4840 , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.924539719 (528h0m9.795622754s) [noise=true , address=64.49.13.139:4470 , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.925024673 (528h0m9.796107708s) [noise=true , address=145.79.227.49:4368 , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.925500990 (528h0m9.796584035s) [noise=true , address=77.90.31.17:4795 , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:11.205949065 (528h0m11.0770321s) [upload=463114]: Next returned nil segment Dec 27 15:29:11.205963343 (528h0m11.077046378s) [upload=463114]: Waiting for error group managing segments... Dec 27 15:29:11.610155861 (528h0m11.481238906s) [noise=true , address=64.49.13.139:4470 , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.812270605 (528h0m11.683353641s) [noise=true , address=145.79.227.35:4512 , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.818242477 (528h0m11.689325522s) [noise=true , address=145.79.227.13:4816 , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.938211621 (528h0m11.809294666s) [noise=true , address=145.79.227.49:4368 , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.064896693 (528h0m11.935979738s) [noise=true , address=77.90.31.17:4795 , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.070731273 (528h0m11.941814308s) [noise=true , address=77.90.31.21:4883 , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.107949174 (528h0m11.979032239s) [noise=true , address=77.90.31.11:4573 , piece_id=SRXHTEXDEKKJ2WPGY66BOAP7WD7Z75HK3B5DL2XIGFWDYQGI76YA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.151478532 (528h0m12.022561567s) [noise=true , address=77.90.31.13:4645 , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.335397247 (528h0m12.206480292s) [noise=true , address=45.91.116.10:4103 , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.410397992 (528h0m12.281481027s) [noise=true , address=77.90.31.18:4840 , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.667101894 (528h0m12.538184929s) [noise=true , address=77.90.31.12:4593 , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.785380610 (528h0m12.656463655s) [noise=true , address=145.79.227.44:4210 , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.828722731 (528h0m12.699805766s) [noise=true , address=193.149.164.143:4697 , piece_id=S4CQ3366KVFS5RMZR67MPGSLXKDWRAGEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.947757466 (528h0m12.818840501s) [noise=true , address=145.79.227.34:4926 , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.947777434 (528h0m12.818860469s) [seg_pos={0 0}, upload=463114]: Segment reached optimal threshold of 14 pieces. Dec 27 15:29:12.947947177 (528h0m12.819030212s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy): protocol: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection Dec 27 15:29:12.947953169 (528h0m12.819036204s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948200800 (528h0m12.819283845s) [noise=true , address=77.90.31.19:4869 , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8): protocol: read tcp 188.166.225.170:32932->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 15:29:12.948216079 (528h0m12.819299124s) [noise=true , address=77.90.31.19:4869 , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948310158 (528h0m12.819393203s) [noise=true , address=64.49.13.169:4711 , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection Dec 27 15:29:12.948316139 (528h0m12.819399174s) [noise=true , address=64.49.13.169:4711 , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948360164 (528h0m12.819443219s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 27 15:29:12.948372617 (528h0m12.819455652s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948398968 (528h0m12.819482013s) [seg_pos={0 0}, upload=463114]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:29:12.948477878 (528h0m12.819560913s) [upload=463114]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:29:12.948483889 (528h0m12.819566934s) [upload=463114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:29:13.237238452 (528h0m13.108321498s) [upload=463114]: Done uploading segments... Dec 27 15:29:13.237262939 (528h0m13.108345984s) [upload=463114]: Upload finished. err: <nil> Duration: 53.649041972s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [.....XXX.X.XXXX.X.XXXX...XXXX.XXXXX...X.....X...XX.XXXX....X] duration: 5.001654126s trace: https://jaeger.dev.storj.io/trace/75a9d68e437be04 FAIL (2025-12-27 15:42:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001681337s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [X.....XXX.X.XXXX.X.XXXX...XXXX.XXXXX...X.....X...XX.XXXX....] duration: 3.570635335s trace: https://jaeger.dev.storj.io/trace/1d83433c359667d FAIL (2025-12-27 15:41:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.57066943s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X....X...................X.....XX......X.............] duration: 45.703612593s trace: https://jaeger.dev.storj.io/trace/24120a988203c524 FAIL (2025-12-27 15:41:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 191MKpEnoarS7sK4Gs63ZM8zGc5jevFX8VeqViP82duW33nJJz, Piece ID: 6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A) context deadline exceeded; piecestore: (Node ID: 1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, Piece ID: JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA) context deadline exceeded; piecestore: (Node ID: 12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, Piece ID: L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA) context deadline exceeded ======LOGS====== GEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.918056404 (528h0m9.789139439s) [noise=true , address=77.90.31.13:4645 , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.918708286 (528h0m9.789791331s) [noise=true , address=77.90.31.12:4593 , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.919456690 (528h0m9.790539736s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920039571 (528h0m9.791122616s) [noise=true , address=145.79.227.35:4512 , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920276141 (528h0m9.791359186s) [noise=true , address=145.79.227.34:4926 , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920474529 (528h0m9.791557564s) [noise=true , address=145.79.227.13:4816 , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.921046528 (528h0m9.792129563s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.921603759 (528h0m9.792686794s) [noise=true , address=77.90.31.21:4883 , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.922032846 (528h0m9.793115881s) [noise=true , address=145.79.227.44:4210 , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.923593568 (528h0m9.794676613s) [noise=true , address=45.91.116.10:4103 , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.924048133 (528h0m9.795131169s) [noise=true , address=77.90.31.18:4840 , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.924539719 (528h0m9.795622754s) [noise=true , address=64.49.13.139:4470 , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.925024673 (528h0m9.796107708s) [noise=true , address=145.79.227.49:4368 , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.925500990 (528h0m9.796584035s) [noise=true , address=77.90.31.17:4795 , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:11.205949065 (528h0m11.0770321s) [upload=463114]: Next returned nil segment Dec 27 15:29:11.205963343 (528h0m11.077046378s) [upload=463114]: Waiting for error group managing segments... Dec 27 15:29:11.610155861 (528h0m11.481238906s) [noise=true , address=64.49.13.139:4470 , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.812270605 (528h0m11.683353641s) [noise=true , address=145.79.227.35:4512 , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.818242477 (528h0m11.689325522s) [noise=true , address=145.79.227.13:4816 , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.938211621 (528h0m11.809294666s) [noise=true , address=145.79.227.49:4368 , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.064896693 (528h0m11.935979738s) [noise=true , address=77.90.31.17:4795 , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.070731273 (528h0m11.941814308s) [noise=true , address=77.90.31.21:4883 , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.107949174 (528h0m11.979032239s) [noise=true , address=77.90.31.11:4573 , piece_id=SRXHTEXDEKKJ2WPGY66BOAP7WD7Z75HK3B5DL2XIGFWDYQGI76YA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.151478532 (528h0m12.022561567s) [noise=true , address=77.90.31.13:4645 , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.335397247 (528h0m12.206480292s) [noise=true , address=45.91.116.10:4103 , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.410397992 (528h0m12.281481027s) [noise=true , address=77.90.31.18:4840 , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.667101894 (528h0m12.538184929s) [noise=true , address=77.90.31.12:4593 , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.785380610 (528h0m12.656463655s) [noise=true , address=145.79.227.44:4210 , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.828722731 (528h0m12.699805766s) [noise=true , address=193.149.164.143:4697 , piece_id=S4CQ3366KVFS5RMZR67MPGSLXKDWRAGEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.947757466 (528h0m12.818840501s) [noise=true , address=145.79.227.34:4926 , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.947777434 (528h0m12.818860469s) [seg_pos={0 0}, upload=463114]: Segment reached optimal threshold of 14 pieces. Dec 27 15:29:12.947947177 (528h0m12.819030212s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy): protocol: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection Dec 27 15:29:12.947953169 (528h0m12.819036204s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948200800 (528h0m12.819283845s) [noise=true , address=77.90.31.19:4869 , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8): protocol: read tcp 188.166.225.170:32932->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 15:29:12.948216079 (528h0m12.819299124s) [noise=true , address=77.90.31.19:4869 , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948310158 (528h0m12.819393203s) [noise=true , address=64.49.13.169:4711 , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection Dec 27 15:29:12.948316139 (528h0m12.819399174s) [noise=true , address=64.49.13.169:4711 , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948360164 (528h0m12.819443219s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 27 15:29:12.948372617 (528h0m12.819455652s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948398968 (528h0m12.819482013s) [seg_pos={0 0}, upload=463114]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:29:12.948477878 (528h0m12.819560913s) [upload=463114]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:29:12.948483889 (528h0m12.819566934s) [upload=463114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:29:13.237238452 (528h0m13.108321498s) [upload=463114]: Done uploading segments... Dec 27 15:29:13.237262939 (528h0m13.108345984s) [upload=463114]: Upload finished. err: <nil> Duration: 45.703639604s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X..X...XXXXXXX..X.X..X.X.....X...X......X.X..X..XXX.........] duration: 5.922826427s trace: https://jaeger.dev.storj.io/trace/1509babc40b8daba FAIL (2025-12-27 15:41:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: 3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA) context deadline exceeded ======LOGS====== Dec 27 15:35:36.834619921 (528h6m36.705702997s) [upload=463210]: Starting upload Dec 27 15:35:36.834705574 (528h6m36.705788649s) [upload=463210]: Done starting upload Dec 27 15:35:36.834811706 (528h6m36.705894761s) [upload=463210]: Uploading segments... Dec 27 15:35:36.834963957 (528h6m36.706047002s) [upload=463210]: Got next segment. Inline: false Dec 27 15:35:36.835004414 (528h6m36.706087449s) [upload=463210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:35:36.835010325 (528h6m36.70609337s) [upload=463210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:35:37.088013461 (528h6m36.959096506s) [seg_pos={0 0}, upload=463210]: Begin upload segment... Dec 27 15:35:37.088072674 (528h6m36.959155709s) [seg_pos={0 0}, upload=463210]: Done begin upload segment. Dec 27 15:35:37.088134742 (528h6m36.959217787s) [noise=true , address=45.91.116.6:4973 , piece_id=YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.088160852 (528h6m36.959243887s) [noise=true , address=145.79.227.12:4767 , piece_id=UYPFM2ZW6CVDQXHADJ2GJBC6KX35TBHBHSKG5BBIHZPGYDAFNEZA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.089097435 (528h6m36.96018048s) [noise=true , address=145.79.227.52:4445 , piece_id=DKSWMR77ALMRSOW47G44IFI7W3HZYQWHVV4JXB5QZZEUOVPZ7QDA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.089544757 (528h6m36.960627802s) [noise=true , address=45.91.116.11:4118 , piece_id=3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090044959 (528h6m36.961128004s) [noise=true , address=145.79.227.45:4231 , piece_id=5E2OU3ID7U7JKH7MB2EHTAQZX5UZ2I2NTD6VACRSRQM2SCOV723Q, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090414994 (528h6m36.961498039s) [noise=true , address=145.79.227.49:4338 , piece_id=EP7IUXGYAF2LHM5BDTEK6HGN4O4OTPWTVYHWTYRWQ7CBLK26MZSQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090784278 (528h6m36.961867323s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.091246458 (528h6m36.962329503s) [noise=true , address=145.79.227.19:4078 , piece_id=UDFCQU6FOVIICU25EQOBN57JPCXNONYAW4Z2SQH7RBH2XSEFD2HA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.091740087 (528h6m36.962823122s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.093138921 (528h6m36.964221966s) [noise=true , address=145.79.227.10:4402 , piece_id=45HK3GSRJBMWALEJK3T4RZME4OZOJSU57JP4KXP6CHJZMSKK3Y4Q, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.093160833 (528h6m36.964243878s) [noise=true , address=145.79.227.48:4303 , piece_id=EKTFWLS3BYKVOL7NTVN7VIGWT3ET3MAPKM46RIFZK73ZQFST4J4A, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.094811165 (528h6m36.96589421s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.095010345 (528h6m36.96609338s) [noise=true , address=145.79.227.39:4015 , piece_id=B5G2XXRPZTWKSSA37SBWNV5BX3POFELY2UBRVQIPNS5LDGBH47PQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.096583099 (528h6m36.967666144s) [noise=true , address=145.79.227.3:4151 , piece_id=NTDEUAYE6X2QKXWO543TPLFMUDSHNXPFYZ42SDYNMIR7LQBIHIGQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.098185200 (528h6m36.969268235s) [noise=true , address=145.79.227.26:4655 , piece_id=CEBD7AHETY7SJ2563LQ57ZX6ALLYBTFIDSLTFOLOSRLPOBIS3CAA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.099692360 (528h6m36.970775406s) [noise=true , address=145.79.227.2:4113 , piece_id=VS5QG2XZA3YTDRHEO6ILLOH7H3GPSPGCL2NOP73CVA7VCBXMUDGA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:38.706744778 (528h6m38.577827823s) [upload=463210]: Next returned nil segment Dec 27 15:35:38.706800425 (528h6m38.57788346s) [upload=463210]: Waiting for error group managing segments... Dec 27 15:35:39.091009801 (528h6m38.962092847s) [noise=true , address=145.79.227.49:4338 , piece_id=EP7IUXGYAF2LHM5BDTEK6HGN4O4OTPWTVYHWTYRWQ7CBLK26MZSQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.091660060 (528h6m38.962743105s) [noise=true , address=145.79.227.12:4767 , piece_id=UYPFM2ZW6CVDQXHADJ2GJBC6KX35TBHBHSKG5BBIHZPGYDAFNEZA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.101083924 (528h6m38.972166959s) [noise=true , address=145.79.227.48:4303 , piece_id=EKTFWLS3BYKVOL7NTVN7VIGWT3ET3MAPKM46RIFZK73ZQFST4J4A, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.106377303 (528h6m38.977460338s) [noise=true , address=145.79.227.2:4113 , piece_id=VS5QG2XZA3YTDRHEO6ILLOH7H3GPSPGCL2NOP73CVA7VCBXMUDGA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.109339505 (528h6m38.98042254s) [noise=true , address=145.79.227.39:4015 , piece_id=B5G2XXRPZTWKSSA37SBWNV5BX3POFELY2UBRVQIPNS5LDGBH47PQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.109995724 (528h6m38.981078759s) [noise=true , address=145.79.227.10:4402 , piece_id=45HK3GSRJBMWALEJK3T4RZME4OZOJSU57JP4KXP6CHJZMSKK3Y4Q, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.112087207 (528h6m38.983170252s) [noise=true , address=145.79.227.26:4655 , piece_id=CEBD7AHETY7SJ2563LQ57ZX6ALLYBTFIDSLTFOLOSRLPOBIS3CAA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.124988133 (528h6m38.996071178s) [noise=true , address=145.79.227.45:4231 , piece_id=5E2OU3ID7U7JKH7MB2EHTAQZX5UZ2I2NTD6VACRSRQM2SCOV723Q, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.130435766 (528h6m39.001518801s) [noise=true , address=145.79.227.3:4151 , piece_id=NTDEUAYE6X2QKXWO543TPLFMUDSHNXPFYZ42SDYNMIR7LQBIHIGQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.133234746 (528h6m39.004317781s) [noise=true , address=145.79.227.52:4445 , piece_id=DKSWMR77ALMRSOW47G44IFI7W3HZYQWHVV4JXB5QZZEUOVPZ7QDA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.139036994 (528h6m39.010120039s) [noise=true , address=145.79.227.19:4078 , piece_id=UDFCQU6FOVIICU25EQOBN57JPCXNONYAW4Z2SQH7RBH2XSEFD2HA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.728671419 (528h6m39.599754454s) [noise=true , address=45.91.116.11:4118 , piece_id=3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.801455083 (528h6m39.672538128s) [noise=true , address=45.91.116.6:4973 , piece_id=YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.801481042 (528h6m39.672564087s) [seg_pos={0 0}, upload=463210]: Segment reached optimal threshold of 13 pieces. Dec 27 15:35:39.801753310 (528h6m39.672836345s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection Dec 27 15:35:39.801763870 (528h6m39.672846915s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.801890311 (528h6m39.672973356s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 15:35:39.801899779 (528h6m39.672982814s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.801963491 (528h6m39.673046526s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 27 15:35:39.801983249 (528h6m39.673066284s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.802063842 (528h6m39.673146877s) [seg_pos={0 0}, upload=463210]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:35:39.802150638 (528h6m39.673233683s) [upload=463210]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:35:39.802158653 (528h6m39.673241708s) [upload=463210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:35:40.131666538 (528h6m40.002749573s) [upload=463210]: Done uploading segments... Dec 27 15:35:40.131691745 (528h6m40.00277478s) [upload=463210]: Upload finished. err: <nil> Duration: 5.922854562s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [XX.....XXX.X.XXXX.X.XXXX...XXXX.XXXXX...X.....X...XX.XXXX...] duration: 5.001168259s trace: https://jaeger.dev.storj.io/trace/1d83433c359667d FAIL (2025-12-27 15:40:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001200892s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [........X....X...................X.....XX......X............] duration: 50.006997652s trace: https://jaeger.dev.storj.io/trace/24120a988203c524 FAIL (2025-12-27 15:40:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 191MKpEnoarS7sK4Gs63ZM8zGc5jevFX8VeqViP82duW33nJJz, Piece ID: 6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A) context deadline exceeded; piecestore: (Node ID: 1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, Piece ID: JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA) context deadline exceeded; piecestore: (Node ID: 12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, Piece ID: L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA) context deadline exceeded ======LOGS====== GEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.918056404 (528h0m9.789139439s) [noise=true , address=77.90.31.13:4645 , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.918708286 (528h0m9.789791331s) [noise=true , address=77.90.31.12:4593 , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.919456690 (528h0m9.790539736s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920039571 (528h0m9.791122616s) [noise=true , address=145.79.227.35:4512 , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920276141 (528h0m9.791359186s) [noise=true , address=145.79.227.34:4926 , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920474529 (528h0m9.791557564s) [noise=true , address=145.79.227.13:4816 , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.921046528 (528h0m9.792129563s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.921603759 (528h0m9.792686794s) [noise=true , address=77.90.31.21:4883 , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.922032846 (528h0m9.793115881s) [noise=true , address=145.79.227.44:4210 , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.923593568 (528h0m9.794676613s) [noise=true , address=45.91.116.10:4103 , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.924048133 (528h0m9.795131169s) [noise=true , address=77.90.31.18:4840 , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.924539719 (528h0m9.795622754s) [noise=true , address=64.49.13.139:4470 , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.925024673 (528h0m9.796107708s) [noise=true , address=145.79.227.49:4368 , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.925500990 (528h0m9.796584035s) [noise=true , address=77.90.31.17:4795 , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:11.205949065 (528h0m11.0770321s) [upload=463114]: Next returned nil segment Dec 27 15:29:11.205963343 (528h0m11.077046378s) [upload=463114]: Waiting for error group managing segments... Dec 27 15:29:11.610155861 (528h0m11.481238906s) [noise=true , address=64.49.13.139:4470 , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.812270605 (528h0m11.683353641s) [noise=true , address=145.79.227.35:4512 , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.818242477 (528h0m11.689325522s) [noise=true , address=145.79.227.13:4816 , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.938211621 (528h0m11.809294666s) [noise=true , address=145.79.227.49:4368 , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.064896693 (528h0m11.935979738s) [noise=true , address=77.90.31.17:4795 , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.070731273 (528h0m11.941814308s) [noise=true , address=77.90.31.21:4883 , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.107949174 (528h0m11.979032239s) [noise=true , address=77.90.31.11:4573 , piece_id=SRXHTEXDEKKJ2WPGY66BOAP7WD7Z75HK3B5DL2XIGFWDYQGI76YA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.151478532 (528h0m12.022561567s) [noise=true , address=77.90.31.13:4645 , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.335397247 (528h0m12.206480292s) [noise=true , address=45.91.116.10:4103 , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.410397992 (528h0m12.281481027s) [noise=true , address=77.90.31.18:4840 , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.667101894 (528h0m12.538184929s) [noise=true , address=77.90.31.12:4593 , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.785380610 (528h0m12.656463655s) [noise=true , address=145.79.227.44:4210 , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.828722731 (528h0m12.699805766s) [noise=true , address=193.149.164.143:4697 , piece_id=S4CQ3366KVFS5RMZR67MPGSLXKDWRAGEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.947757466 (528h0m12.818840501s) [noise=true , address=145.79.227.34:4926 , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.947777434 (528h0m12.818860469s) [seg_pos={0 0}, upload=463114]: Segment reached optimal threshold of 14 pieces. Dec 27 15:29:12.947947177 (528h0m12.819030212s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy): protocol: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection Dec 27 15:29:12.947953169 (528h0m12.819036204s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948200800 (528h0m12.819283845s) [noise=true , address=77.90.31.19:4869 , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8): protocol: read tcp 188.166.225.170:32932->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 15:29:12.948216079 (528h0m12.819299124s) [noise=true , address=77.90.31.19:4869 , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948310158 (528h0m12.819393203s) [noise=true , address=64.49.13.169:4711 , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection Dec 27 15:29:12.948316139 (528h0m12.819399174s) [noise=true , address=64.49.13.169:4711 , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948360164 (528h0m12.819443219s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 27 15:29:12.948372617 (528h0m12.819455652s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948398968 (528h0m12.819482013s) [seg_pos={0 0}, upload=463114]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:29:12.948477878 (528h0m12.819560913s) [upload=463114]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:29:12.948483889 (528h0m12.819566934s) [upload=463114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:29:13.237238452 (528h0m13.108321498s) [upload=463114]: Done uploading segments... Dec 27 15:29:13.237262939 (528h0m13.108345984s) [upload=463114]: Upload finished. err: <nil> Duration: 50.007015105s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X..X...XXXXXXX..X.X..X.X.....X...X......X.X..X..XXX........] duration: 8.348167985s trace: https://jaeger.dev.storj.io/trace/1509babc40b8daba FAIL (2025-12-27 15:40:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: 3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA) context deadline exceeded ======LOGS====== Dec 27 15:35:36.834619921 (528h6m36.705702997s) [upload=463210]: Starting upload Dec 27 15:35:36.834705574 (528h6m36.705788649s) [upload=463210]: Done starting upload Dec 27 15:35:36.834811706 (528h6m36.705894761s) [upload=463210]: Uploading segments... Dec 27 15:35:36.834963957 (528h6m36.706047002s) [upload=463210]: Got next segment. Inline: false Dec 27 15:35:36.835004414 (528h6m36.706087449s) [upload=463210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:35:36.835010325 (528h6m36.70609337s) [upload=463210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:35:37.088013461 (528h6m36.959096506s) [seg_pos={0 0}, upload=463210]: Begin upload segment... Dec 27 15:35:37.088072674 (528h6m36.959155709s) [seg_pos={0 0}, upload=463210]: Done begin upload segment. Dec 27 15:35:37.088134742 (528h6m36.959217787s) [noise=true , address=45.91.116.6:4973 , piece_id=YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.088160852 (528h6m36.959243887s) [noise=true , address=145.79.227.12:4767 , piece_id=UYPFM2ZW6CVDQXHADJ2GJBC6KX35TBHBHSKG5BBIHZPGYDAFNEZA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.089097435 (528h6m36.96018048s) [noise=true , address=145.79.227.52:4445 , piece_id=DKSWMR77ALMRSOW47G44IFI7W3HZYQWHVV4JXB5QZZEUOVPZ7QDA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.089544757 (528h6m36.960627802s) [noise=true , address=45.91.116.11:4118 , piece_id=3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090044959 (528h6m36.961128004s) [noise=true , address=145.79.227.45:4231 , piece_id=5E2OU3ID7U7JKH7MB2EHTAQZX5UZ2I2NTD6VACRSRQM2SCOV723Q, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090414994 (528h6m36.961498039s) [noise=true , address=145.79.227.49:4338 , piece_id=EP7IUXGYAF2LHM5BDTEK6HGN4O4OTPWTVYHWTYRWQ7CBLK26MZSQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090784278 (528h6m36.961867323s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.091246458 (528h6m36.962329503s) [noise=true , address=145.79.227.19:4078 , piece_id=UDFCQU6FOVIICU25EQOBN57JPCXNONYAW4Z2SQH7RBH2XSEFD2HA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.091740087 (528h6m36.962823122s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.093138921 (528h6m36.964221966s) [noise=true , address=145.79.227.10:4402 , piece_id=45HK3GSRJBMWALEJK3T4RZME4OZOJSU57JP4KXP6CHJZMSKK3Y4Q, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.093160833 (528h6m36.964243878s) [noise=true , address=145.79.227.48:4303 , piece_id=EKTFWLS3BYKVOL7NTVN7VIGWT3ET3MAPKM46RIFZK73ZQFST4J4A, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.094811165 (528h6m36.96589421s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.095010345 (528h6m36.96609338s) [noise=true , address=145.79.227.39:4015 , piece_id=B5G2XXRPZTWKSSA37SBWNV5BX3POFELY2UBRVQIPNS5LDGBH47PQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.096583099 (528h6m36.967666144s) [noise=true , address=145.79.227.3:4151 , piece_id=NTDEUAYE6X2QKXWO543TPLFMUDSHNXPFYZ42SDYNMIR7LQBIHIGQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.098185200 (528h6m36.969268235s) [noise=true , address=145.79.227.26:4655 , piece_id=CEBD7AHETY7SJ2563LQ57ZX6ALLYBTFIDSLTFOLOSRLPOBIS3CAA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.099692360 (528h6m36.970775406s) [noise=true , address=145.79.227.2:4113 , piece_id=VS5QG2XZA3YTDRHEO6ILLOH7H3GPSPGCL2NOP73CVA7VCBXMUDGA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:38.706744778 (528h6m38.577827823s) [upload=463210]: Next returned nil segment Dec 27 15:35:38.706800425 (528h6m38.57788346s) [upload=463210]: Waiting for error group managing segments... Dec 27 15:35:39.091009801 (528h6m38.962092847s) [noise=true , address=145.79.227.49:4338 , piece_id=EP7IUXGYAF2LHM5BDTEK6HGN4O4OTPWTVYHWTYRWQ7CBLK26MZSQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.091660060 (528h6m38.962743105s) [noise=true , address=145.79.227.12:4767 , piece_id=UYPFM2ZW6CVDQXHADJ2GJBC6KX35TBHBHSKG5BBIHZPGYDAFNEZA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.101083924 (528h6m38.972166959s) [noise=true , address=145.79.227.48:4303 , piece_id=EKTFWLS3BYKVOL7NTVN7VIGWT3ET3MAPKM46RIFZK73ZQFST4J4A, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.106377303 (528h6m38.977460338s) [noise=true , address=145.79.227.2:4113 , piece_id=VS5QG2XZA3YTDRHEO6ILLOH7H3GPSPGCL2NOP73CVA7VCBXMUDGA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.109339505 (528h6m38.98042254s) [noise=true , address=145.79.227.39:4015 , piece_id=B5G2XXRPZTWKSSA37SBWNV5BX3POFELY2UBRVQIPNS5LDGBH47PQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.109995724 (528h6m38.981078759s) [noise=true , address=145.79.227.10:4402 , piece_id=45HK3GSRJBMWALEJK3T4RZME4OZOJSU57JP4KXP6CHJZMSKK3Y4Q, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.112087207 (528h6m38.983170252s) [noise=true , address=145.79.227.26:4655 , piece_id=CEBD7AHETY7SJ2563LQ57ZX6ALLYBTFIDSLTFOLOSRLPOBIS3CAA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.124988133 (528h6m38.996071178s) [noise=true , address=145.79.227.45:4231 , piece_id=5E2OU3ID7U7JKH7MB2EHTAQZX5UZ2I2NTD6VACRSRQM2SCOV723Q, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.130435766 (528h6m39.001518801s) [noise=true , address=145.79.227.3:4151 , piece_id=NTDEUAYE6X2QKXWO543TPLFMUDSHNXPFYZ42SDYNMIR7LQBIHIGQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.133234746 (528h6m39.004317781s) [noise=true , address=145.79.227.52:4445 , piece_id=DKSWMR77ALMRSOW47G44IFI7W3HZYQWHVV4JXB5QZZEUOVPZ7QDA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.139036994 (528h6m39.010120039s) [noise=true , address=145.79.227.19:4078 , piece_id=UDFCQU6FOVIICU25EQOBN57JPCXNONYAW4Z2SQH7RBH2XSEFD2HA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.728671419 (528h6m39.599754454s) [noise=true , address=45.91.116.11:4118 , piece_id=3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.801455083 (528h6m39.672538128s) [noise=true , address=45.91.116.6:4973 , piece_id=YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.801481042 (528h6m39.672564087s) [seg_pos={0 0}, upload=463210]: Segment reached optimal threshold of 13 pieces. Dec 27 15:35:39.801753310 (528h6m39.672836345s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection Dec 27 15:35:39.801763870 (528h6m39.672846915s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.801890311 (528h6m39.672973356s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 15:35:39.801899779 (528h6m39.672982814s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.801963491 (528h6m39.673046526s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 27 15:35:39.801983249 (528h6m39.673066284s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.802063842 (528h6m39.673146877s) [seg_pos={0 0}, upload=463210]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:35:39.802150638 (528h6m39.673233683s) [upload=463210]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:35:39.802158653 (528h6m39.673241708s) [upload=463210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:35:40.131666538 (528h6m40.002749573s) [upload=463210]: Done uploading segments... Dec 27 15:35:40.131691745 (528h6m40.00277478s) [upload=463210]: Upload finished. err: <nil> Duration: 8.348202149s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [XXX.....XXX.X.XXXX.X.XXXX...XXXX.XXXXX...X.....X...XX.XXXX..] duration: 5.001235798s trace: https://jaeger.dev.storj.io/trace/1d83433c359667d FAIL (2025-12-27 15:40:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001268731s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.X..X...XXXXXXX..X.X..X.X.....X...X......X.X..X..XXX.......] duration: 5.642725938s trace: https://jaeger.dev.storj.io/trace/1509babc40b8daba FAIL (2025-12-27 15:39:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: 3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA) context deadline exceeded ======LOGS====== Dec 27 15:35:36.834619921 (528h6m36.705702997s) [upload=463210]: Starting upload Dec 27 15:35:36.834705574 (528h6m36.705788649s) [upload=463210]: Done starting upload Dec 27 15:35:36.834811706 (528h6m36.705894761s) [upload=463210]: Uploading segments... Dec 27 15:35:36.834963957 (528h6m36.706047002s) [upload=463210]: Got next segment. Inline: false Dec 27 15:35:36.835004414 (528h6m36.706087449s) [upload=463210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:35:36.835010325 (528h6m36.70609337s) [upload=463210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:35:37.088013461 (528h6m36.959096506s) [seg_pos={0 0}, upload=463210]: Begin upload segment... Dec 27 15:35:37.088072674 (528h6m36.959155709s) [seg_pos={0 0}, upload=463210]: Done begin upload segment. Dec 27 15:35:37.088134742 (528h6m36.959217787s) [noise=true , address=45.91.116.6:4973 , piece_id=YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.088160852 (528h6m36.959243887s) [noise=true , address=145.79.227.12:4767 , piece_id=UYPFM2ZW6CVDQXHADJ2GJBC6KX35TBHBHSKG5BBIHZPGYDAFNEZA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.089097435 (528h6m36.96018048s) [noise=true , address=145.79.227.52:4445 , piece_id=DKSWMR77ALMRSOW47G44IFI7W3HZYQWHVV4JXB5QZZEUOVPZ7QDA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.089544757 (528h6m36.960627802s) [noise=true , address=45.91.116.11:4118 , piece_id=3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090044959 (528h6m36.961128004s) [noise=true , address=145.79.227.45:4231 , piece_id=5E2OU3ID7U7JKH7MB2EHTAQZX5UZ2I2NTD6VACRSRQM2SCOV723Q, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090414994 (528h6m36.961498039s) [noise=true , address=145.79.227.49:4338 , piece_id=EP7IUXGYAF2LHM5BDTEK6HGN4O4OTPWTVYHWTYRWQ7CBLK26MZSQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090784278 (528h6m36.961867323s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.091246458 (528h6m36.962329503s) [noise=true , address=145.79.227.19:4078 , piece_id=UDFCQU6FOVIICU25EQOBN57JPCXNONYAW4Z2SQH7RBH2XSEFD2HA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.091740087 (528h6m36.962823122s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.093138921 (528h6m36.964221966s) [noise=true , address=145.79.227.10:4402 , piece_id=45HK3GSRJBMWALEJK3T4RZME4OZOJSU57JP4KXP6CHJZMSKK3Y4Q, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.093160833 (528h6m36.964243878s) [noise=true , address=145.79.227.48:4303 , piece_id=EKTFWLS3BYKVOL7NTVN7VIGWT3ET3MAPKM46RIFZK73ZQFST4J4A, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.094811165 (528h6m36.96589421s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.095010345 (528h6m36.96609338s) [noise=true , address=145.79.227.39:4015 , piece_id=B5G2XXRPZTWKSSA37SBWNV5BX3POFELY2UBRVQIPNS5LDGBH47PQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.096583099 (528h6m36.967666144s) [noise=true , address=145.79.227.3:4151 , piece_id=NTDEUAYE6X2QKXWO543TPLFMUDSHNXPFYZ42SDYNMIR7LQBIHIGQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.098185200 (528h6m36.969268235s) [noise=true , address=145.79.227.26:4655 , piece_id=CEBD7AHETY7SJ2563LQ57ZX6ALLYBTFIDSLTFOLOSRLPOBIS3CAA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.099692360 (528h6m36.970775406s) [noise=true , address=145.79.227.2:4113 , piece_id=VS5QG2XZA3YTDRHEO6ILLOH7H3GPSPGCL2NOP73CVA7VCBXMUDGA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:38.706744778 (528h6m38.577827823s) [upload=463210]: Next returned nil segment Dec 27 15:35:38.706800425 (528h6m38.57788346s) [upload=463210]: Waiting for error group managing segments... Dec 27 15:35:39.091009801 (528h6m38.962092847s) [noise=true , address=145.79.227.49:4338 , piece_id=EP7IUXGYAF2LHM5BDTEK6HGN4O4OTPWTVYHWTYRWQ7CBLK26MZSQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.091660060 (528h6m38.962743105s) [noise=true , address=145.79.227.12:4767 , piece_id=UYPFM2ZW6CVDQXHADJ2GJBC6KX35TBHBHSKG5BBIHZPGYDAFNEZA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.101083924 (528h6m38.972166959s) [noise=true , address=145.79.227.48:4303 , piece_id=EKTFWLS3BYKVOL7NTVN7VIGWT3ET3MAPKM46RIFZK73ZQFST4J4A, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.106377303 (528h6m38.977460338s) [noise=true , address=145.79.227.2:4113 , piece_id=VS5QG2XZA3YTDRHEO6ILLOH7H3GPSPGCL2NOP73CVA7VCBXMUDGA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.109339505 (528h6m38.98042254s) [noise=true , address=145.79.227.39:4015 , piece_id=B5G2XXRPZTWKSSA37SBWNV5BX3POFELY2UBRVQIPNS5LDGBH47PQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.109995724 (528h6m38.981078759s) [noise=true , address=145.79.227.10:4402 , piece_id=45HK3GSRJBMWALEJK3T4RZME4OZOJSU57JP4KXP6CHJZMSKK3Y4Q, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.112087207 (528h6m38.983170252s) [noise=true , address=145.79.227.26:4655 , piece_id=CEBD7AHETY7SJ2563LQ57ZX6ALLYBTFIDSLTFOLOSRLPOBIS3CAA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.124988133 (528h6m38.996071178s) [noise=true , address=145.79.227.45:4231 , piece_id=5E2OU3ID7U7JKH7MB2EHTAQZX5UZ2I2NTD6VACRSRQM2SCOV723Q, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.130435766 (528h6m39.001518801s) [noise=true , address=145.79.227.3:4151 , piece_id=NTDEUAYE6X2QKXWO543TPLFMUDSHNXPFYZ42SDYNMIR7LQBIHIGQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.133234746 (528h6m39.004317781s) [noise=true , address=145.79.227.52:4445 , piece_id=DKSWMR77ALMRSOW47G44IFI7W3HZYQWHVV4JXB5QZZEUOVPZ7QDA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.139036994 (528h6m39.010120039s) [noise=true , address=145.79.227.19:4078 , piece_id=UDFCQU6FOVIICU25EQOBN57JPCXNONYAW4Z2SQH7RBH2XSEFD2HA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.728671419 (528h6m39.599754454s) [noise=true , address=45.91.116.11:4118 , piece_id=3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.801455083 (528h6m39.672538128s) [noise=true , address=45.91.116.6:4973 , piece_id=YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.801481042 (528h6m39.672564087s) [seg_pos={0 0}, upload=463210]: Segment reached optimal threshold of 13 pieces. Dec 27 15:35:39.801753310 (528h6m39.672836345s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection Dec 27 15:35:39.801763870 (528h6m39.672846915s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.801890311 (528h6m39.672973356s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 15:35:39.801899779 (528h6m39.672982814s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.801963491 (528h6m39.673046526s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 27 15:35:39.801983249 (528h6m39.673066284s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.802063842 (528h6m39.673146877s) [seg_pos={0 0}, upload=463210]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:35:39.802150638 (528h6m39.673233683s) [upload=463210]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:35:39.802158653 (528h6m39.673241708s) [upload=463210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:35:40.131666538 (528h6m40.002749573s) [upload=463210]: Done uploading segments... Dec 27 15:35:40.131691745 (528h6m40.00277478s) [upload=463210]: Upload finished. err: <nil> Duration: 5.642829905s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........X....X...................X.....XX......X...........] duration: 46.188238125s trace: https://jaeger.dev.storj.io/trace/24120a988203c524 FAIL (2025-12-27 15:39:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 191MKpEnoarS7sK4Gs63ZM8zGc5jevFX8VeqViP82duW33nJJz, Piece ID: 6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A) context deadline exceeded; piecestore: (Node ID: 1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, Piece ID: JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA) context deadline exceeded; piecestore: (Node ID: 12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, Piece ID: L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA) context deadline exceeded ======LOGS====== GEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.918056404 (528h0m9.789139439s) [noise=true , address=77.90.31.13:4645 , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.918708286 (528h0m9.789791331s) [noise=true , address=77.90.31.12:4593 , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.919456690 (528h0m9.790539736s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920039571 (528h0m9.791122616s) [noise=true , address=145.79.227.35:4512 , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920276141 (528h0m9.791359186s) [noise=true , address=145.79.227.34:4926 , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920474529 (528h0m9.791557564s) [noise=true , address=145.79.227.13:4816 , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.921046528 (528h0m9.792129563s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.921603759 (528h0m9.792686794s) [noise=true , address=77.90.31.21:4883 , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.922032846 (528h0m9.793115881s) [noise=true , address=145.79.227.44:4210 , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.923593568 (528h0m9.794676613s) [noise=true , address=45.91.116.10:4103 , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.924048133 (528h0m9.795131169s) [noise=true , address=77.90.31.18:4840 , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.924539719 (528h0m9.795622754s) [noise=true , address=64.49.13.139:4470 , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.925024673 (528h0m9.796107708s) [noise=true , address=145.79.227.49:4368 , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.925500990 (528h0m9.796584035s) [noise=true , address=77.90.31.17:4795 , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:11.205949065 (528h0m11.0770321s) [upload=463114]: Next returned nil segment Dec 27 15:29:11.205963343 (528h0m11.077046378s) [upload=463114]: Waiting for error group managing segments... Dec 27 15:29:11.610155861 (528h0m11.481238906s) [noise=true , address=64.49.13.139:4470 , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.812270605 (528h0m11.683353641s) [noise=true , address=145.79.227.35:4512 , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.818242477 (528h0m11.689325522s) [noise=true , address=145.79.227.13:4816 , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.938211621 (528h0m11.809294666s) [noise=true , address=145.79.227.49:4368 , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.064896693 (528h0m11.935979738s) [noise=true , address=77.90.31.17:4795 , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.070731273 (528h0m11.941814308s) [noise=true , address=77.90.31.21:4883 , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.107949174 (528h0m11.979032239s) [noise=true , address=77.90.31.11:4573 , piece_id=SRXHTEXDEKKJ2WPGY66BOAP7WD7Z75HK3B5DL2XIGFWDYQGI76YA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.151478532 (528h0m12.022561567s) [noise=true , address=77.90.31.13:4645 , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.335397247 (528h0m12.206480292s) [noise=true , address=45.91.116.10:4103 , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.410397992 (528h0m12.281481027s) [noise=true , address=77.90.31.18:4840 , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.667101894 (528h0m12.538184929s) [noise=true , address=77.90.31.12:4593 , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.785380610 (528h0m12.656463655s) [noise=true , address=145.79.227.44:4210 , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.828722731 (528h0m12.699805766s) [noise=true , address=193.149.164.143:4697 , piece_id=S4CQ3366KVFS5RMZR67MPGSLXKDWRAGEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.947757466 (528h0m12.818840501s) [noise=true , address=145.79.227.34:4926 , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.947777434 (528h0m12.818860469s) [seg_pos={0 0}, upload=463114]: Segment reached optimal threshold of 14 pieces. Dec 27 15:29:12.947947177 (528h0m12.819030212s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy): protocol: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection Dec 27 15:29:12.947953169 (528h0m12.819036204s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948200800 (528h0m12.819283845s) [noise=true , address=77.90.31.19:4869 , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8): protocol: read tcp 188.166.225.170:32932->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 15:29:12.948216079 (528h0m12.819299124s) [noise=true , address=77.90.31.19:4869 , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948310158 (528h0m12.819393203s) [noise=true , address=64.49.13.169:4711 , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection Dec 27 15:29:12.948316139 (528h0m12.819399174s) [noise=true , address=64.49.13.169:4711 , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948360164 (528h0m12.819443219s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 27 15:29:12.948372617 (528h0m12.819455652s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948398968 (528h0m12.819482013s) [seg_pos={0 0}, upload=463114]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:29:12.948477878 (528h0m12.819560913s) [upload=463114]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:29:12.948483889 (528h0m12.819566934s) [upload=463114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:29:13.237238452 (528h0m13.108321498s) [upload=463114]: Done uploading segments... Dec 27 15:29:13.237262939 (528h0m13.108345984s) [upload=463114]: Upload finished. err: <nil> Duration: 46.18826738s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XXXXX...X.....X...XX.XXXX.] duration: 4.288473084s trace: https://jaeger.dev.storj.io/trace/1d83433c359667d FAIL (2025-12-27 15:39:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.288499694s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XXXXX...X.....X...XX.XXXX] duration: 5.002092279s trace: https://jaeger.dev.storj.io/trace/1d83433c359667d FAIL (2025-12-27 15:39:05 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002116065s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........X....X...................X.....XX......X..........] duration: 50.411016364s trace: https://jaeger.dev.storj.io/trace/24120a988203c524 FAIL (2025-12-27 15:38:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 191MKpEnoarS7sK4Gs63ZM8zGc5jevFX8VeqViP82duW33nJJz, Piece ID: 6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A) context deadline exceeded; piecestore: (Node ID: 1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, Piece ID: JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA) context deadline exceeded; piecestore: (Node ID: 12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, Piece ID: L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA) context deadline exceeded ======LOGS====== GEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.918056404 (528h0m9.789139439s) [noise=true , address=77.90.31.13:4645 , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.918708286 (528h0m9.789791331s) [noise=true , address=77.90.31.12:4593 , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.919456690 (528h0m9.790539736s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920039571 (528h0m9.791122616s) [noise=true , address=145.79.227.35:4512 , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920276141 (528h0m9.791359186s) [noise=true , address=145.79.227.34:4926 , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920474529 (528h0m9.791557564s) [noise=true , address=145.79.227.13:4816 , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.921046528 (528h0m9.792129563s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.921603759 (528h0m9.792686794s) [noise=true , address=77.90.31.21:4883 , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.922032846 (528h0m9.793115881s) [noise=true , address=145.79.227.44:4210 , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.923593568 (528h0m9.794676613s) [noise=true , address=45.91.116.10:4103 , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.924048133 (528h0m9.795131169s) [noise=true , address=77.90.31.18:4840 , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.924539719 (528h0m9.795622754s) [noise=true , address=64.49.13.139:4470 , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.925024673 (528h0m9.796107708s) [noise=true , address=145.79.227.49:4368 , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.925500990 (528h0m9.796584035s) [noise=true , address=77.90.31.17:4795 , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:11.205949065 (528h0m11.0770321s) [upload=463114]: Next returned nil segment Dec 27 15:29:11.205963343 (528h0m11.077046378s) [upload=463114]: Waiting for error group managing segments... Dec 27 15:29:11.610155861 (528h0m11.481238906s) [noise=true , address=64.49.13.139:4470 , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.812270605 (528h0m11.683353641s) [noise=true , address=145.79.227.35:4512 , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.818242477 (528h0m11.689325522s) [noise=true , address=145.79.227.13:4816 , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.938211621 (528h0m11.809294666s) [noise=true , address=145.79.227.49:4368 , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.064896693 (528h0m11.935979738s) [noise=true , address=77.90.31.17:4795 , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.070731273 (528h0m11.941814308s) [noise=true , address=77.90.31.21:4883 , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.107949174 (528h0m11.979032239s) [noise=true , address=77.90.31.11:4573 , piece_id=SRXHTEXDEKKJ2WPGY66BOAP7WD7Z75HK3B5DL2XIGFWDYQGI76YA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.151478532 (528h0m12.022561567s) [noise=true , address=77.90.31.13:4645 , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.335397247 (528h0m12.206480292s) [noise=true , address=45.91.116.10:4103 , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.410397992 (528h0m12.281481027s) [noise=true , address=77.90.31.18:4840 , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.667101894 (528h0m12.538184929s) [noise=true , address=77.90.31.12:4593 , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.785380610 (528h0m12.656463655s) [noise=true , address=145.79.227.44:4210 , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.828722731 (528h0m12.699805766s) [noise=true , address=193.149.164.143:4697 , piece_id=S4CQ3366KVFS5RMZR67MPGSLXKDWRAGEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.947757466 (528h0m12.818840501s) [noise=true , address=145.79.227.34:4926 , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.947777434 (528h0m12.818860469s) [seg_pos={0 0}, upload=463114]: Segment reached optimal threshold of 14 pieces. Dec 27 15:29:12.947947177 (528h0m12.819030212s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy): protocol: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection Dec 27 15:29:12.947953169 (528h0m12.819036204s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948200800 (528h0m12.819283845s) [noise=true , address=77.90.31.19:4869 , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8): protocol: read tcp 188.166.225.170:32932->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 15:29:12.948216079 (528h0m12.819299124s) [noise=true , address=77.90.31.19:4869 , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948310158 (528h0m12.819393203s) [noise=true , address=64.49.13.169:4711 , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection Dec 27 15:29:12.948316139 (528h0m12.819399174s) [noise=true , address=64.49.13.169:4711 , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948360164 (528h0m12.819443219s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 27 15:29:12.948372617 (528h0m12.819455652s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948398968 (528h0m12.819482013s) [seg_pos={0 0}, upload=463114]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:29:12.948477878 (528h0m12.819560913s) [upload=463114]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:29:12.948483889 (528h0m12.819566934s) [upload=463114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:29:13.237238452 (528h0m13.108321498s) [upload=463114]: Done uploading segments... Dec 27 15:29:13.237262939 (528h0m13.108345984s) [upload=463114]: Upload finished. err: <nil> Duration: 50.41104109s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.X..X...XXXXXXX..X.X..X.X.....X...X......X.X..X..XXX......] duration: 5.816613722s trace: https://jaeger.dev.storj.io/trace/1509babc40b8daba FAIL (2025-12-27 15:38:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: 3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA) context deadline exceeded ======LOGS====== Dec 27 15:35:36.834619921 (528h6m36.705702997s) [upload=463210]: Starting upload Dec 27 15:35:36.834705574 (528h6m36.705788649s) [upload=463210]: Done starting upload Dec 27 15:35:36.834811706 (528h6m36.705894761s) [upload=463210]: Uploading segments... Dec 27 15:35:36.834963957 (528h6m36.706047002s) [upload=463210]: Got next segment. Inline: false Dec 27 15:35:36.835004414 (528h6m36.706087449s) [upload=463210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:35:36.835010325 (528h6m36.70609337s) [upload=463210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:35:37.088013461 (528h6m36.959096506s) [seg_pos={0 0}, upload=463210]: Begin upload segment... Dec 27 15:35:37.088072674 (528h6m36.959155709s) [seg_pos={0 0}, upload=463210]: Done begin upload segment. Dec 27 15:35:37.088134742 (528h6m36.959217787s) [noise=true , address=45.91.116.6:4973 , piece_id=YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.088160852 (528h6m36.959243887s) [noise=true , address=145.79.227.12:4767 , piece_id=UYPFM2ZW6CVDQXHADJ2GJBC6KX35TBHBHSKG5BBIHZPGYDAFNEZA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.089097435 (528h6m36.96018048s) [noise=true , address=145.79.227.52:4445 , piece_id=DKSWMR77ALMRSOW47G44IFI7W3HZYQWHVV4JXB5QZZEUOVPZ7QDA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.089544757 (528h6m36.960627802s) [noise=true , address=45.91.116.11:4118 , piece_id=3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090044959 (528h6m36.961128004s) [noise=true , address=145.79.227.45:4231 , piece_id=5E2OU3ID7U7JKH7MB2EHTAQZX5UZ2I2NTD6VACRSRQM2SCOV723Q, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090414994 (528h6m36.961498039s) [noise=true , address=145.79.227.49:4338 , piece_id=EP7IUXGYAF2LHM5BDTEK6HGN4O4OTPWTVYHWTYRWQ7CBLK26MZSQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090784278 (528h6m36.961867323s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.091246458 (528h6m36.962329503s) [noise=true , address=145.79.227.19:4078 , piece_id=UDFCQU6FOVIICU25EQOBN57JPCXNONYAW4Z2SQH7RBH2XSEFD2HA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.091740087 (528h6m36.962823122s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.093138921 (528h6m36.964221966s) [noise=true , address=145.79.227.10:4402 , piece_id=45HK3GSRJBMWALEJK3T4RZME4OZOJSU57JP4KXP6CHJZMSKK3Y4Q, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.093160833 (528h6m36.964243878s) [noise=true , address=145.79.227.48:4303 , piece_id=EKTFWLS3BYKVOL7NTVN7VIGWT3ET3MAPKM46RIFZK73ZQFST4J4A, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.094811165 (528h6m36.96589421s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.095010345 (528h6m36.96609338s) [noise=true , address=145.79.227.39:4015 , piece_id=B5G2XXRPZTWKSSA37SBWNV5BX3POFELY2UBRVQIPNS5LDGBH47PQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.096583099 (528h6m36.967666144s) [noise=true , address=145.79.227.3:4151 , piece_id=NTDEUAYE6X2QKXWO543TPLFMUDSHNXPFYZ42SDYNMIR7LQBIHIGQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.098185200 (528h6m36.969268235s) [noise=true , address=145.79.227.26:4655 , piece_id=CEBD7AHETY7SJ2563LQ57ZX6ALLYBTFIDSLTFOLOSRLPOBIS3CAA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.099692360 (528h6m36.970775406s) [noise=true , address=145.79.227.2:4113 , piece_id=VS5QG2XZA3YTDRHEO6ILLOH7H3GPSPGCL2NOP73CVA7VCBXMUDGA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:38.706744778 (528h6m38.577827823s) [upload=463210]: Next returned nil segment Dec 27 15:35:38.706800425 (528h6m38.57788346s) [upload=463210]: Waiting for error group managing segments... Dec 27 15:35:39.091009801 (528h6m38.962092847s) [noise=true , address=145.79.227.49:4338 , piece_id=EP7IUXGYAF2LHM5BDTEK6HGN4O4OTPWTVYHWTYRWQ7CBLK26MZSQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.091660060 (528h6m38.962743105s) [noise=true , address=145.79.227.12:4767 , piece_id=UYPFM2ZW6CVDQXHADJ2GJBC6KX35TBHBHSKG5BBIHZPGYDAFNEZA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.101083924 (528h6m38.972166959s) [noise=true , address=145.79.227.48:4303 , piece_id=EKTFWLS3BYKVOL7NTVN7VIGWT3ET3MAPKM46RIFZK73ZQFST4J4A, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.106377303 (528h6m38.977460338s) [noise=true , address=145.79.227.2:4113 , piece_id=VS5QG2XZA3YTDRHEO6ILLOH7H3GPSPGCL2NOP73CVA7VCBXMUDGA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.109339505 (528h6m38.98042254s) [noise=true , address=145.79.227.39:4015 , piece_id=B5G2XXRPZTWKSSA37SBWNV5BX3POFELY2UBRVQIPNS5LDGBH47PQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.109995724 (528h6m38.981078759s) [noise=true , address=145.79.227.10:4402 , piece_id=45HK3GSRJBMWALEJK3T4RZME4OZOJSU57JP4KXP6CHJZMSKK3Y4Q, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.112087207 (528h6m38.983170252s) [noise=true , address=145.79.227.26:4655 , piece_id=CEBD7AHETY7SJ2563LQ57ZX6ALLYBTFIDSLTFOLOSRLPOBIS3CAA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.124988133 (528h6m38.996071178s) [noise=true , address=145.79.227.45:4231 , piece_id=5E2OU3ID7U7JKH7MB2EHTAQZX5UZ2I2NTD6VACRSRQM2SCOV723Q, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.130435766 (528h6m39.001518801s) [noise=true , address=145.79.227.3:4151 , piece_id=NTDEUAYE6X2QKXWO543TPLFMUDSHNXPFYZ42SDYNMIR7LQBIHIGQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.133234746 (528h6m39.004317781s) [noise=true , address=145.79.227.52:4445 , piece_id=DKSWMR77ALMRSOW47G44IFI7W3HZYQWHVV4JXB5QZZEUOVPZ7QDA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.139036994 (528h6m39.010120039s) [noise=true , address=145.79.227.19:4078 , piece_id=UDFCQU6FOVIICU25EQOBN57JPCXNONYAW4Z2SQH7RBH2XSEFD2HA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.728671419 (528h6m39.599754454s) [noise=true , address=45.91.116.11:4118 , piece_id=3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.801455083 (528h6m39.672538128s) [noise=true , address=45.91.116.6:4973 , piece_id=YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.801481042 (528h6m39.672564087s) [seg_pos={0 0}, upload=463210]: Segment reached optimal threshold of 13 pieces. Dec 27 15:35:39.801753310 (528h6m39.672836345s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection Dec 27 15:35:39.801763870 (528h6m39.672846915s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.801890311 (528h6m39.672973356s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 15:35:39.801899779 (528h6m39.672982814s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.801963491 (528h6m39.673046526s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 27 15:35:39.801983249 (528h6m39.673066284s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.802063842 (528h6m39.673146877s) [seg_pos={0 0}, upload=463210]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:35:39.802150638 (528h6m39.673233683s) [upload=463210]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:35:39.802158653 (528h6m39.673241708s) [upload=463210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:35:40.131666538 (528h6m40.002749573s) [upload=463210]: Done uploading segments... Dec 27 15:35:40.131691745 (528h6m40.00277478s) [upload=463210]: Upload finished. err: <nil> Duration: 5.81663955s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [X.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XXXXX...X.....X...XX.XXX] duration: 5.001366629s trace: https://jaeger.dev.storj.io/trace/81ad0e877ca327f FAIL (2025-12-27 15:38:48 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001386977s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..X.X..X...XXXXXXX..X.X..X.X.....X...X......X.X..X..XXX.....] duration: 5.647633193s trace: https://jaeger.dev.storj.io/trace/1509babc40b8daba FAIL (2025-12-27 15:38:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: 3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA) context deadline exceeded ======LOGS====== Dec 27 15:35:36.834619921 (528h6m36.705702997s) [upload=463210]: Starting upload Dec 27 15:35:36.834705574 (528h6m36.705788649s) [upload=463210]: Done starting upload Dec 27 15:35:36.834811706 (528h6m36.705894761s) [upload=463210]: Uploading segments... Dec 27 15:35:36.834963957 (528h6m36.706047002s) [upload=463210]: Got next segment. Inline: false Dec 27 15:35:36.835004414 (528h6m36.706087449s) [upload=463210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:35:36.835010325 (528h6m36.70609337s) [upload=463210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:35:37.088013461 (528h6m36.959096506s) [seg_pos={0 0}, upload=463210]: Begin upload segment... Dec 27 15:35:37.088072674 (528h6m36.959155709s) [seg_pos={0 0}, upload=463210]: Done begin upload segment. Dec 27 15:35:37.088134742 (528h6m36.959217787s) [noise=true , address=45.91.116.6:4973 , piece_id=YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.088160852 (528h6m36.959243887s) [noise=true , address=145.79.227.12:4767 , piece_id=UYPFM2ZW6CVDQXHADJ2GJBC6KX35TBHBHSKG5BBIHZPGYDAFNEZA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.089097435 (528h6m36.96018048s) [noise=true , address=145.79.227.52:4445 , piece_id=DKSWMR77ALMRSOW47G44IFI7W3HZYQWHVV4JXB5QZZEUOVPZ7QDA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.089544757 (528h6m36.960627802s) [noise=true , address=45.91.116.11:4118 , piece_id=3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090044959 (528h6m36.961128004s) [noise=true , address=145.79.227.45:4231 , piece_id=5E2OU3ID7U7JKH7MB2EHTAQZX5UZ2I2NTD6VACRSRQM2SCOV723Q, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090414994 (528h6m36.961498039s) [noise=true , address=145.79.227.49:4338 , piece_id=EP7IUXGYAF2LHM5BDTEK6HGN4O4OTPWTVYHWTYRWQ7CBLK26MZSQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090784278 (528h6m36.961867323s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.091246458 (528h6m36.962329503s) [noise=true , address=145.79.227.19:4078 , piece_id=UDFCQU6FOVIICU25EQOBN57JPCXNONYAW4Z2SQH7RBH2XSEFD2HA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.091740087 (528h6m36.962823122s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.093138921 (528h6m36.964221966s) [noise=true , address=145.79.227.10:4402 , piece_id=45HK3GSRJBMWALEJK3T4RZME4OZOJSU57JP4KXP6CHJZMSKK3Y4Q, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.093160833 (528h6m36.964243878s) [noise=true , address=145.79.227.48:4303 , piece_id=EKTFWLS3BYKVOL7NTVN7VIGWT3ET3MAPKM46RIFZK73ZQFST4J4A, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.094811165 (528h6m36.96589421s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.095010345 (528h6m36.96609338s) [noise=true , address=145.79.227.39:4015 , piece_id=B5G2XXRPZTWKSSA37SBWNV5BX3POFELY2UBRVQIPNS5LDGBH47PQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.096583099 (528h6m36.967666144s) [noise=true , address=145.79.227.3:4151 , piece_id=NTDEUAYE6X2QKXWO543TPLFMUDSHNXPFYZ42SDYNMIR7LQBIHIGQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.098185200 (528h6m36.969268235s) [noise=true , address=145.79.227.26:4655 , piece_id=CEBD7AHETY7SJ2563LQ57ZX6ALLYBTFIDSLTFOLOSRLPOBIS3CAA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.099692360 (528h6m36.970775406s) [noise=true , address=145.79.227.2:4113 , piece_id=VS5QG2XZA3YTDRHEO6ILLOH7H3GPSPGCL2NOP73CVA7VCBXMUDGA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:38.706744778 (528h6m38.577827823s) [upload=463210]: Next returned nil segment Dec 27 15:35:38.706800425 (528h6m38.57788346s) [upload=463210]: Waiting for error group managing segments... Dec 27 15:35:39.091009801 (528h6m38.962092847s) [noise=true , address=145.79.227.49:4338 , piece_id=EP7IUXGYAF2LHM5BDTEK6HGN4O4OTPWTVYHWTYRWQ7CBLK26MZSQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.091660060 (528h6m38.962743105s) [noise=true , address=145.79.227.12:4767 , piece_id=UYPFM2ZW6CVDQXHADJ2GJBC6KX35TBHBHSKG5BBIHZPGYDAFNEZA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.101083924 (528h6m38.972166959s) [noise=true , address=145.79.227.48:4303 , piece_id=EKTFWLS3BYKVOL7NTVN7VIGWT3ET3MAPKM46RIFZK73ZQFST4J4A, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.106377303 (528h6m38.977460338s) [noise=true , address=145.79.227.2:4113 , piece_id=VS5QG2XZA3YTDRHEO6ILLOH7H3GPSPGCL2NOP73CVA7VCBXMUDGA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.109339505 (528h6m38.98042254s) [noise=true , address=145.79.227.39:4015 , piece_id=B5G2XXRPZTWKSSA37SBWNV5BX3POFELY2UBRVQIPNS5LDGBH47PQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.109995724 (528h6m38.981078759s) [noise=true , address=145.79.227.10:4402 , piece_id=45HK3GSRJBMWALEJK3T4RZME4OZOJSU57JP4KXP6CHJZMSKK3Y4Q, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.112087207 (528h6m38.983170252s) [noise=true , address=145.79.227.26:4655 , piece_id=CEBD7AHETY7SJ2563LQ57ZX6ALLYBTFIDSLTFOLOSRLPOBIS3CAA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.124988133 (528h6m38.996071178s) [noise=true , address=145.79.227.45:4231 , piece_id=5E2OU3ID7U7JKH7MB2EHTAQZX5UZ2I2NTD6VACRSRQM2SCOV723Q, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.130435766 (528h6m39.001518801s) [noise=true , address=145.79.227.3:4151 , piece_id=NTDEUAYE6X2QKXWO543TPLFMUDSHNXPFYZ42SDYNMIR7LQBIHIGQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.133234746 (528h6m39.004317781s) [noise=true , address=145.79.227.52:4445 , piece_id=DKSWMR77ALMRSOW47G44IFI7W3HZYQWHVV4JXB5QZZEUOVPZ7QDA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.139036994 (528h6m39.010120039s) [noise=true , address=145.79.227.19:4078 , piece_id=UDFCQU6FOVIICU25EQOBN57JPCXNONYAW4Z2SQH7RBH2XSEFD2HA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.728671419 (528h6m39.599754454s) [noise=true , address=45.91.116.11:4118 , piece_id=3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.801455083 (528h6m39.672538128s) [noise=true , address=45.91.116.6:4973 , piece_id=YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.801481042 (528h6m39.672564087s) [seg_pos={0 0}, upload=463210]: Segment reached optimal threshold of 13 pieces. Dec 27 15:35:39.801753310 (528h6m39.672836345s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection Dec 27 15:35:39.801763870 (528h6m39.672846915s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.801890311 (528h6m39.672973356s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 15:35:39.801899779 (528h6m39.672982814s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.801963491 (528h6m39.673046526s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 27 15:35:39.801983249 (528h6m39.673066284s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.802063842 (528h6m39.673146877s) [seg_pos={0 0}, upload=463210]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:35:39.802150638 (528h6m39.673233683s) [upload=463210]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:35:39.802158653 (528h6m39.673241708s) [upload=463210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:35:40.131666538 (528h6m40.002749573s) [upload=463210]: Done uploading segments... Dec 27 15:35:40.131691745 (528h6m40.00277478s) [upload=463210]: Upload finished. err: <nil> Duration: 5.647663249s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X..X.X..X...XXXXXXX..X.X..X.X.....X...X......X.X..X..XXX....] duration: 5.938605516s trace: https://jaeger.dev.storj.io/trace/1509babc40b8daba FAIL (2025-12-27 15:38:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: 3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA) context deadline exceeded ======LOGS====== Dec 27 15:35:36.834619921 (528h6m36.705702997s) [upload=463210]: Starting upload Dec 27 15:35:36.834705574 (528h6m36.705788649s) [upload=463210]: Done starting upload Dec 27 15:35:36.834811706 (528h6m36.705894761s) [upload=463210]: Uploading segments... Dec 27 15:35:36.834963957 (528h6m36.706047002s) [upload=463210]: Got next segment. Inline: false Dec 27 15:35:36.835004414 (528h6m36.706087449s) [upload=463210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:35:36.835010325 (528h6m36.70609337s) [upload=463210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:35:37.088013461 (528h6m36.959096506s) [seg_pos={0 0}, upload=463210]: Begin upload segment... Dec 27 15:35:37.088072674 (528h6m36.959155709s) [seg_pos={0 0}, upload=463210]: Done begin upload segment. Dec 27 15:35:37.088134742 (528h6m36.959217787s) [noise=true , address=45.91.116.6:4973 , piece_id=YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.088160852 (528h6m36.959243887s) [noise=true , address=145.79.227.12:4767 , piece_id=UYPFM2ZW6CVDQXHADJ2GJBC6KX35TBHBHSKG5BBIHZPGYDAFNEZA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.089097435 (528h6m36.96018048s) [noise=true , address=145.79.227.52:4445 , piece_id=DKSWMR77ALMRSOW47G44IFI7W3HZYQWHVV4JXB5QZZEUOVPZ7QDA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.089544757 (528h6m36.960627802s) [noise=true , address=45.91.116.11:4118 , piece_id=3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090044959 (528h6m36.961128004s) [noise=true , address=145.79.227.45:4231 , piece_id=5E2OU3ID7U7JKH7MB2EHTAQZX5UZ2I2NTD6VACRSRQM2SCOV723Q, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090414994 (528h6m36.961498039s) [noise=true , address=145.79.227.49:4338 , piece_id=EP7IUXGYAF2LHM5BDTEK6HGN4O4OTPWTVYHWTYRWQ7CBLK26MZSQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090784278 (528h6m36.961867323s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.091246458 (528h6m36.962329503s) [noise=true , address=145.79.227.19:4078 , piece_id=UDFCQU6FOVIICU25EQOBN57JPCXNONYAW4Z2SQH7RBH2XSEFD2HA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.091740087 (528h6m36.962823122s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.093138921 (528h6m36.964221966s) [noise=true , address=145.79.227.10:4402 , piece_id=45HK3GSRJBMWALEJK3T4RZME4OZOJSU57JP4KXP6CHJZMSKK3Y4Q, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.093160833 (528h6m36.964243878s) [noise=true , address=145.79.227.48:4303 , piece_id=EKTFWLS3BYKVOL7NTVN7VIGWT3ET3MAPKM46RIFZK73ZQFST4J4A, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.094811165 (528h6m36.96589421s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.095010345 (528h6m36.96609338s) [noise=true , address=145.79.227.39:4015 , piece_id=B5G2XXRPZTWKSSA37SBWNV5BX3POFELY2UBRVQIPNS5LDGBH47PQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.096583099 (528h6m36.967666144s) [noise=true , address=145.79.227.3:4151 , piece_id=NTDEUAYE6X2QKXWO543TPLFMUDSHNXPFYZ42SDYNMIR7LQBIHIGQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.098185200 (528h6m36.969268235s) [noise=true , address=145.79.227.26:4655 , piece_id=CEBD7AHETY7SJ2563LQ57ZX6ALLYBTFIDSLTFOLOSRLPOBIS3CAA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.099692360 (528h6m36.970775406s) [noise=true , address=145.79.227.2:4113 , piece_id=VS5QG2XZA3YTDRHEO6ILLOH7H3GPSPGCL2NOP73CVA7VCBXMUDGA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:38.706744778 (528h6m38.577827823s) [upload=463210]: Next returned nil segment Dec 27 15:35:38.706800425 (528h6m38.57788346s) [upload=463210]: Waiting for error group managing segments... Dec 27 15:35:39.091009801 (528h6m38.962092847s) [noise=true , address=145.79.227.49:4338 , piece_id=EP7IUXGYAF2LHM5BDTEK6HGN4O4OTPWTVYHWTYRWQ7CBLK26MZSQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.091660060 (528h6m38.962743105s) [noise=true , address=145.79.227.12:4767 , piece_id=UYPFM2ZW6CVDQXHADJ2GJBC6KX35TBHBHSKG5BBIHZPGYDAFNEZA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.101083924 (528h6m38.972166959s) [noise=true , address=145.79.227.48:4303 , piece_id=EKTFWLS3BYKVOL7NTVN7VIGWT3ET3MAPKM46RIFZK73ZQFST4J4A, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.106377303 (528h6m38.977460338s) [noise=true , address=145.79.227.2:4113 , piece_id=VS5QG2XZA3YTDRHEO6ILLOH7H3GPSPGCL2NOP73CVA7VCBXMUDGA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.109339505 (528h6m38.98042254s) [noise=true , address=145.79.227.39:4015 , piece_id=B5G2XXRPZTWKSSA37SBWNV5BX3POFELY2UBRVQIPNS5LDGBH47PQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.109995724 (528h6m38.981078759s) [noise=true , address=145.79.227.10:4402 , piece_id=45HK3GSRJBMWALEJK3T4RZME4OZOJSU57JP4KXP6CHJZMSKK3Y4Q, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.112087207 (528h6m38.983170252s) [noise=true , address=145.79.227.26:4655 , piece_id=CEBD7AHETY7SJ2563LQ57ZX6ALLYBTFIDSLTFOLOSRLPOBIS3CAA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.124988133 (528h6m38.996071178s) [noise=true , address=145.79.227.45:4231 , piece_id=5E2OU3ID7U7JKH7MB2EHTAQZX5UZ2I2NTD6VACRSRQM2SCOV723Q, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.130435766 (528h6m39.001518801s) [noise=true , address=145.79.227.3:4151 , piece_id=NTDEUAYE6X2QKXWO543TPLFMUDSHNXPFYZ42SDYNMIR7LQBIHIGQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.133234746 (528h6m39.004317781s) [noise=true , address=145.79.227.52:4445 , piece_id=DKSWMR77ALMRSOW47G44IFI7W3HZYQWHVV4JXB5QZZEUOVPZ7QDA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.139036994 (528h6m39.010120039s) [noise=true , address=145.79.227.19:4078 , piece_id=UDFCQU6FOVIICU25EQOBN57JPCXNONYAW4Z2SQH7RBH2XSEFD2HA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.728671419 (528h6m39.599754454s) [noise=true , address=45.91.116.11:4118 , piece_id=3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.801455083 (528h6m39.672538128s) [noise=true , address=45.91.116.6:4973 , piece_id=YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.801481042 (528h6m39.672564087s) [seg_pos={0 0}, upload=463210]: Segment reached optimal threshold of 13 pieces. Dec 27 15:35:39.801753310 (528h6m39.672836345s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection Dec 27 15:35:39.801763870 (528h6m39.672846915s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.801890311 (528h6m39.672973356s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 15:35:39.801899779 (528h6m39.672982814s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.801963491 (528h6m39.673046526s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 27 15:35:39.801983249 (528h6m39.673066284s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.802063842 (528h6m39.673146877s) [seg_pos={0 0}, upload=463210]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:35:39.802150638 (528h6m39.673233683s) [upload=463210]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:35:39.802158653 (528h6m39.673241708s) [upload=463210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:35:40.131666538 (528h6m40.002749573s) [upload=463210]: Done uploading segments... Dec 27 15:35:40.131691745 (528h6m40.00277478s) [upload=463210]: Upload finished. err: <nil> Duration: 5.93863317s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XXXXX...X.....X...XX.XX] duration: 5.001896226s trace: https://jaeger.dev.storj.io/trace/4c22ae0faf745291 FAIL (2025-12-27 15:38:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002035612s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........X....X...................X.....XX......X.........] duration: 44.64474274s trace: https://jaeger.dev.storj.io/trace/24120a988203c524 FAIL (2025-12-27 15:38:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 191MKpEnoarS7sK4Gs63ZM8zGc5jevFX8VeqViP82duW33nJJz, Piece ID: 6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A) context deadline exceeded; piecestore: (Node ID: 1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, Piece ID: JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA) context deadline exceeded; piecestore: (Node ID: 12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, Piece ID: L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA) context deadline exceeded ======LOGS====== GEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.918056404 (528h0m9.789139439s) [noise=true , address=77.90.31.13:4645 , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.918708286 (528h0m9.789791331s) [noise=true , address=77.90.31.12:4593 , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.919456690 (528h0m9.790539736s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920039571 (528h0m9.791122616s) [noise=true , address=145.79.227.35:4512 , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920276141 (528h0m9.791359186s) [noise=true , address=145.79.227.34:4926 , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920474529 (528h0m9.791557564s) [noise=true , address=145.79.227.13:4816 , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.921046528 (528h0m9.792129563s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.921603759 (528h0m9.792686794s) [noise=true , address=77.90.31.21:4883 , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.922032846 (528h0m9.793115881s) [noise=true , address=145.79.227.44:4210 , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.923593568 (528h0m9.794676613s) [noise=true , address=45.91.116.10:4103 , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.924048133 (528h0m9.795131169s) [noise=true , address=77.90.31.18:4840 , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.924539719 (528h0m9.795622754s) [noise=true , address=64.49.13.139:4470 , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.925024673 (528h0m9.796107708s) [noise=true , address=145.79.227.49:4368 , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.925500990 (528h0m9.796584035s) [noise=true , address=77.90.31.17:4795 , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:11.205949065 (528h0m11.0770321s) [upload=463114]: Next returned nil segment Dec 27 15:29:11.205963343 (528h0m11.077046378s) [upload=463114]: Waiting for error group managing segments... Dec 27 15:29:11.610155861 (528h0m11.481238906s) [noise=true , address=64.49.13.139:4470 , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.812270605 (528h0m11.683353641s) [noise=true , address=145.79.227.35:4512 , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.818242477 (528h0m11.689325522s) [noise=true , address=145.79.227.13:4816 , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.938211621 (528h0m11.809294666s) [noise=true , address=145.79.227.49:4368 , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.064896693 (528h0m11.935979738s) [noise=true , address=77.90.31.17:4795 , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.070731273 (528h0m11.941814308s) [noise=true , address=77.90.31.21:4883 , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.107949174 (528h0m11.979032239s) [noise=true , address=77.90.31.11:4573 , piece_id=SRXHTEXDEKKJ2WPGY66BOAP7WD7Z75HK3B5DL2XIGFWDYQGI76YA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.151478532 (528h0m12.022561567s) [noise=true , address=77.90.31.13:4645 , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.335397247 (528h0m12.206480292s) [noise=true , address=45.91.116.10:4103 , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.410397992 (528h0m12.281481027s) [noise=true , address=77.90.31.18:4840 , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.667101894 (528h0m12.538184929s) [noise=true , address=77.90.31.12:4593 , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.785380610 (528h0m12.656463655s) [noise=true , address=145.79.227.44:4210 , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.828722731 (528h0m12.699805766s) [noise=true , address=193.149.164.143:4697 , piece_id=S4CQ3366KVFS5RMZR67MPGSLXKDWRAGEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.947757466 (528h0m12.818840501s) [noise=true , address=145.79.227.34:4926 , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.947777434 (528h0m12.818860469s) [seg_pos={0 0}, upload=463114]: Segment reached optimal threshold of 14 pieces. Dec 27 15:29:12.947947177 (528h0m12.819030212s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy): protocol: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection Dec 27 15:29:12.947953169 (528h0m12.819036204s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948200800 (528h0m12.819283845s) [noise=true , address=77.90.31.19:4869 , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8): protocol: read tcp 188.166.225.170:32932->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 15:29:12.948216079 (528h0m12.819299124s) [noise=true , address=77.90.31.19:4869 , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948310158 (528h0m12.819393203s) [noise=true , address=64.49.13.169:4711 , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection Dec 27 15:29:12.948316139 (528h0m12.819399174s) [noise=true , address=64.49.13.169:4711 , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948360164 (528h0m12.819443219s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 27 15:29:12.948372617 (528h0m12.819455652s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948398968 (528h0m12.819482013s) [seg_pos={0 0}, upload=463114]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:29:12.948477878 (528h0m12.819560913s) [upload=463114]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:29:12.948483889 (528h0m12.819566934s) [upload=463114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:29:13.237238452 (528h0m13.108321498s) [upload=463114]: Done uploading segments... Dec 27 15:29:13.237262939 (528h0m13.108345984s) [upload=463114]: Upload finished. err: <nil> Duration: 44.644760523s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X..X.X..X...XXXXXXX..X.X..X.X.....X...X......X.X..X..XXX...] duration: 5.6755593s trace: https://jaeger.dev.storj.io/trace/1509babc40b8daba FAIL (2025-12-27 15:37:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: 3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA) context deadline exceeded ======LOGS====== Dec 27 15:35:36.834619921 (528h6m36.705702997s) [upload=463210]: Starting upload Dec 27 15:35:36.834705574 (528h6m36.705788649s) [upload=463210]: Done starting upload Dec 27 15:35:36.834811706 (528h6m36.705894761s) [upload=463210]: Uploading segments... Dec 27 15:35:36.834963957 (528h6m36.706047002s) [upload=463210]: Got next segment. Inline: false Dec 27 15:35:36.835004414 (528h6m36.706087449s) [upload=463210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:35:36.835010325 (528h6m36.70609337s) [upload=463210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:35:37.088013461 (528h6m36.959096506s) [seg_pos={0 0}, upload=463210]: Begin upload segment... Dec 27 15:35:37.088072674 (528h6m36.959155709s) [seg_pos={0 0}, upload=463210]: Done begin upload segment. Dec 27 15:35:37.088134742 (528h6m36.959217787s) [noise=true , address=45.91.116.6:4973 , piece_id=YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.088160852 (528h6m36.959243887s) [noise=true , address=145.79.227.12:4767 , piece_id=UYPFM2ZW6CVDQXHADJ2GJBC6KX35TBHBHSKG5BBIHZPGYDAFNEZA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.089097435 (528h6m36.96018048s) [noise=true , address=145.79.227.52:4445 , piece_id=DKSWMR77ALMRSOW47G44IFI7W3HZYQWHVV4JXB5QZZEUOVPZ7QDA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.089544757 (528h6m36.960627802s) [noise=true , address=45.91.116.11:4118 , piece_id=3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090044959 (528h6m36.961128004s) [noise=true , address=145.79.227.45:4231 , piece_id=5E2OU3ID7U7JKH7MB2EHTAQZX5UZ2I2NTD6VACRSRQM2SCOV723Q, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090414994 (528h6m36.961498039s) [noise=true , address=145.79.227.49:4338 , piece_id=EP7IUXGYAF2LHM5BDTEK6HGN4O4OTPWTVYHWTYRWQ7CBLK26MZSQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090784278 (528h6m36.961867323s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.091246458 (528h6m36.962329503s) [noise=true , address=145.79.227.19:4078 , piece_id=UDFCQU6FOVIICU25EQOBN57JPCXNONYAW4Z2SQH7RBH2XSEFD2HA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.091740087 (528h6m36.962823122s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.093138921 (528h6m36.964221966s) [noise=true , address=145.79.227.10:4402 , piece_id=45HK3GSRJBMWALEJK3T4RZME4OZOJSU57JP4KXP6CHJZMSKK3Y4Q, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.093160833 (528h6m36.964243878s) [noise=true , address=145.79.227.48:4303 , piece_id=EKTFWLS3BYKVOL7NTVN7VIGWT3ET3MAPKM46RIFZK73ZQFST4J4A, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.094811165 (528h6m36.96589421s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.095010345 (528h6m36.96609338s) [noise=true , address=145.79.227.39:4015 , piece_id=B5G2XXRPZTWKSSA37SBWNV5BX3POFELY2UBRVQIPNS5LDGBH47PQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.096583099 (528h6m36.967666144s) [noise=true , address=145.79.227.3:4151 , piece_id=NTDEUAYE6X2QKXWO543TPLFMUDSHNXPFYZ42SDYNMIR7LQBIHIGQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.098185200 (528h6m36.969268235s) [noise=true , address=145.79.227.26:4655 , piece_id=CEBD7AHETY7SJ2563LQ57ZX6ALLYBTFIDSLTFOLOSRLPOBIS3CAA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.099692360 (528h6m36.970775406s) [noise=true , address=145.79.227.2:4113 , piece_id=VS5QG2XZA3YTDRHEO6ILLOH7H3GPSPGCL2NOP73CVA7VCBXMUDGA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:38.706744778 (528h6m38.577827823s) [upload=463210]: Next returned nil segment Dec 27 15:35:38.706800425 (528h6m38.57788346s) [upload=463210]: Waiting for error group managing segments... Dec 27 15:35:39.091009801 (528h6m38.962092847s) [noise=true , address=145.79.227.49:4338 , piece_id=EP7IUXGYAF2LHM5BDTEK6HGN4O4OTPWTVYHWTYRWQ7CBLK26MZSQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.091660060 (528h6m38.962743105s) [noise=true , address=145.79.227.12:4767 , piece_id=UYPFM2ZW6CVDQXHADJ2GJBC6KX35TBHBHSKG5BBIHZPGYDAFNEZA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.101083924 (528h6m38.972166959s) [noise=true , address=145.79.227.48:4303 , piece_id=EKTFWLS3BYKVOL7NTVN7VIGWT3ET3MAPKM46RIFZK73ZQFST4J4A, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.106377303 (528h6m38.977460338s) [noise=true , address=145.79.227.2:4113 , piece_id=VS5QG2XZA3YTDRHEO6ILLOH7H3GPSPGCL2NOP73CVA7VCBXMUDGA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.109339505 (528h6m38.98042254s) [noise=true , address=145.79.227.39:4015 , piece_id=B5G2XXRPZTWKSSA37SBWNV5BX3POFELY2UBRVQIPNS5LDGBH47PQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.109995724 (528h6m38.981078759s) [noise=true , address=145.79.227.10:4402 , piece_id=45HK3GSRJBMWALEJK3T4RZME4OZOJSU57JP4KXP6CHJZMSKK3Y4Q, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.112087207 (528h6m38.983170252s) [noise=true , address=145.79.227.26:4655 , piece_id=CEBD7AHETY7SJ2563LQ57ZX6ALLYBTFIDSLTFOLOSRLPOBIS3CAA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.124988133 (528h6m38.996071178s) [noise=true , address=145.79.227.45:4231 , piece_id=5E2OU3ID7U7JKH7MB2EHTAQZX5UZ2I2NTD6VACRSRQM2SCOV723Q, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.130435766 (528h6m39.001518801s) [noise=true , address=145.79.227.3:4151 , piece_id=NTDEUAYE6X2QKXWO543TPLFMUDSHNXPFYZ42SDYNMIR7LQBIHIGQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.133234746 (528h6m39.004317781s) [noise=true , address=145.79.227.52:4445 , piece_id=DKSWMR77ALMRSOW47G44IFI7W3HZYQWHVV4JXB5QZZEUOVPZ7QDA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.139036994 (528h6m39.010120039s) [noise=true , address=145.79.227.19:4078 , piece_id=UDFCQU6FOVIICU25EQOBN57JPCXNONYAW4Z2SQH7RBH2XSEFD2HA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.728671419 (528h6m39.599754454s) [noise=true , address=45.91.116.11:4118 , piece_id=3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.801455083 (528h6m39.672538128s) [noise=true , address=45.91.116.6:4973 , piece_id=YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.801481042 (528h6m39.672564087s) [seg_pos={0 0}, upload=463210]: Segment reached optimal threshold of 13 pieces. Dec 27 15:35:39.801753310 (528h6m39.672836345s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection Dec 27 15:35:39.801763870 (528h6m39.672846915s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.801890311 (528h6m39.672973356s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 15:35:39.801899779 (528h6m39.672982814s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.801963491 (528h6m39.673046526s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 27 15:35:39.801983249 (528h6m39.673066284s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.802063842 (528h6m39.673146877s) [seg_pos={0 0}, upload=463210]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:35:39.802150638 (528h6m39.673233683s) [upload=463210]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:35:39.802158653 (528h6m39.673241708s) [upload=463210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:35:40.131666538 (528h6m40.002749573s) [upload=463210]: Done uploading segments... Dec 27 15:35:40.131691745 (528h6m40.00277478s) [upload=463210]: Upload finished. err: <nil> Duration: 5.675593324s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [............X....X...................X.....XX......X........] duration: 38.656340364s trace: https://jaeger.dev.storj.io/trace/24120a988203c524 FAIL (2025-12-27 15:37:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 191MKpEnoarS7sK4Gs63ZM8zGc5jevFX8VeqViP82duW33nJJz, Piece ID: 6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A) context deadline exceeded; piecestore: (Node ID: 1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, Piece ID: JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA) context deadline exceeded; piecestore: (Node ID: 12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, Piece ID: L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA) context deadline exceeded ======LOGS====== GEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.918056404 (528h0m9.789139439s) [noise=true , address=77.90.31.13:4645 , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.918708286 (528h0m9.789791331s) [noise=true , address=77.90.31.12:4593 , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.919456690 (528h0m9.790539736s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920039571 (528h0m9.791122616s) [noise=true , address=145.79.227.35:4512 , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920276141 (528h0m9.791359186s) [noise=true , address=145.79.227.34:4926 , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920474529 (528h0m9.791557564s) [noise=true , address=145.79.227.13:4816 , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.921046528 (528h0m9.792129563s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.921603759 (528h0m9.792686794s) [noise=true , address=77.90.31.21:4883 , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.922032846 (528h0m9.793115881s) [noise=true , address=145.79.227.44:4210 , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.923593568 (528h0m9.794676613s) [noise=true , address=45.91.116.10:4103 , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.924048133 (528h0m9.795131169s) [noise=true , address=77.90.31.18:4840 , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.924539719 (528h0m9.795622754s) [noise=true , address=64.49.13.139:4470 , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.925024673 (528h0m9.796107708s) [noise=true , address=145.79.227.49:4368 , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.925500990 (528h0m9.796584035s) [noise=true , address=77.90.31.17:4795 , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:11.205949065 (528h0m11.0770321s) [upload=463114]: Next returned nil segment Dec 27 15:29:11.205963343 (528h0m11.077046378s) [upload=463114]: Waiting for error group managing segments... Dec 27 15:29:11.610155861 (528h0m11.481238906s) [noise=true , address=64.49.13.139:4470 , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.812270605 (528h0m11.683353641s) [noise=true , address=145.79.227.35:4512 , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.818242477 (528h0m11.689325522s) [noise=true , address=145.79.227.13:4816 , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.938211621 (528h0m11.809294666s) [noise=true , address=145.79.227.49:4368 , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.064896693 (528h0m11.935979738s) [noise=true , address=77.90.31.17:4795 , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.070731273 (528h0m11.941814308s) [noise=true , address=77.90.31.21:4883 , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.107949174 (528h0m11.979032239s) [noise=true , address=77.90.31.11:4573 , piece_id=SRXHTEXDEKKJ2WPGY66BOAP7WD7Z75HK3B5DL2XIGFWDYQGI76YA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.151478532 (528h0m12.022561567s) [noise=true , address=77.90.31.13:4645 , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.335397247 (528h0m12.206480292s) [noise=true , address=45.91.116.10:4103 , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.410397992 (528h0m12.281481027s) [noise=true , address=77.90.31.18:4840 , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.667101894 (528h0m12.538184929s) [noise=true , address=77.90.31.12:4593 , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.785380610 (528h0m12.656463655s) [noise=true , address=145.79.227.44:4210 , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.828722731 (528h0m12.699805766s) [noise=true , address=193.149.164.143:4697 , piece_id=S4CQ3366KVFS5RMZR67MPGSLXKDWRAGEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.947757466 (528h0m12.818840501s) [noise=true , address=145.79.227.34:4926 , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.947777434 (528h0m12.818860469s) [seg_pos={0 0}, upload=463114]: Segment reached optimal threshold of 14 pieces. Dec 27 15:29:12.947947177 (528h0m12.819030212s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy): protocol: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection Dec 27 15:29:12.947953169 (528h0m12.819036204s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948200800 (528h0m12.819283845s) [noise=true , address=77.90.31.19:4869 , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8): protocol: read tcp 188.166.225.170:32932->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 15:29:12.948216079 (528h0m12.819299124s) [noise=true , address=77.90.31.19:4869 , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948310158 (528h0m12.819393203s) [noise=true , address=64.49.13.169:4711 , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection Dec 27 15:29:12.948316139 (528h0m12.819399174s) [noise=true , address=64.49.13.169:4711 , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948360164 (528h0m12.819443219s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 27 15:29:12.948372617 (528h0m12.819455652s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948398968 (528h0m12.819482013s) [seg_pos={0 0}, upload=463114]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:29:12.948477878 (528h0m12.819560913s) [upload=463114]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:29:12.948483889 (528h0m12.819566934s) [upload=463114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:29:13.237238452 (528h0m13.108321498s) [upload=463114]: Done uploading segments... Dec 27 15:29:13.237262939 (528h0m13.108345984s) [upload=463114]: Upload finished. err: <nil> Duration: 38.656360933s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..X..X.X..X...XXXXXXX..X.X..X.X.....X...X......X.X..X..XXX..] duration: 5.834821138s trace: https://jaeger.dev.storj.io/trace/1509babc40b8daba FAIL (2025-12-27 15:37:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: 3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA) context deadline exceeded ======LOGS====== Dec 27 15:35:36.834619921 (528h6m36.705702997s) [upload=463210]: Starting upload Dec 27 15:35:36.834705574 (528h6m36.705788649s) [upload=463210]: Done starting upload Dec 27 15:35:36.834811706 (528h6m36.705894761s) [upload=463210]: Uploading segments... Dec 27 15:35:36.834963957 (528h6m36.706047002s) [upload=463210]: Got next segment. Inline: false Dec 27 15:35:36.835004414 (528h6m36.706087449s) [upload=463210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:35:36.835010325 (528h6m36.70609337s) [upload=463210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:35:37.088013461 (528h6m36.959096506s) [seg_pos={0 0}, upload=463210]: Begin upload segment... Dec 27 15:35:37.088072674 (528h6m36.959155709s) [seg_pos={0 0}, upload=463210]: Done begin upload segment. Dec 27 15:35:37.088134742 (528h6m36.959217787s) [noise=true , address=45.91.116.6:4973 , piece_id=YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.088160852 (528h6m36.959243887s) [noise=true , address=145.79.227.12:4767 , piece_id=UYPFM2ZW6CVDQXHADJ2GJBC6KX35TBHBHSKG5BBIHZPGYDAFNEZA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.089097435 (528h6m36.96018048s) [noise=true , address=145.79.227.52:4445 , piece_id=DKSWMR77ALMRSOW47G44IFI7W3HZYQWHVV4JXB5QZZEUOVPZ7QDA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.089544757 (528h6m36.960627802s) [noise=true , address=45.91.116.11:4118 , piece_id=3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090044959 (528h6m36.961128004s) [noise=true , address=145.79.227.45:4231 , piece_id=5E2OU3ID7U7JKH7MB2EHTAQZX5UZ2I2NTD6VACRSRQM2SCOV723Q, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090414994 (528h6m36.961498039s) [noise=true , address=145.79.227.49:4338 , piece_id=EP7IUXGYAF2LHM5BDTEK6HGN4O4OTPWTVYHWTYRWQ7CBLK26MZSQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090784278 (528h6m36.961867323s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.091246458 (528h6m36.962329503s) [noise=true , address=145.79.227.19:4078 , piece_id=UDFCQU6FOVIICU25EQOBN57JPCXNONYAW4Z2SQH7RBH2XSEFD2HA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.091740087 (528h6m36.962823122s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.093138921 (528h6m36.964221966s) [noise=true , address=145.79.227.10:4402 , piece_id=45HK3GSRJBMWALEJK3T4RZME4OZOJSU57JP4KXP6CHJZMSKK3Y4Q, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.093160833 (528h6m36.964243878s) [noise=true , address=145.79.227.48:4303 , piece_id=EKTFWLS3BYKVOL7NTVN7VIGWT3ET3MAPKM46RIFZK73ZQFST4J4A, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.094811165 (528h6m36.96589421s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.095010345 (528h6m36.96609338s) [noise=true , address=145.79.227.39:4015 , piece_id=B5G2XXRPZTWKSSA37SBWNV5BX3POFELY2UBRVQIPNS5LDGBH47PQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.096583099 (528h6m36.967666144s) [noise=true , address=145.79.227.3:4151 , piece_id=NTDEUAYE6X2QKXWO543TPLFMUDSHNXPFYZ42SDYNMIR7LQBIHIGQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.098185200 (528h6m36.969268235s) [noise=true , address=145.79.227.26:4655 , piece_id=CEBD7AHETY7SJ2563LQ57ZX6ALLYBTFIDSLTFOLOSRLPOBIS3CAA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.099692360 (528h6m36.970775406s) [noise=true , address=145.79.227.2:4113 , piece_id=VS5QG2XZA3YTDRHEO6ILLOH7H3GPSPGCL2NOP73CVA7VCBXMUDGA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:38.706744778 (528h6m38.577827823s) [upload=463210]: Next returned nil segment Dec 27 15:35:38.706800425 (528h6m38.57788346s) [upload=463210]: Waiting for error group managing segments... Dec 27 15:35:39.091009801 (528h6m38.962092847s) [noise=true , address=145.79.227.49:4338 , piece_id=EP7IUXGYAF2LHM5BDTEK6HGN4O4OTPWTVYHWTYRWQ7CBLK26MZSQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.091660060 (528h6m38.962743105s) [noise=true , address=145.79.227.12:4767 , piece_id=UYPFM2ZW6CVDQXHADJ2GJBC6KX35TBHBHSKG5BBIHZPGYDAFNEZA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.101083924 (528h6m38.972166959s) [noise=true , address=145.79.227.48:4303 , piece_id=EKTFWLS3BYKVOL7NTVN7VIGWT3ET3MAPKM46RIFZK73ZQFST4J4A, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.106377303 (528h6m38.977460338s) [noise=true , address=145.79.227.2:4113 , piece_id=VS5QG2XZA3YTDRHEO6ILLOH7H3GPSPGCL2NOP73CVA7VCBXMUDGA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.109339505 (528h6m38.98042254s) [noise=true , address=145.79.227.39:4015 , piece_id=B5G2XXRPZTWKSSA37SBWNV5BX3POFELY2UBRVQIPNS5LDGBH47PQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.109995724 (528h6m38.981078759s) [noise=true , address=145.79.227.10:4402 , piece_id=45HK3GSRJBMWALEJK3T4RZME4OZOJSU57JP4KXP6CHJZMSKK3Y4Q, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.112087207 (528h6m38.983170252s) [noise=true , address=145.79.227.26:4655 , piece_id=CEBD7AHETY7SJ2563LQ57ZX6ALLYBTFIDSLTFOLOSRLPOBIS3CAA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.124988133 (528h6m38.996071178s) [noise=true , address=145.79.227.45:4231 , piece_id=5E2OU3ID7U7JKH7MB2EHTAQZX5UZ2I2NTD6VACRSRQM2SCOV723Q, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.130435766 (528h6m39.001518801s) [noise=true , address=145.79.227.3:4151 , piece_id=NTDEUAYE6X2QKXWO543TPLFMUDSHNXPFYZ42SDYNMIR7LQBIHIGQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.133234746 (528h6m39.004317781s) [noise=true , address=145.79.227.52:4445 , piece_id=DKSWMR77ALMRSOW47G44IFI7W3HZYQWHVV4JXB5QZZEUOVPZ7QDA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.139036994 (528h6m39.010120039s) [noise=true , address=145.79.227.19:4078 , piece_id=UDFCQU6FOVIICU25EQOBN57JPCXNONYAW4Z2SQH7RBH2XSEFD2HA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.728671419 (528h6m39.599754454s) [noise=true , address=45.91.116.11:4118 , piece_id=3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.801455083 (528h6m39.672538128s) [noise=true , address=45.91.116.6:4973 , piece_id=YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.801481042 (528h6m39.672564087s) [seg_pos={0 0}, upload=463210]: Segment reached optimal threshold of 13 pieces. Dec 27 15:35:39.801753310 (528h6m39.672836345s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection Dec 27 15:35:39.801763870 (528h6m39.672846915s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.801890311 (528h6m39.672973356s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 15:35:39.801899779 (528h6m39.672982814s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.801963491 (528h6m39.673046526s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 27 15:35:39.801983249 (528h6m39.673066284s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.802063842 (528h6m39.673146877s) [seg_pos={0 0}, upload=463210]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:35:39.802150638 (528h6m39.673233683s) [upload=463210]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:35:39.802158653 (528h6m39.673241708s) [upload=463210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:35:40.131666538 (528h6m40.002749573s) [upload=463210]: Done uploading segments... Dec 27 15:35:40.131691745 (528h6m40.00277478s) [upload=463210]: Upload finished. err: <nil> Duration: 5.834853469s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [.XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XXXXX...X.....X...XX.X] duration: 5.00083552s trace: https://jaeger.dev.storj.io/trace/4e903f99a8f5b206 FAIL (2025-12-27 15:37:02 UTC) orbiter warning: context deadline exceeded Duration: 5.000861179s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............X....X...................X.....XX......X.......] duration: 53.622053623s trace: https://jaeger.dev.storj.io/trace/24120a988203c524 FAIL (2025-12-27 15:36:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 191MKpEnoarS7sK4Gs63ZM8zGc5jevFX8VeqViP82duW33nJJz, Piece ID: 6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A) context deadline exceeded; piecestore: (Node ID: 1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, Piece ID: JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA) context deadline exceeded; piecestore: (Node ID: 12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, Piece ID: L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA) context deadline exceeded ======LOGS====== GEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.918056404 (528h0m9.789139439s) [noise=true , address=77.90.31.13:4645 , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.918708286 (528h0m9.789791331s) [noise=true , address=77.90.31.12:4593 , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.919456690 (528h0m9.790539736s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920039571 (528h0m9.791122616s) [noise=true , address=145.79.227.35:4512 , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920276141 (528h0m9.791359186s) [noise=true , address=145.79.227.34:4926 , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920474529 (528h0m9.791557564s) [noise=true , address=145.79.227.13:4816 , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.921046528 (528h0m9.792129563s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.921603759 (528h0m9.792686794s) [noise=true , address=77.90.31.21:4883 , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.922032846 (528h0m9.793115881s) [noise=true , address=145.79.227.44:4210 , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.923593568 (528h0m9.794676613s) [noise=true , address=45.91.116.10:4103 , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.924048133 (528h0m9.795131169s) [noise=true , address=77.90.31.18:4840 , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.924539719 (528h0m9.795622754s) [noise=true , address=64.49.13.139:4470 , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.925024673 (528h0m9.796107708s) [noise=true , address=145.79.227.49:4368 , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.925500990 (528h0m9.796584035s) [noise=true , address=77.90.31.17:4795 , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:11.205949065 (528h0m11.0770321s) [upload=463114]: Next returned nil segment Dec 27 15:29:11.205963343 (528h0m11.077046378s) [upload=463114]: Waiting for error group managing segments... Dec 27 15:29:11.610155861 (528h0m11.481238906s) [noise=true , address=64.49.13.139:4470 , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.812270605 (528h0m11.683353641s) [noise=true , address=145.79.227.35:4512 , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.818242477 (528h0m11.689325522s) [noise=true , address=145.79.227.13:4816 , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.938211621 (528h0m11.809294666s) [noise=true , address=145.79.227.49:4368 , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.064896693 (528h0m11.935979738s) [noise=true , address=77.90.31.17:4795 , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.070731273 (528h0m11.941814308s) [noise=true , address=77.90.31.21:4883 , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.107949174 (528h0m11.979032239s) [noise=true , address=77.90.31.11:4573 , piece_id=SRXHTEXDEKKJ2WPGY66BOAP7WD7Z75HK3B5DL2XIGFWDYQGI76YA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.151478532 (528h0m12.022561567s) [noise=true , address=77.90.31.13:4645 , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.335397247 (528h0m12.206480292s) [noise=true , address=45.91.116.10:4103 , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.410397992 (528h0m12.281481027s) [noise=true , address=77.90.31.18:4840 , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.667101894 (528h0m12.538184929s) [noise=true , address=77.90.31.12:4593 , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.785380610 (528h0m12.656463655s) [noise=true , address=145.79.227.44:4210 , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.828722731 (528h0m12.699805766s) [noise=true , address=193.149.164.143:4697 , piece_id=S4CQ3366KVFS5RMZR67MPGSLXKDWRAGEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.947757466 (528h0m12.818840501s) [noise=true , address=145.79.227.34:4926 , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.947777434 (528h0m12.818860469s) [seg_pos={0 0}, upload=463114]: Segment reached optimal threshold of 14 pieces. Dec 27 15:29:12.947947177 (528h0m12.819030212s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy): protocol: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection Dec 27 15:29:12.947953169 (528h0m12.819036204s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948200800 (528h0m12.819283845s) [noise=true , address=77.90.31.19:4869 , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8): protocol: read tcp 188.166.225.170:32932->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 15:29:12.948216079 (528h0m12.819299124s) [noise=true , address=77.90.31.19:4869 , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948310158 (528h0m12.819393203s) [noise=true , address=64.49.13.169:4711 , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection Dec 27 15:29:12.948316139 (528h0m12.819399174s) [noise=true , address=64.49.13.169:4711 , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948360164 (528h0m12.819443219s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 27 15:29:12.948372617 (528h0m12.819455652s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948398968 (528h0m12.819482013s) [seg_pos={0 0}, upload=463114]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:29:12.948477878 (528h0m12.819560913s) [upload=463114]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:29:12.948483889 (528h0m12.819566934s) [upload=463114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:29:13.237238452 (528h0m13.108321498s) [upload=463114]: Done uploading segments... Dec 27 15:29:13.237262939 (528h0m13.108345984s) [upload=463114]: Upload finished. err: <nil> Duration: 53.622071627s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XXXXX...X.....X...XX.] duration: 3.71103724s trace: https://jaeger.dev.storj.io/trace/6da1a55e00b56666 FAIL (2025-12-27 15:36:24 UTC) orbiter warning: context deadline exceeded Duration: 3.711066958s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [...X..X.X..X...XXXXXXX..X.X..X.X.....X...X......X.X..X..XXX.] duration: 5.630764964s trace: https://jaeger.dev.storj.io/trace/1509babc40b8daba FAIL (2025-12-27 15:36:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: 3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA) context deadline exceeded ======LOGS====== Dec 27 15:35:36.834619921 (528h6m36.705702997s) [upload=463210]: Starting upload Dec 27 15:35:36.834705574 (528h6m36.705788649s) [upload=463210]: Done starting upload Dec 27 15:35:36.834811706 (528h6m36.705894761s) [upload=463210]: Uploading segments... Dec 27 15:35:36.834963957 (528h6m36.706047002s) [upload=463210]: Got next segment. Inline: false Dec 27 15:35:36.835004414 (528h6m36.706087449s) [upload=463210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:35:36.835010325 (528h6m36.70609337s) [upload=463210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:35:37.088013461 (528h6m36.959096506s) [seg_pos={0 0}, upload=463210]: Begin upload segment... Dec 27 15:35:37.088072674 (528h6m36.959155709s) [seg_pos={0 0}, upload=463210]: Done begin upload segment. Dec 27 15:35:37.088134742 (528h6m36.959217787s) [noise=true , address=45.91.116.6:4973 , piece_id=YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.088160852 (528h6m36.959243887s) [noise=true , address=145.79.227.12:4767 , piece_id=UYPFM2ZW6CVDQXHADJ2GJBC6KX35TBHBHSKG5BBIHZPGYDAFNEZA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.089097435 (528h6m36.96018048s) [noise=true , address=145.79.227.52:4445 , piece_id=DKSWMR77ALMRSOW47G44IFI7W3HZYQWHVV4JXB5QZZEUOVPZ7QDA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.089544757 (528h6m36.960627802s) [noise=true , address=45.91.116.11:4118 , piece_id=3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090044959 (528h6m36.961128004s) [noise=true , address=145.79.227.45:4231 , piece_id=5E2OU3ID7U7JKH7MB2EHTAQZX5UZ2I2NTD6VACRSRQM2SCOV723Q, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090414994 (528h6m36.961498039s) [noise=true , address=145.79.227.49:4338 , piece_id=EP7IUXGYAF2LHM5BDTEK6HGN4O4OTPWTVYHWTYRWQ7CBLK26MZSQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090784278 (528h6m36.961867323s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.091246458 (528h6m36.962329503s) [noise=true , address=145.79.227.19:4078 , piece_id=UDFCQU6FOVIICU25EQOBN57JPCXNONYAW4Z2SQH7RBH2XSEFD2HA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.091740087 (528h6m36.962823122s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.093138921 (528h6m36.964221966s) [noise=true , address=145.79.227.10:4402 , piece_id=45HK3GSRJBMWALEJK3T4RZME4OZOJSU57JP4KXP6CHJZMSKK3Y4Q, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.093160833 (528h6m36.964243878s) [noise=true , address=145.79.227.48:4303 , piece_id=EKTFWLS3BYKVOL7NTVN7VIGWT3ET3MAPKM46RIFZK73ZQFST4J4A, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.094811165 (528h6m36.96589421s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.095010345 (528h6m36.96609338s) [noise=true , address=145.79.227.39:4015 , piece_id=B5G2XXRPZTWKSSA37SBWNV5BX3POFELY2UBRVQIPNS5LDGBH47PQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.096583099 (528h6m36.967666144s) [noise=true , address=145.79.227.3:4151 , piece_id=NTDEUAYE6X2QKXWO543TPLFMUDSHNXPFYZ42SDYNMIR7LQBIHIGQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.098185200 (528h6m36.969268235s) [noise=true , address=145.79.227.26:4655 , piece_id=CEBD7AHETY7SJ2563LQ57ZX6ALLYBTFIDSLTFOLOSRLPOBIS3CAA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.099692360 (528h6m36.970775406s) [noise=true , address=145.79.227.2:4113 , piece_id=VS5QG2XZA3YTDRHEO6ILLOH7H3GPSPGCL2NOP73CVA7VCBXMUDGA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:38.706744778 (528h6m38.577827823s) [upload=463210]: Next returned nil segment Dec 27 15:35:38.706800425 (528h6m38.57788346s) [upload=463210]: Waiting for error group managing segments... Dec 27 15:35:39.091009801 (528h6m38.962092847s) [noise=true , address=145.79.227.49:4338 , piece_id=EP7IUXGYAF2LHM5BDTEK6HGN4O4OTPWTVYHWTYRWQ7CBLK26MZSQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.091660060 (528h6m38.962743105s) [noise=true , address=145.79.227.12:4767 , piece_id=UYPFM2ZW6CVDQXHADJ2GJBC6KX35TBHBHSKG5BBIHZPGYDAFNEZA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.101083924 (528h6m38.972166959s) [noise=true , address=145.79.227.48:4303 , piece_id=EKTFWLS3BYKVOL7NTVN7VIGWT3ET3MAPKM46RIFZK73ZQFST4J4A, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.106377303 (528h6m38.977460338s) [noise=true , address=145.79.227.2:4113 , piece_id=VS5QG2XZA3YTDRHEO6ILLOH7H3GPSPGCL2NOP73CVA7VCBXMUDGA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.109339505 (528h6m38.98042254s) [noise=true , address=145.79.227.39:4015 , piece_id=B5G2XXRPZTWKSSA37SBWNV5BX3POFELY2UBRVQIPNS5LDGBH47PQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.109995724 (528h6m38.981078759s) [noise=true , address=145.79.227.10:4402 , piece_id=45HK3GSRJBMWALEJK3T4RZME4OZOJSU57JP4KXP6CHJZMSKK3Y4Q, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.112087207 (528h6m38.983170252s) [noise=true , address=145.79.227.26:4655 , piece_id=CEBD7AHETY7SJ2563LQ57ZX6ALLYBTFIDSLTFOLOSRLPOBIS3CAA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.124988133 (528h6m38.996071178s) [noise=true , address=145.79.227.45:4231 , piece_id=5E2OU3ID7U7JKH7MB2EHTAQZX5UZ2I2NTD6VACRSRQM2SCOV723Q, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.130435766 (528h6m39.001518801s) [noise=true , address=145.79.227.3:4151 , piece_id=NTDEUAYE6X2QKXWO543TPLFMUDSHNXPFYZ42SDYNMIR7LQBIHIGQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.133234746 (528h6m39.004317781s) [noise=true , address=145.79.227.52:4445 , piece_id=DKSWMR77ALMRSOW47G44IFI7W3HZYQWHVV4JXB5QZZEUOVPZ7QDA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.139036994 (528h6m39.010120039s) [noise=true , address=145.79.227.19:4078 , piece_id=UDFCQU6FOVIICU25EQOBN57JPCXNONYAW4Z2SQH7RBH2XSEFD2HA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.728671419 (528h6m39.599754454s) [noise=true , address=45.91.116.11:4118 , piece_id=3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.801455083 (528h6m39.672538128s) [noise=true , address=45.91.116.6:4973 , piece_id=YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.801481042 (528h6m39.672564087s) [seg_pos={0 0}, upload=463210]: Segment reached optimal threshold of 13 pieces. Dec 27 15:35:39.801753310 (528h6m39.672836345s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection Dec 27 15:35:39.801763870 (528h6m39.672846915s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.801890311 (528h6m39.672973356s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 15:35:39.801899779 (528h6m39.672982814s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.801963491 (528h6m39.673046526s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 27 15:35:39.801983249 (528h6m39.673066284s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.802063842 (528h6m39.673146877s) [seg_pos={0 0}, upload=463210]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:35:39.802150638 (528h6m39.673233683s) [upload=463210]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:35:39.802158653 (528h6m39.673241708s) [upload=463210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:35:40.131666538 (528h6m40.002749573s) [upload=463210]: Done uploading segments... Dec 27 15:35:40.131691745 (528h6m40.00277478s) [upload=463210]: Upload finished. err: <nil> Duration: 5.630780795s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [....X..X.X..X...XXXXXXX..X.X..X.X.....X...X......X.X..X..XXX] duration: 30.002976902s trace: https://jaeger.dev.storj.io/trace/1509babc40b8daba FAIL (2025-12-27 15:36:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: 3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA) context deadline exceeded ======LOGS====== Dec 27 15:35:36.834619921 (528h6m36.705702997s) [upload=463210]: Starting upload Dec 27 15:35:36.834705574 (528h6m36.705788649s) [upload=463210]: Done starting upload Dec 27 15:35:36.834811706 (528h6m36.705894761s) [upload=463210]: Uploading segments... Dec 27 15:35:36.834963957 (528h6m36.706047002s) [upload=463210]: Got next segment. Inline: false Dec 27 15:35:36.835004414 (528h6m36.706087449s) [upload=463210]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:35:36.835010325 (528h6m36.70609337s) [upload=463210]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:35:37.088013461 (528h6m36.959096506s) [seg_pos={0 0}, upload=463210]: Begin upload segment... Dec 27 15:35:37.088072674 (528h6m36.959155709s) [seg_pos={0 0}, upload=463210]: Done begin upload segment. Dec 27 15:35:37.088134742 (528h6m36.959217787s) [noise=true , address=45.91.116.6:4973 , piece_id=YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.088160852 (528h6m36.959243887s) [noise=true , address=145.79.227.12:4767 , piece_id=UYPFM2ZW6CVDQXHADJ2GJBC6KX35TBHBHSKG5BBIHZPGYDAFNEZA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.089097435 (528h6m36.96018048s) [noise=true , address=145.79.227.52:4445 , piece_id=DKSWMR77ALMRSOW47G44IFI7W3HZYQWHVV4JXB5QZZEUOVPZ7QDA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.089544757 (528h6m36.960627802s) [noise=true , address=45.91.116.11:4118 , piece_id=3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090044959 (528h6m36.961128004s) [noise=true , address=145.79.227.45:4231 , piece_id=5E2OU3ID7U7JKH7MB2EHTAQZX5UZ2I2NTD6VACRSRQM2SCOV723Q, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090414994 (528h6m36.961498039s) [noise=true , address=145.79.227.49:4338 , piece_id=EP7IUXGYAF2LHM5BDTEK6HGN4O4OTPWTVYHWTYRWQ7CBLK26MZSQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.090784278 (528h6m36.961867323s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.091246458 (528h6m36.962329503s) [noise=true , address=145.79.227.19:4078 , piece_id=UDFCQU6FOVIICU25EQOBN57JPCXNONYAW4Z2SQH7RBH2XSEFD2HA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.091740087 (528h6m36.962823122s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.093138921 (528h6m36.964221966s) [noise=true , address=145.79.227.10:4402 , piece_id=45HK3GSRJBMWALEJK3T4RZME4OZOJSU57JP4KXP6CHJZMSKK3Y4Q, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.093160833 (528h6m36.964243878s) [noise=true , address=145.79.227.48:4303 , piece_id=EKTFWLS3BYKVOL7NTVN7VIGWT3ET3MAPKM46RIFZK73ZQFST4J4A, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.094811165 (528h6m36.96589421s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.095010345 (528h6m36.96609338s) [noise=true , address=145.79.227.39:4015 , piece_id=B5G2XXRPZTWKSSA37SBWNV5BX3POFELY2UBRVQIPNS5LDGBH47PQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.096583099 (528h6m36.967666144s) [noise=true , address=145.79.227.3:4151 , piece_id=NTDEUAYE6X2QKXWO543TPLFMUDSHNXPFYZ42SDYNMIR7LQBIHIGQ, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.098185200 (528h6m36.969268235s) [noise=true , address=145.79.227.26:4655 , piece_id=CEBD7AHETY7SJ2563LQ57ZX6ALLYBTFIDSLTFOLOSRLPOBIS3CAA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:37.099692360 (528h6m36.970775406s) [noise=true , address=145.79.227.2:4113 , piece_id=VS5QG2XZA3YTDRHEO6ILLOH7H3GPSPGCL2NOP73CVA7VCBXMUDGA, seg_pos={0 0}, upload=463210]: Uploading piece... Dec 27 15:35:38.706744778 (528h6m38.577827823s) [upload=463210]: Next returned nil segment Dec 27 15:35:38.706800425 (528h6m38.57788346s) [upload=463210]: Waiting for error group managing segments... Dec 27 15:35:39.091009801 (528h6m38.962092847s) [noise=true , address=145.79.227.49:4338 , piece_id=EP7IUXGYAF2LHM5BDTEK6HGN4O4OTPWTVYHWTYRWQ7CBLK26MZSQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.091660060 (528h6m38.962743105s) [noise=true , address=145.79.227.12:4767 , piece_id=UYPFM2ZW6CVDQXHADJ2GJBC6KX35TBHBHSKG5BBIHZPGYDAFNEZA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.101083924 (528h6m38.972166959s) [noise=true , address=145.79.227.48:4303 , piece_id=EKTFWLS3BYKVOL7NTVN7VIGWT3ET3MAPKM46RIFZK73ZQFST4J4A, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.106377303 (528h6m38.977460338s) [noise=true , address=145.79.227.2:4113 , piece_id=VS5QG2XZA3YTDRHEO6ILLOH7H3GPSPGCL2NOP73CVA7VCBXMUDGA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.109339505 (528h6m38.98042254s) [noise=true , address=145.79.227.39:4015 , piece_id=B5G2XXRPZTWKSSA37SBWNV5BX3POFELY2UBRVQIPNS5LDGBH47PQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.109995724 (528h6m38.981078759s) [noise=true , address=145.79.227.10:4402 , piece_id=45HK3GSRJBMWALEJK3T4RZME4OZOJSU57JP4KXP6CHJZMSKK3Y4Q, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.112087207 (528h6m38.983170252s) [noise=true , address=145.79.227.26:4655 , piece_id=CEBD7AHETY7SJ2563LQ57ZX6ALLYBTFIDSLTFOLOSRLPOBIS3CAA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.124988133 (528h6m38.996071178s) [noise=true , address=145.79.227.45:4231 , piece_id=5E2OU3ID7U7JKH7MB2EHTAQZX5UZ2I2NTD6VACRSRQM2SCOV723Q, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.130435766 (528h6m39.001518801s) [noise=true , address=145.79.227.3:4151 , piece_id=NTDEUAYE6X2QKXWO543TPLFMUDSHNXPFYZ42SDYNMIR7LQBIHIGQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.133234746 (528h6m39.004317781s) [noise=true , address=145.79.227.52:4445 , piece_id=DKSWMR77ALMRSOW47G44IFI7W3HZYQWHVV4JXB5QZZEUOVPZ7QDA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.139036994 (528h6m39.010120039s) [noise=true , address=145.79.227.19:4078 , piece_id=UDFCQU6FOVIICU25EQOBN57JPCXNONYAW4Z2SQH7RBH2XSEFD2HA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.728671419 (528h6m39.599754454s) [noise=true , address=45.91.116.11:4118 , piece_id=3XOP6HRC4HPYMXTWYME2FGVNLFAOP4FF6Z4TWTFIFXGD6CKQVGKQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.801455083 (528h6m39.672538128s) [noise=true , address=45.91.116.6:4973 , piece_id=YAKKDQ43PN5TMNZIDDUBY2Y6PBBJJSQNZYK6Y7FZVJKC23IX7INA, seg_pos={0 0}, upload=463210]: Done uploading piece. err: <nil> Dec 27 15:35:39.801481042 (528h6m39.672564087s) [seg_pos={0 0}, upload=463210]: Segment reached optimal threshold of 13 pieces. Dec 27 15:35:39.801753310 (528h6m39.672836345s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->45.91.116.8:4045: use of closed network connection Dec 27 15:35:39.801763870 (528h6m39.672846915s) [noise=true , address=45.91.116.8:4045 , piece_id=OUYHP7MW3HACC376JXWLNDNWWRKENWK22HZFPH4E445OSMZBDAUQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.801890311 (528h6m39.672973356s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 15:35:39.801899779 (528h6m39.672982814s) [noise=true , address=45.91.116.12:4226 , piece_id=U6WFTFXCUZWJXY6JDO6PICXOSNRIJ5E4GENL63KKMIJCHATRMQQQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.801963491 (528h6m39.673046526s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 27 15:35:39.801983249 (528h6m39.673066284s) [noise=true , address=145.79.227.44:4194 , piece_id=5P363I4CST5FZD7SHSLH5JFEELAWZTPYHQ7AVEIL2H6ZZUWPLPTQ, seg_pos={0 0}, upload=463210]: Long tail cancellation due to optimal threshold Dec 27 15:35:39.802063842 (528h6m39.673146877s) [seg_pos={0 0}, upload=463210]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:35:39.802150638 (528h6m39.673233683s) [upload=463210]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:35:39.802158653 (528h6m39.673241708s) [upload=463210]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:35:40.131666538 (528h6m40.002749573s) [upload=463210]: Done uploading segments... Dec 27 15:35:40.131691745 (528h6m40.00277478s) [upload=463210]: Upload finished. err: <nil> Duration: 30.002999705s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XXXXX...X.....X...XX] duration: 5.001630445s trace: https://jaeger.dev.storj.io/trace/6da1a55e00b56666 FAIL (2025-12-27 15:36:00 UTC) orbiter warning: context deadline exceeded Duration: 5.001661403s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............X....X...................X.....XX......X......] duration: 56.17531047s trace: https://jaeger.dev.storj.io/trace/24120a988203c524 FAIL (2025-12-27 15:35:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 191MKpEnoarS7sK4Gs63ZM8zGc5jevFX8VeqViP82duW33nJJz, Piece ID: 6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A) context deadline exceeded; piecestore: (Node ID: 1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, Piece ID: JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA) context deadline exceeded; piecestore: (Node ID: 12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, Piece ID: L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA) context deadline exceeded ======LOGS====== GEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.918056404 (528h0m9.789139439s) [noise=true , address=77.90.31.13:4645 , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.918708286 (528h0m9.789791331s) [noise=true , address=77.90.31.12:4593 , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.919456690 (528h0m9.790539736s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920039571 (528h0m9.791122616s) [noise=true , address=145.79.227.35:4512 , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920276141 (528h0m9.791359186s) [noise=true , address=145.79.227.34:4926 , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920474529 (528h0m9.791557564s) [noise=true , address=145.79.227.13:4816 , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.921046528 (528h0m9.792129563s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.921603759 (528h0m9.792686794s) [noise=true , address=77.90.31.21:4883 , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.922032846 (528h0m9.793115881s) [noise=true , address=145.79.227.44:4210 , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.923593568 (528h0m9.794676613s) [noise=true , address=45.91.116.10:4103 , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.924048133 (528h0m9.795131169s) [noise=true , address=77.90.31.18:4840 , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.924539719 (528h0m9.795622754s) [noise=true , address=64.49.13.139:4470 , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.925024673 (528h0m9.796107708s) [noise=true , address=145.79.227.49:4368 , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.925500990 (528h0m9.796584035s) [noise=true , address=77.90.31.17:4795 , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:11.205949065 (528h0m11.0770321s) [upload=463114]: Next returned nil segment Dec 27 15:29:11.205963343 (528h0m11.077046378s) [upload=463114]: Waiting for error group managing segments... Dec 27 15:29:11.610155861 (528h0m11.481238906s) [noise=true , address=64.49.13.139:4470 , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.812270605 (528h0m11.683353641s) [noise=true , address=145.79.227.35:4512 , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.818242477 (528h0m11.689325522s) [noise=true , address=145.79.227.13:4816 , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.938211621 (528h0m11.809294666s) [noise=true , address=145.79.227.49:4368 , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.064896693 (528h0m11.935979738s) [noise=true , address=77.90.31.17:4795 , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.070731273 (528h0m11.941814308s) [noise=true , address=77.90.31.21:4883 , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.107949174 (528h0m11.979032239s) [noise=true , address=77.90.31.11:4573 , piece_id=SRXHTEXDEKKJ2WPGY66BOAP7WD7Z75HK3B5DL2XIGFWDYQGI76YA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.151478532 (528h0m12.022561567s) [noise=true , address=77.90.31.13:4645 , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.335397247 (528h0m12.206480292s) [noise=true , address=45.91.116.10:4103 , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.410397992 (528h0m12.281481027s) [noise=true , address=77.90.31.18:4840 , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.667101894 (528h0m12.538184929s) [noise=true , address=77.90.31.12:4593 , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.785380610 (528h0m12.656463655s) [noise=true , address=145.79.227.44:4210 , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.828722731 (528h0m12.699805766s) [noise=true , address=193.149.164.143:4697 , piece_id=S4CQ3366KVFS5RMZR67MPGSLXKDWRAGEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.947757466 (528h0m12.818840501s) [noise=true , address=145.79.227.34:4926 , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.947777434 (528h0m12.818860469s) [seg_pos={0 0}, upload=463114]: Segment reached optimal threshold of 14 pieces. Dec 27 15:29:12.947947177 (528h0m12.819030212s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy): protocol: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection Dec 27 15:29:12.947953169 (528h0m12.819036204s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948200800 (528h0m12.819283845s) [noise=true , address=77.90.31.19:4869 , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8): protocol: read tcp 188.166.225.170:32932->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 15:29:12.948216079 (528h0m12.819299124s) [noise=true , address=77.90.31.19:4869 , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948310158 (528h0m12.819393203s) [noise=true , address=64.49.13.169:4711 , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection Dec 27 15:29:12.948316139 (528h0m12.819399174s) [noise=true , address=64.49.13.169:4711 , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948360164 (528h0m12.819443219s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 27 15:29:12.948372617 (528h0m12.819455652s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948398968 (528h0m12.819482013s) [seg_pos={0 0}, upload=463114]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:29:12.948477878 (528h0m12.819560913s) [upload=463114]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:29:12.948483889 (528h0m12.819566934s) [upload=463114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:29:13.237238452 (528h0m13.108321498s) [upload=463114]: Done uploading segments... Dec 27 15:29:13.237262939 (528h0m13.108345984s) [upload=463114]: Upload finished. err: <nil> Duration: 56.175328104s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.....X..X.X..X...XXXXXXX..X.X..X.X.....X...X......X.X..X..XX] duration: 30.003408884s trace: https://jaeger.dev.storj.io/trace/483a3e79dcda5b86 FAIL (2025-12-27 15:35:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: VNRHTBLMJZ3XCB5VVOX2LTI7AT6QJATVCMCUTK6SBVSGCW23NASQ) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: PLKXCKHAQTFAQQKVT6ECVB46JAXUFBLWIPECA6YGCUBHIWB4GUTA) context deadline exceeded ======LOGS====== Dec 27 15:34:40.001108485 (528h5m39.87219155s) [upload=463199]: Starting upload Dec 27 15:34:40.001197174 (528h5m39.872280209s) [upload=463199]: Done starting upload Dec 27 15:34:40.001434667 (528h5m39.872517702s) [upload=463199]: Uploading segments... Dec 27 15:34:40.001503568 (528h5m39.872586603s) [upload=463199]: Got next segment. Inline: false Dec 27 15:34:40.001596103 (528h5m39.872679158s) [upload=463199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:34:40.001603217 (528h5m39.872686252s) [upload=463199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:34:40.275248730 (528h5m40.146331765s) [seg_pos={0 0}, upload=463199]: Begin upload segment... Dec 27 15:34:40.275293044 (528h5m40.146376089s) [seg_pos={0 0}, upload=463199]: Done begin upload segment. Dec 27 15:34:40.275325426 (528h5m40.146408461s) [noise=true , address=145.79.227.16:4912 , piece_id=NPKLSQYVPUWCVCACKOZKZQ3HJRGCBFU5SIRKCCG64NQXFTLMBEZQ, seg_pos={0 0}, upload=463199]: Uploading piece... Dec 27 15:34:40.275348870 (528h5m40.146431915s) [noise=true , address=145.79.227.4:4186 , piece_id=CCBNBERF4EVD3O5ORXENMJHJUX5NH4PLH2WKEY36VP3X3B5SJ5RQ, seg_pos={0 0}, upload=463199]: Uploading piece... Dec 27 15:34:40.275444252 (528h5m40.146527297s) [noise=true , address=145.79.227.29:4943 , piece_id=TDETSVAIZUUZJQ2MJVAGX7UFWZLRN5JKXMWRVK5ZHHESPHWHO2CA, seg_pos={0 0}, upload=463199]: Uploading piece... Dec 27 15:34:40.275830056 (528h5m40.146913091s) [noise=true , address=145.79.227.51:4411 , piece_id=EXUQJO73DVBOHLE765DOXCZNIHA3XFHYP3NBVZNREAUE7MZ4POPA, seg_pos={0 0}, upload=463199]: Uploading piece... Dec 27 15:34:40.276179592 (528h5m40.147262627s) [noise=true , address=77.90.31.18:4808 , piece_id=VNRHTBLMJZ3XCB5VVOX2LTI7AT6QJATVCMCUTK6SBVSGCW23NASQ, seg_pos={0 0}, upload=463199]: Uploading piece... Dec 27 15:34:40.277932711 (528h5m40.149015756s) [noise=true , address=145.79.227.42:4121 , piece_id=OQ2AW3SKWJVHJSJYMLY33ACYDYQW7CAQQIISU3E6N6UMVJ4DN6EQ, seg_pos={0 0}, upload=463199]: Uploading piece... Dec 27 15:34:40.278183378 (528h5m40.149266423s) [noise=true , address=145.79.227.21:4474 , piece_id=IGVY6DBZU3PED7QDMW72XNZTBDB6253WBS5BN3TKOYU2AMX5WAXA, seg_pos={0 0}, upload=463199]: Uploading piece... Dec 27 15:34:40.283373992 (528h5m40.154457037s) [noise=true , address=45.91.116.9:4262 , piece_id=FPTV2BPMYLTORYWB3XMTAM3SUJHDUUEGGURRGBTBBCC3SFAURGQA, seg_pos={0 0}, upload=463199]: Uploading piece... Dec 27 15:34:40.284994808 (528h5m40.156077843s) [noise=true , address=145.79.227.11:4730 , piece_id=B6G4FARXYYAIOEJKVXOU63DZAJU7QV5J7UTE44FSWICESCDEB75A, seg_pos={0 0}, upload=463199]: Uploading piece... Dec 27 15:34:40.285655516 (528h5m40.156738562s) [noise=true , address=45.91.116.7:4009 , piece_id=PLKXCKHAQTFAQQKVT6ECVB46JAXUFBLWIPECA6YGCUBHIWB4GUTA, seg_pos={0 0}, upload=463199]: Uploading piece... Dec 27 15:34:40.287096520 (528h5m40.158179555s) [noise=true , address=145.79.227.24:4581 , piece_id=CLGOFPOJS2EOO2QIDAH5WLCDRQ7MWNVAWBL3IYE2WCJ7WLDBH76A, seg_pos={0 0}, upload=463199]: Uploading piece... Dec 27 15:34:40.288257831 (528h5m40.159340866s) [noise=true , address=145.79.227.12:4765 , piece_id=UA2CLOYXY3JGNRY6IRF2KRUIAI3S2YUV3KRU7DSS5NSFGCHHTZSA, seg_pos={0 0}, upload=463199]: Uploading piece... Dec 27 15:34:40.288857744 (528h5m40.159940779s) [noise=true , address=145.79.227.45:4230 , piece_id=VLGSP6N4LMB3LGI4AMEKVRVVBWZUTP7O6GVKBYVLTYW3D3CQZV6A, seg_pos={0 0}, upload=463199]: Uploading piece... Dec 27 15:34:40.290359904 (528h5m40.161442959s) [noise=true , address=77.90.31.21:4880 , piece_id=WICDIUHLXEM44HLNHB5OQN5A36METYKDOF6VXYOY5PENP3KHCHHA, seg_pos={0 0}, upload=463199]: Uploading piece... Dec 27 15:34:40.290793259 (528h5m40.161876314s) [noise=true , address=45.91.116.14:4190 , piece_id=WNS6CKKSYT7PW27TVCL57BPWY4VMOQ3UBGNF7YA4M5BKAY77LZXQ, seg_pos={0 0}, upload=463199]: Uploading piece... Dec 27 15:34:40.292160874 (528h5m40.163243909s) [noise=true , address=145.79.227.46:4603 , piece_id=FH7X2XQK6UFVIVHS4YZ2CIK5AUINJ5MHTJD52ENIV3L6XL7MA7DQ, seg_pos={0 0}, upload=463199]: Uploading piece... Dec 27 15:34:41.896979980 (528h5m41.768063025s) [upload=463199]: Next returned nil segment Dec 27 15:34:41.896998114 (528h5m41.768081159s) [upload=463199]: Waiting for error group managing segments... Dec 27 15:34:42.282200472 (528h5m42.153283517s) [noise=true , address=145.79.227.16:4912 , piece_id=NPKLSQYVPUWCVCACKOZKZQ3HJRGCBFU5SIRKCCG64NQXFTLMBEZQ, seg_pos={0 0}, upload=463199]: Done uploading piece. err: <nil> Dec 27 15:34:42.282727055 (528h5m42.15381009s) [noise=true , address=145.79.227.4:4186 , piece_id=CCBNBERF4EVD3O5ORXENMJHJUX5NH4PLH2WKEY36VP3X3B5SJ5RQ, seg_pos={0 0}, upload=463199]: Done uploading piece. err: <nil> Dec 27 15:34:42.289445829 (528h5m42.160528874s) [noise=true , address=145.79.227.51:4411 , piece_id=EXUQJO73DVBOHLE765DOXCZNIHA3XFHYP3NBVZNREAUE7MZ4POPA, seg_pos={0 0}, upload=463199]: Done uploading piece. err: <nil> Dec 27 15:34:42.292255028 (528h5m42.163338063s) [noise=true , address=145.79.227.42:4121 , piece_id=OQ2AW3SKWJVHJSJYMLY33ACYDYQW7CAQQIISU3E6N6UMVJ4DN6EQ, seg_pos={0 0}, upload=463199]: Done uploading piece. err: <nil> Dec 27 15:34:42.296660147 (528h5m42.167743192s) [noise=true , address=145.79.227.12:4765 , piece_id=UA2CLOYXY3JGNRY6IRF2KRUIAI3S2YUV3KRU7DSS5NSFGCHHTZSA, seg_pos={0 0}, upload=463199]: Done uploading piece. err: <nil> Dec 27 15:34:42.299046230 (528h5m42.170129265s) [noise=true , address=145.79.227.29:4943 , piece_id=TDETSVAIZUUZJQ2MJVAGX7UFWZLRN5JKXMWRVK5ZHHESPHWHO2CA, seg_pos={0 0}, upload=463199]: Done uploading piece. err: <nil> Dec 27 15:34:42.301507246 (528h5m42.172590291s) [noise=true , address=145.79.227.11:4730 , piece_id=B6G4FARXYYAIOEJKVXOU63DZAJU7QV5J7UTE44FSWICESCDEB75A, seg_pos={0 0}, upload=463199]: Done uploading piece. err: <nil> Dec 27 15:34:42.302501149 (528h5m42.173584194s) [noise=true , address=145.79.227.21:4474 , piece_id=IGVY6DBZU3PED7QDMW72XNZTBDB6253WBS5BN3TKOYU2AMX5WAXA, seg_pos={0 0}, upload=463199]: Done uploading piece. err: <nil> Dec 27 15:34:42.306976200 (528h5m42.178059235s) [noise=true , address=145.79.227.46:4603 , piece_id=FH7X2XQK6UFVIVHS4YZ2CIK5AUINJ5MHTJD52ENIV3L6XL7MA7DQ, seg_pos={0 0}, upload=463199]: Done uploading piece. err: <nil> Dec 27 15:34:42.315025157 (528h5m42.186108192s) [noise=true , address=145.79.227.24:4581 , piece_id=CLGOFPOJS2EOO2QIDAH5WLCDRQ7MWNVAWBL3IYE2WCJ7WLDBH76A, seg_pos={0 0}, upload=463199]: Done uploading piece. err: <nil> Dec 27 15:34:42.317702294 (528h5m42.188785339s) [noise=true , address=145.79.227.45:4230 , piece_id=VLGSP6N4LMB3LGI4AMEKVRVVBWZUTP7O6GVKBYVLTYW3D3CQZV6A, seg_pos={0 0}, upload=463199]: Done uploading piece. err: <nil> Dec 27 15:34:42.565403845 (528h5m42.43648689s) [noise=true , address=45.91.116.7:4009 , piece_id=PLKXCKHAQTFAQQKVT6ECVB46JAXUFBLWIPECA6YGCUBHIWB4GUTA, seg_pos={0 0}, upload=463199]: Done uploading piece. err: <nil> Dec 27 15:34:42.618370403 (528h5m42.489453438s) [noise=true , address=77.90.31.18:4808 , piece_id=VNRHTBLMJZ3XCB5VVOX2LTI7AT6QJATVCMCUTK6SBVSGCW23NASQ, seg_pos={0 0}, upload=463199]: Done uploading piece. err: <nil> Dec 27 15:34:42.618394229 (528h5m42.489477264s) [seg_pos={0 0}, upload=463199]: Segment reached optimal threshold of 13 pieces. Dec 27 15:34:42.618582538 (528h5m42.489665573s) [noise=true , address=77.90.31.21:4880 , piece_id=WICDIUHLXEM44HLNHB5OQN5A36METYKDOF6VXYOY5PENP3KHCHHA, seg_pos={0 0}, upload=463199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 27 15:34:42.618589371 (528h5m42.489672406s) [noise=true , address=77.90.31.21:4880 , piece_id=WICDIUHLXEM44HLNHB5OQN5A36METYKDOF6VXYOY5PENP3KHCHHA, seg_pos={0 0}, upload=463199]: Long tail cancellation due to optimal threshold Dec 27 15:34:42.618778300 (528h5m42.489861335s) [noise=true , address=45.91.116.14:4190 , piece_id=WNS6CKKSYT7PW27TVCL57BPWY4VMOQ3UBGNF7YA4M5BKAY77LZXQ, seg_pos={0 0}, upload=463199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:53282->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:53282->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53282->45.91.116.14:4190: use of closed network connection Dec 27 15:34:42.618785313 (528h5m42.489868348s) [noise=true , address=45.91.116.14:4190 , piece_id=WNS6CKKSYT7PW27TVCL57BPWY4VMOQ3UBGNF7YA4M5BKAY77LZXQ, seg_pos={0 0}, upload=463199]: Long tail cancellation due to optimal threshold Dec 27 15:34:42.618960947 (528h5m42.490044002s) [noise=true , address=45.91.116.9:4262 , piece_id=FPTV2BPMYLTORYWB3XMTAM3SUJHDUUEGGURRGBTBBCC3SFAURGQA, seg_pos={0 0}, upload=463199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 27 15:34:42.618983901 (528h5m42.490066967s) [noise=true , address=45.91.116.9:4262 , piece_id=FPTV2BPMYLTORYWB3XMTAM3SUJHDUUEGGURRGBTBBCC3SFAURGQA, seg_pos={0 0}, upload=463199]: Long tail cancellation due to optimal threshold Dec 27 15:34:42.619042223 (528h5m42.490125268s) [seg_pos={0 0}, upload=463199]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:34:42.619201085 (528h5m42.49028415s) [upload=463199]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:34:42.619209171 (528h5m42.490292216s) [upload=463199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:34:42.910316264 (528h5m42.781399309s) [upload=463199]: Done uploading segments... Dec 27 15:34:42.910340139 (528h5m42.781423184s) [upload=463199]: Upload finished. err: <nil> Duration: 30.003449061s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [X..............X....X...................X.....XX......X.....] duration: 48.931585545s trace: https://jaeger.dev.storj.io/trace/24120a988203c524 FAIL (2025-12-27 15:34:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 191MKpEnoarS7sK4Gs63ZM8zGc5jevFX8VeqViP82duW33nJJz, Piece ID: 6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A) context deadline exceeded; piecestore: (Node ID: 1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, Piece ID: JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA) context deadline exceeded; piecestore: (Node ID: 12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, Piece ID: L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA) context deadline exceeded ======LOGS====== GEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.918056404 (528h0m9.789139439s) [noise=true , address=77.90.31.13:4645 , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.918708286 (528h0m9.789791331s) [noise=true , address=77.90.31.12:4593 , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.919456690 (528h0m9.790539736s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920039571 (528h0m9.791122616s) [noise=true , address=145.79.227.35:4512 , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920276141 (528h0m9.791359186s) [noise=true , address=145.79.227.34:4926 , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920474529 (528h0m9.791557564s) [noise=true , address=145.79.227.13:4816 , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.921046528 (528h0m9.792129563s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.921603759 (528h0m9.792686794s) [noise=true , address=77.90.31.21:4883 , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.922032846 (528h0m9.793115881s) [noise=true , address=145.79.227.44:4210 , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.923593568 (528h0m9.794676613s) [noise=true , address=45.91.116.10:4103 , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.924048133 (528h0m9.795131169s) [noise=true , address=77.90.31.18:4840 , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.924539719 (528h0m9.795622754s) [noise=true , address=64.49.13.139:4470 , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.925024673 (528h0m9.796107708s) [noise=true , address=145.79.227.49:4368 , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.925500990 (528h0m9.796584035s) [noise=true , address=77.90.31.17:4795 , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:11.205949065 (528h0m11.0770321s) [upload=463114]: Next returned nil segment Dec 27 15:29:11.205963343 (528h0m11.077046378s) [upload=463114]: Waiting for error group managing segments... Dec 27 15:29:11.610155861 (528h0m11.481238906s) [noise=true , address=64.49.13.139:4470 , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.812270605 (528h0m11.683353641s) [noise=true , address=145.79.227.35:4512 , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.818242477 (528h0m11.689325522s) [noise=true , address=145.79.227.13:4816 , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.938211621 (528h0m11.809294666s) [noise=true , address=145.79.227.49:4368 , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.064896693 (528h0m11.935979738s) [noise=true , address=77.90.31.17:4795 , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.070731273 (528h0m11.941814308s) [noise=true , address=77.90.31.21:4883 , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.107949174 (528h0m11.979032239s) [noise=true , address=77.90.31.11:4573 , piece_id=SRXHTEXDEKKJ2WPGY66BOAP7WD7Z75HK3B5DL2XIGFWDYQGI76YA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.151478532 (528h0m12.022561567s) [noise=true , address=77.90.31.13:4645 , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.335397247 (528h0m12.206480292s) [noise=true , address=45.91.116.10:4103 , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.410397992 (528h0m12.281481027s) [noise=true , address=77.90.31.18:4840 , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.667101894 (528h0m12.538184929s) [noise=true , address=77.90.31.12:4593 , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.785380610 (528h0m12.656463655s) [noise=true , address=145.79.227.44:4210 , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.828722731 (528h0m12.699805766s) [noise=true , address=193.149.164.143:4697 , piece_id=S4CQ3366KVFS5RMZR67MPGSLXKDWRAGEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.947757466 (528h0m12.818840501s) [noise=true , address=145.79.227.34:4926 , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.947777434 (528h0m12.818860469s) [seg_pos={0 0}, upload=463114]: Segment reached optimal threshold of 14 pieces. Dec 27 15:29:12.947947177 (528h0m12.819030212s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy): protocol: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection Dec 27 15:29:12.947953169 (528h0m12.819036204s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948200800 (528h0m12.819283845s) [noise=true , address=77.90.31.19:4869 , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8): protocol: read tcp 188.166.225.170:32932->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 15:29:12.948216079 (528h0m12.819299124s) [noise=true , address=77.90.31.19:4869 , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948310158 (528h0m12.819393203s) [noise=true , address=64.49.13.169:4711 , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection Dec 27 15:29:12.948316139 (528h0m12.819399174s) [noise=true , address=64.49.13.169:4711 , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948360164 (528h0m12.819443219s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 27 15:29:12.948372617 (528h0m12.819455652s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948398968 (528h0m12.819482013s) [seg_pos={0 0}, upload=463114]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:29:12.948477878 (528h0m12.819560913s) [upload=463114]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:29:12.948483889 (528h0m12.819566934s) [upload=463114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:29:13.237238452 (528h0m13.108321498s) [upload=463114]: Done uploading segments... Dec 27 15:29:13.237262939 (528h0m13.108345984s) [upload=463114]: Upload finished. err: <nil> Duration: 48.93161844s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [.X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XXXXX...X.....X...X] duration: 5.000604688s trace: https://jaeger.dev.storj.io/trace/9913457140563cd FAIL (2025-12-27 15:34:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000630616s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [......X..X.X..X...XXXXXXX..X.X..X.X.....X...X......X.X..X..X] duration: 30.003577691s trace: https://jaeger.dev.storj.io/trace/413b43a1104be5a8 FAIL (2025-12-27 15:34:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 15:33:40.904542674 (528h4m40.77562571s) [upload=463185]: Starting upload Dec 27 15:33:40.904603450 (528h4m40.775686485s) [upload=463185]: Done starting upload Dec 27 15:33:40.904778253 (528h4m40.775861298s) [upload=463185]: Uploading segments... Dec 27 15:33:40.904814342 (528h4m40.775897377s) [upload=463185]: Got next segment. Inline: false Dec 27 15:33:40.904842285 (528h4m40.77592532s) [upload=463185]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:33:40.904846904 (528h4m40.775929939s) [upload=463185]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:33:41.154525763 (528h4m41.025608798s) [seg_pos={0 0}, upload=463185]: Begin upload segment... Dec 27 15:33:41.154583333 (528h4m41.025666378s) [seg_pos={0 0}, upload=463185]: Done begin upload segment. Dec 27 15:33:41.154616536 (528h4m41.025699571s) [noise=true , address=45.91.116.11:4120 , piece_id=T4FZLGAHROKTPLQU4MGZLNNAYMCDTSERKRHJ7ZS5SDM3GQ5FYILQ, seg_pos={0 0}, upload=463185]: Uploading piece... Dec 27 15:33:41.154714743 (528h4m41.025797858s) [noise=true , address=45.91.116.8:4047 , piece_id=ZI4EZQDCQ6P4P2PLDWDWSTI2VGCIGYOKPYHMT3BVK3H2VUBJPXAA, seg_pos={0 0}, upload=463185]: Uploading piece... Dec 27 15:33:41.155512682 (528h4m41.026595737s) [noise=true , address=145.79.227.12:4765 , piece_id=TTDSQMFLHGVX2LC226QUQ3VAT6CNYTCAZJ6RDQ62YW4K6YIIS3SA, seg_pos={0 0}, upload=463185]: Uploading piece... Dec 27 15:33:41.156144426 (528h4m41.027227471s) [noise=true , address=145.79.227.28:4727 , piece_id=JC5CJENHAVAMJW7YNOFVTE5Q2TZ3F4PJ3NCKNESHEX2RTQVZZCUA, seg_pos={0 0}, upload=463185]: Uploading piece... Dec 27 15:33:41.156713118 (528h4m41.027796164s) [noise=true , address=145.79.227.3:4149 , piece_id=XSQB5L5UWZ3WGQW6RCQYQHSGQWSCJZQ6XXY5R3BJ2ERW6JHOJF5A, seg_pos={0 0}, upload=463185]: Uploading piece... Dec 27 15:33:41.157258427 (528h4m41.028341472s) [noise=true , address=45.91.116.14:4190 , piece_id=PAN4HVLFJYEVEBLFSZ7WKSVTFDSSIEOIV34UZABKYLD4TDEINJ6Q, seg_pos={0 0}, upload=463185]: Uploading piece... Dec 27 15:33:41.157941177 (528h4m41.029024232s) [noise=true , address=145.79.227.51:4409 , piece_id=JB3MYE44TWNZK3PDRAHCCDNOSZLXR4BIRRQS4F5LQCTUMH4OERNA, seg_pos={0 0}, upload=463185]: Uploading piece... Dec 27 15:33:41.158502125 (528h4m41.02958519s) [noise=true , address=145.79.227.42:4121 , piece_id=QQRVBFCC55S4KKZQJCDYXK5U25BVZ6MTKBFENIBCNUWJE27VKDBA, seg_pos={0 0}, upload=463185]: Uploading piece... Dec 27 15:33:41.159878015 (528h4m41.03096105s) [noise=true , address=145.79.227.36:4519 , piece_id=7VOXDOQOMR7KNO4CGOJBIXNHIHPEIYSZWUO5I7GJERQB77KZKOAA, seg_pos={0 0}, upload=463185]: Uploading piece... Dec 27 15:33:41.161330010 (528h4m41.032413055s) [noise=true , address=145.79.227.4:4186 , piece_id=NJWRCXAND6X34SL334WVLIP3A2SCQCPENASQFQANTO2L5XUUA4QA, seg_pos={0 0}, upload=463185]: Uploading piece... Dec 27 15:33:41.161564756 (528h4m41.032647802s) [noise=true , address=77.90.31.10:4518 , piece_id=YO3KPNEV2TW55LXF5HLX7JUTTJY4Q3VGNDQNTK6MFRXVBLTGQ63A, seg_pos={0 0}, upload=463185]: Uploading piece... Dec 27 15:33:41.163366808 (528h4m41.034449853s) [noise=true , address=45.91.116.5:4299 , piece_id=JH5ZWKP6LW3ZXH4UIDEAQWCMESFLD4VDPYE77XW3XR5PVSDROL6Q, seg_pos={0 0}, upload=463185]: Uploading piece... Dec 27 15:33:41.164938010 (528h4m41.036021045s) [noise=true , address=45.91.116.10:4084 , piece_id=GCWEO5PQ2RHP74BV3IGNRCT5L4RGAAETQ45VBSRQFPQ6DLFAUERA, seg_pos={0 0}, upload=463185]: Uploading piece... Dec 27 15:33:41.166719382 (528h4m41.037802457s) [noise=true , address=145.79.227.11:4730 , piece_id=CRK7XWZUTMVTSIVNMVGFZNS6KZUODK6D6AJIZ2WQZALUM7ENBC7A, seg_pos={0 0}, upload=463185]: Uploading piece... Dec 27 15:33:41.169021977 (528h4m41.040105022s) [noise=true , address=45.91.116.6:4974 , piece_id=3WGYMX4CQCWF7RRR5M3LHLADVDFGVSUU6JJ7QQMMEB4DTAWA5BZQ, seg_pos={0 0}, upload=463185]: Uploading piece... Dec 27 15:33:41.170732254 (528h4m41.041815299s) [noise=true , address=145.79.227.45:4231 , piece_id=T62ZF2IXZ2ODXWFJSEAIRJISYC55YKLBECESVQOKJCSZRGNU5QKA, seg_pos={0 0}, upload=463185]: Uploading piece... Dec 27 15:33:42.881552113 (528h4m42.752635168s) [upload=463185]: Next returned nil segment Dec 27 15:33:42.881568915 (528h4m42.75265196s) [upload=463185]: Waiting for error group managing segments... Dec 27 15:33:43.165145187 (528h4m43.036228232s) [noise=true , address=145.79.227.4:4186 , piece_id=NJWRCXAND6X34SL334WVLIP3A2SCQCPENASQFQANTO2L5XUUA4QA, seg_pos={0 0}, upload=463185]: Done uploading piece. err: <nil> Dec 27 15:33:43.170998913 (528h4m43.042081948s) [noise=true , address=145.79.227.12:4765 , piece_id=TTDSQMFLHGVX2LC226QUQ3VAT6CNYTCAZJ6RDQ62YW4K6YIIS3SA, seg_pos={0 0}, upload=463185]: Done uploading piece. err: <nil> Dec 27 15:33:43.180219401 (528h4m43.051302436s) [noise=true , address=145.79.227.45:4231 , piece_id=T62ZF2IXZ2ODXWFJSEAIRJISYC55YKLBECESVQOKJCSZRGNU5QKA, seg_pos={0 0}, upload=463185]: Done uploading piece. err: <nil> Dec 27 15:33:43.187523790 (528h4m43.058606825s) [noise=true , address=145.79.227.3:4149 , piece_id=XSQB5L5UWZ3WGQW6RCQYQHSGQWSCJZQ6XXY5R3BJ2ERW6JHOJF5A, seg_pos={0 0}, upload=463185]: Done uploading piece. err: <nil> Dec 27 15:33:43.188058318 (528h4m43.059141363s) [noise=true , address=145.79.227.42:4121 , piece_id=QQRVBFCC55S4KKZQJCDYXK5U25BVZ6MTKBFENIBCNUWJE27VKDBA, seg_pos={0 0}, upload=463185]: Done uploading piece. err: <nil> Dec 27 15:33:43.188181021 (528h4m43.059264057s) [noise=true , address=145.79.227.28:4727 , piece_id=JC5CJENHAVAMJW7YNOFVTE5Q2TZ3F4PJ3NCKNESHEX2RTQVZZCUA, seg_pos={0 0}, upload=463185]: Done uploading piece. err: <nil> Dec 27 15:33:43.192558768 (528h4m43.063641803s) [noise=true , address=145.79.227.51:4409 , piece_id=JB3MYE44TWNZK3PDRAHCCDNOSZLXR4BIRRQS4F5LQCTUMH4OERNA, seg_pos={0 0}, upload=463185]: Done uploading piece. err: <nil> Dec 27 15:33:43.201197086 (528h4m43.072280131s) [noise=true , address=145.79.227.11:4730 , piece_id=CRK7XWZUTMVTSIVNMVGFZNS6KZUODK6D6AJIZ2WQZALUM7ENBC7A, seg_pos={0 0}, upload=463185]: Done uploading piece. err: <nil> Dec 27 15:33:43.203203828 (528h4m43.074286863s) [noise=true , address=145.79.227.36:4519 , piece_id=7VOXDOQOMR7KNO4CGOJBIXNHIHPEIYSZWUO5I7GJERQB77KZKOAA, seg_pos={0 0}, upload=463185]: Done uploading piece. err: <nil> Dec 27 15:33:43.460559332 (528h4m43.331642367s) [noise=true , address=45.91.116.6:4974 , piece_id=3WGYMX4CQCWF7RRR5M3LHLADVDFGVSUU6JJ7QQMMEB4DTAWA5BZQ, seg_pos={0 0}, upload=463185]: Done uploading piece. err: <nil> Dec 27 15:33:43.488865131 (528h4m43.359948186s) [noise=true , address=45.91.116.14:4190 , piece_id=PAN4HVLFJYEVEBLFSZ7WKSVTFDSSIEOIV34UZABKYLD4TDEINJ6Q, seg_pos={0 0}, upload=463185]: Done uploading piece. err: <nil> Dec 27 15:33:43.519728401 (528h4m43.390811446s) [noise=true , address=45.91.116.8:4047 , piece_id=ZI4EZQDCQ6P4P2PLDWDWSTI2VGCIGYOKPYHMT3BVK3H2VUBJPXAA, seg_pos={0 0}, upload=463185]: Done uploading piece. err: <nil> Dec 27 15:33:43.522657378 (528h4m43.393740413s) [noise=true , address=45.91.116.5:4299 , piece_id=JH5ZWKP6LW3ZXH4UIDEAQWCMESFLD4VDPYE77XW3XR5PVSDROL6Q, seg_pos={0 0}, upload=463185]: Done uploading piece. err: <nil> Dec 27 15:33:43.522675653 (528h4m43.393758698s) [seg_pos={0 0}, upload=463185]: Segment reached optimal threshold of 13 pieces. Dec 27 15:33:43.523070214 (528h4m43.394153259s) [noise=true , address=45.91.116.11:4120 , piece_id=T4FZLGAHROKTPLQU4MGZLNNAYMCDTSERKRHJ7ZS5SDM3GQ5FYILQ, seg_pos={0 0}, upload=463185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 15:33:43.523085403 (528h4m43.394168438s) [noise=true , address=45.91.116.11:4120 , piece_id=T4FZLGAHROKTPLQU4MGZLNNAYMCDTSERKRHJ7ZS5SDM3GQ5FYILQ, seg_pos={0 0}, upload=463185]: Long tail cancellation due to optimal threshold Dec 27 15:33:43.523202115 (528h4m43.39428515s) [noise=true , address=45.91.116.10:4084 , piece_id=GCWEO5PQ2RHP74BV3IGNRCT5L4RGAAETQ45VBSRQFPQ6DLFAUERA, seg_pos={0 0}, upload=463185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: read tcp 188.166.225.170:41238->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 15:33:43.523240118 (528h4m43.394323163s) [noise=true , address=45.91.116.10:4084 , piece_id=GCWEO5PQ2RHP74BV3IGNRCT5L4RGAAETQ45VBSRQFPQ6DLFAUERA, seg_pos={0 0}, upload=463185]: Long tail cancellation due to optimal threshold Dec 27 15:33:43.523406074 (528h4m43.394489109s) [noise=true , address=77.90.31.10:4518 , piece_id=YO3KPNEV2TW55LXF5HLX7JUTTJY4Q3VGNDQNTK6MFRXVBLTGQ63A, seg_pos={0 0}, upload=463185]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 27 15:33:43.523413929 (528h4m43.394496974s) [noise=true , address=77.90.31.10:4518 , piece_id=YO3KPNEV2TW55LXF5HLX7JUTTJY4Q3VGNDQNTK6MFRXVBLTGQ63A, seg_pos={0 0}, upload=463185]: Long tail cancellation due to optimal threshold Dec 27 15:33:43.523439127 (528h4m43.394522162s) [seg_pos={0 0}, upload=463185]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:33:43.523522906 (528h4m43.394605941s) [upload=463185]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:33:43.523527845 (528h4m43.39461088s) [upload=463185]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:33:43.822106185 (528h4m43.69318922s) [upload=463185]: Done uploading segments... Dec 27 15:33:43.822126132 (528h4m43.693209167s) [upload=463185]: Upload finished. err: <nil> Duration: 30.003614792s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XXXXX...X.....X...] duration: 5.000603241s trace: https://jaeger.dev.storj.io/trace/57ec6320e984041b FAIL (2025-12-27 15:33:42 UTC) orbiter warning: context deadline exceeded Duration: 5.000618529s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X..............X....X...................X.....XX......X....] duration: 15.461190431s trace: https://jaeger.dev.storj.io/trace/24120a988203c524 FAIL (2025-12-27 15:33:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 191MKpEnoarS7sK4Gs63ZM8zGc5jevFX8VeqViP82duW33nJJz, Piece ID: 6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A) context deadline exceeded; piecestore: (Node ID: 1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, Piece ID: JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA) context deadline exceeded; piecestore: (Node ID: 12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, Piece ID: L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA) context deadline exceeded ======LOGS====== GEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.918056404 (528h0m9.789139439s) [noise=true , address=77.90.31.13:4645 , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.918708286 (528h0m9.789791331s) [noise=true , address=77.90.31.12:4593 , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.919456690 (528h0m9.790539736s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920039571 (528h0m9.791122616s) [noise=true , address=145.79.227.35:4512 , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920276141 (528h0m9.791359186s) [noise=true , address=145.79.227.34:4926 , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920474529 (528h0m9.791557564s) [noise=true , address=145.79.227.13:4816 , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.921046528 (528h0m9.792129563s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.921603759 (528h0m9.792686794s) [noise=true , address=77.90.31.21:4883 , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.922032846 (528h0m9.793115881s) [noise=true , address=145.79.227.44:4210 , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.923593568 (528h0m9.794676613s) [noise=true , address=45.91.116.10:4103 , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.924048133 (528h0m9.795131169s) [noise=true , address=77.90.31.18:4840 , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.924539719 (528h0m9.795622754s) [noise=true , address=64.49.13.139:4470 , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.925024673 (528h0m9.796107708s) [noise=true , address=145.79.227.49:4368 , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.925500990 (528h0m9.796584035s) [noise=true , address=77.90.31.17:4795 , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:11.205949065 (528h0m11.0770321s) [upload=463114]: Next returned nil segment Dec 27 15:29:11.205963343 (528h0m11.077046378s) [upload=463114]: Waiting for error group managing segments... Dec 27 15:29:11.610155861 (528h0m11.481238906s) [noise=true , address=64.49.13.139:4470 , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.812270605 (528h0m11.683353641s) [noise=true , address=145.79.227.35:4512 , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.818242477 (528h0m11.689325522s) [noise=true , address=145.79.227.13:4816 , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.938211621 (528h0m11.809294666s) [noise=true , address=145.79.227.49:4368 , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.064896693 (528h0m11.935979738s) [noise=true , address=77.90.31.17:4795 , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.070731273 (528h0m11.941814308s) [noise=true , address=77.90.31.21:4883 , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.107949174 (528h0m11.979032239s) [noise=true , address=77.90.31.11:4573 , piece_id=SRXHTEXDEKKJ2WPGY66BOAP7WD7Z75HK3B5DL2XIGFWDYQGI76YA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.151478532 (528h0m12.022561567s) [noise=true , address=77.90.31.13:4645 , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.335397247 (528h0m12.206480292s) [noise=true , address=45.91.116.10:4103 , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.410397992 (528h0m12.281481027s) [noise=true , address=77.90.31.18:4840 , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.667101894 (528h0m12.538184929s) [noise=true , address=77.90.31.12:4593 , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.785380610 (528h0m12.656463655s) [noise=true , address=145.79.227.44:4210 , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.828722731 (528h0m12.699805766s) [noise=true , address=193.149.164.143:4697 , piece_id=S4CQ3366KVFS5RMZR67MPGSLXKDWRAGEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.947757466 (528h0m12.818840501s) [noise=true , address=145.79.227.34:4926 , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.947777434 (528h0m12.818860469s) [seg_pos={0 0}, upload=463114]: Segment reached optimal threshold of 14 pieces. Dec 27 15:29:12.947947177 (528h0m12.819030212s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy): protocol: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection Dec 27 15:29:12.947953169 (528h0m12.819036204s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948200800 (528h0m12.819283845s) [noise=true , address=77.90.31.19:4869 , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8): protocol: read tcp 188.166.225.170:32932->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 15:29:12.948216079 (528h0m12.819299124s) [noise=true , address=77.90.31.19:4869 , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948310158 (528h0m12.819393203s) [noise=true , address=64.49.13.169:4711 , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection Dec 27 15:29:12.948316139 (528h0m12.819399174s) [noise=true , address=64.49.13.169:4711 , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948360164 (528h0m12.819443219s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 27 15:29:12.948372617 (528h0m12.819455652s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948398968 (528h0m12.819482013s) [seg_pos={0 0}, upload=463114]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:29:12.948477878 (528h0m12.819560913s) [upload=463114]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:29:12.948483889 (528h0m12.819566934s) [upload=463114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:29:13.237238452 (528h0m13.108321498s) [upload=463114]: Done uploading segments... Dec 27 15:29:13.237262939 (528h0m13.108345984s) [upload=463114]: Upload finished. err: <nil> Duration: 15.461219557s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [X..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XXXXX...X.....X..] duration: 5.001331245s trace: https://jaeger.dev.storj.io/trace/57ec6320e984041b FAIL (2025-12-27 15:33:25 UTC) orbiter warning: context deadline exceeded Duration: 5.001360821s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [XX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XXXXX...X.....X.] duration: 5.00128703s trace: https://jaeger.dev.storj.io/trace/57ec6320e984041b FAIL (2025-12-27 15:33:04 UTC) orbiter warning: context deadline exceeded Duration: 5.001310164s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X..............X....X...................X.....XX......X...] duration: 53.5134836s trace: https://jaeger.dev.storj.io/trace/24120a988203c524 FAIL (2025-12-27 15:32:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 191MKpEnoarS7sK4Gs63ZM8zGc5jevFX8VeqViP82duW33nJJz, Piece ID: 6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A) context deadline exceeded; piecestore: (Node ID: 1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, Piece ID: JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA) context deadline exceeded; piecestore: (Node ID: 12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, Piece ID: L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA) context deadline exceeded ======LOGS====== GEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.918056404 (528h0m9.789139439s) [noise=true , address=77.90.31.13:4645 , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.918708286 (528h0m9.789791331s) [noise=true , address=77.90.31.12:4593 , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.919456690 (528h0m9.790539736s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920039571 (528h0m9.791122616s) [noise=true , address=145.79.227.35:4512 , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920276141 (528h0m9.791359186s) [noise=true , address=145.79.227.34:4926 , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920474529 (528h0m9.791557564s) [noise=true , address=145.79.227.13:4816 , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.921046528 (528h0m9.792129563s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.921603759 (528h0m9.792686794s) [noise=true , address=77.90.31.21:4883 , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.922032846 (528h0m9.793115881s) [noise=true , address=145.79.227.44:4210 , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.923593568 (528h0m9.794676613s) [noise=true , address=45.91.116.10:4103 , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.924048133 (528h0m9.795131169s) [noise=true , address=77.90.31.18:4840 , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.924539719 (528h0m9.795622754s) [noise=true , address=64.49.13.139:4470 , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.925024673 (528h0m9.796107708s) [noise=true , address=145.79.227.49:4368 , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.925500990 (528h0m9.796584035s) [noise=true , address=77.90.31.17:4795 , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:11.205949065 (528h0m11.0770321s) [upload=463114]: Next returned nil segment Dec 27 15:29:11.205963343 (528h0m11.077046378s) [upload=463114]: Waiting for error group managing segments... Dec 27 15:29:11.610155861 (528h0m11.481238906s) [noise=true , address=64.49.13.139:4470 , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.812270605 (528h0m11.683353641s) [noise=true , address=145.79.227.35:4512 , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.818242477 (528h0m11.689325522s) [noise=true , address=145.79.227.13:4816 , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.938211621 (528h0m11.809294666s) [noise=true , address=145.79.227.49:4368 , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.064896693 (528h0m11.935979738s) [noise=true , address=77.90.31.17:4795 , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.070731273 (528h0m11.941814308s) [noise=true , address=77.90.31.21:4883 , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.107949174 (528h0m11.979032239s) [noise=true , address=77.90.31.11:4573 , piece_id=SRXHTEXDEKKJ2WPGY66BOAP7WD7Z75HK3B5DL2XIGFWDYQGI76YA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.151478532 (528h0m12.022561567s) [noise=true , address=77.90.31.13:4645 , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.335397247 (528h0m12.206480292s) [noise=true , address=45.91.116.10:4103 , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.410397992 (528h0m12.281481027s) [noise=true , address=77.90.31.18:4840 , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.667101894 (528h0m12.538184929s) [noise=true , address=77.90.31.12:4593 , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.785380610 (528h0m12.656463655s) [noise=true , address=145.79.227.44:4210 , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.828722731 (528h0m12.699805766s) [noise=true , address=193.149.164.143:4697 , piece_id=S4CQ3366KVFS5RMZR67MPGSLXKDWRAGEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.947757466 (528h0m12.818840501s) [noise=true , address=145.79.227.34:4926 , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.947777434 (528h0m12.818860469s) [seg_pos={0 0}, upload=463114]: Segment reached optimal threshold of 14 pieces. Dec 27 15:29:12.947947177 (528h0m12.819030212s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy): protocol: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection Dec 27 15:29:12.947953169 (528h0m12.819036204s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948200800 (528h0m12.819283845s) [noise=true , address=77.90.31.19:4869 , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8): protocol: read tcp 188.166.225.170:32932->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 15:29:12.948216079 (528h0m12.819299124s) [noise=true , address=77.90.31.19:4869 , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948310158 (528h0m12.819393203s) [noise=true , address=64.49.13.169:4711 , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection Dec 27 15:29:12.948316139 (528h0m12.819399174s) [noise=true , address=64.49.13.169:4711 , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948360164 (528h0m12.819443219s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 27 15:29:12.948372617 (528h0m12.819455652s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948398968 (528h0m12.819482013s) [seg_pos={0 0}, upload=463114]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:29:12.948477878 (528h0m12.819560913s) [upload=463114]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:29:12.948483889 (528h0m12.819566934s) [upload=463114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:29:13.237238452 (528h0m13.108321498s) [upload=463114]: Done uploading segments... Dec 27 15:29:13.237262939 (528h0m13.108345984s) [upload=463114]: Upload finished. err: <nil> Duration: 53.513498158s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XXXXX...X.....X] duration: 5.001521547s trace: https://jaeger.dev.storj.io/trace/57ec6320e984041b FAIL (2025-12-27 15:32:43 UTC) orbiter warning: context deadline exceeded Duration: 5.001575428s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.......X..X.X..X...XXXXXXX..X.X..X.X.....X...X......X.X..X..] duration: 5.706425235s trace: https://jaeger.dev.storj.io/trace/522a43b73399bbba FAIL (2025-12-27 15:32:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 52W7HESUL4YA6OMET3H64EJXHDBESCUC5VAJEEEXEPPM6HBNW7BQ) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: AQFGSNIIUH5ZAA5DSUNN7GZTV3JJ3YRQTVY5XLDLPHPCIYQ4U6QQ) context deadline exceeded ======LOGS====== 2m34.086203997s) [upload=463153]: Uploading segments... Dec 27 15:31:34.215346692 (528h2m34.086429737s) [upload=463153]: Got next segment. Inline: false Dec 27 15:31:34.215374916 (528h2m34.086457951s) [upload=463153]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:31:34.215379064 (528h2m34.086462109s) [upload=463153]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:31:34.462608585 (528h2m34.33369162s) [seg_pos={0 0}, upload=463153]: Begin upload segment... Dec 27 15:31:34.462659352 (528h2m34.333742398s) [seg_pos={0 0}, upload=463153]: Done begin upload segment. Dec 27 15:31:34.462748422 (528h2m34.333831467s) [noise=true , address=145.79.227.52:4445 , piece_id=X2ON65EDXEJVR2NXKMUYGJYJMA62KZ3QXG4VBOUGFDFSG56VWF3A, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.462796123 (528h2m34.333879168s) [noise=true , address=77.90.31.14:4661 , piece_id=AQFGSNIIUH5ZAA5DSUNN7GZTV3JJ3YRQTVY5XLDLPHPCIYQ4U6QQ, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.463437474 (528h2m34.334520519s) [noise=true , address=145.79.227.42:4123 , piece_id=YFKS7437G3M3SMTEHSTUYXMZO2CS7SCZ4YHZODSH4FSL5D64OENA, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.463844980 (528h2m34.334928025s) [noise=true , address=145.79.227.51:4409 , piece_id=CHN6PACDXRAKJECHHWIQWYMXNIOMZCSX6X6YBPJTFBMR5UXC35JA, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.464152144 (528h2m34.335235179s) [noise=true , address=145.79.227.38:4979 , piece_id=UN7DNPRLE2HIO2B3KUXX5Z5554JFMB6XR7ELGUGVGVBZFTXEDODQ, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.464449932 (528h2m34.335532967s) [noise=true , address=145.79.227.11:4729 , piece_id=JWNDP4YNP5FHTRASKVF2LJHZASNOK7T5KRYLEMEIDUGNC2TZ4WCQ, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.464789368 (528h2m34.335872413s) [noise=true , address=145.79.227.3:4149 , piece_id=32LS4UUSAWIDNE6UE4GP2YIXPILHS5IBN6OAPAZ5VA6SQVK42WOQ, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.465139475 (528h2m34.33622251s) [noise=true , address=45.91.116.4:4937 , piece_id=B2U3DALRGH5U2FJLQTTVD2C345ZKIWXP3F57PFDBQNBJAPQY4ARQ, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.465596745 (528h2m34.33667979s) [noise=true , address=45.91.116.10:4084 , piece_id=JUKG7B5WVJ7PWWINVH6VLQH4HVCYHG6IDO2ZD5Y5LBCIJRUCEVMQ, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.467395972 (528h2m34.338479017s) [noise=true , address=45.91.116.12:4225 , piece_id=X2CMM65PCOTDV6SIR6ZNN7F72OICXTQ44CHCKBH62CE45K4EXKJA, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.468960661 (528h2m34.340043706s) [noise=true , address=145.79.227.33:4869 , piece_id=CNIVSRNCUSZJFORFT4SSJLEYACBKLZSW6YPTZPAVC5SHDLJKLPOA, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.470669756 (528h2m34.341752801s) [noise=true , address=145.79.227.50:4373 , piece_id=3QOJVY4KLUIVWT5CYJ3PHS4ZYW4DFYMIB375UR2HBNB4FJEYDMBA, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.472588489 (528h2m34.343671524s) [noise=true , address=145.79.227.36:4519 , piece_id=UEZWKVXIDLC2JIMRPYDFBBES25JWBFEO3JVQ4ESQDOPPXWASPLFQ, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.475052040 (528h2m34.346135075s) [noise=true , address=145.79.227.37:4565 , piece_id=5ZCRHMG2CQPEM4A5R5ZWLUB4KZO2VHD2TDT4AP7Q6IKBYR7YABZQ, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.478883847 (528h2m34.349966892s) [noise=true , address=45.91.116.8:4048 , piece_id=52W7HESUL4YA6OMET3H64EJXHDBESCUC5VAJEEEXEPPM6HBNW7BQ, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.482498600 (528h2m34.353581645s) [noise=true , address=145.79.227.48:4301 , piece_id=2W7VO7654QKTKK5YFWBOTK4H57I3PJ4UIUEISTLOIR2F5WY6BSTQ, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:36.079768051 (528h2m35.950851096s) [upload=463153]: Next returned nil segment Dec 27 15:31:36.079782369 (528h2m35.950865414s) [upload=463153]: Waiting for error group managing segments... Dec 27 15:31:36.456664874 (528h2m36.327747919s) [noise=true , address=145.79.227.51:4409 , piece_id=CHN6PACDXRAKJECHHWIQWYMXNIOMZCSX6X6YBPJTFBMR5UXC35JA, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:36.465589868 (528h2m36.336672913s) [noise=true , address=145.79.227.3:4149 , piece_id=32LS4UUSAWIDNE6UE4GP2YIXPILHS5IBN6OAPAZ5VA6SQVK42WOQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:36.470502723 (528h2m36.341585778s) [noise=true , address=145.79.227.11:4729 , piece_id=JWNDP4YNP5FHTRASKVF2LJHZASNOK7T5KRYLEMEIDUGNC2TZ4WCQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:36.478656260 (528h2m36.349739305s) [noise=true , address=145.79.227.42:4123 , piece_id=YFKS7437G3M3SMTEHSTUYXMZO2CS7SCZ4YHZODSH4FSL5D64OENA, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:36.480936943 (528h2m36.352019978s) [noise=true , address=145.79.227.50:4373 , piece_id=3QOJVY4KLUIVWT5CYJ3PHS4ZYW4DFYMIB375UR2HBNB4FJEYDMBA, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:36.492065783 (528h2m36.363148818s) [noise=true , address=145.79.227.38:4979 , piece_id=UN7DNPRLE2HIO2B3KUXX5Z5554JFMB6XR7ELGUGVGVBZFTXEDODQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:36.494214175 (528h2m36.36529722s) [noise=true , address=145.79.227.52:4445 , piece_id=X2ON65EDXEJVR2NXKMUYGJYJMA62KZ3QXG4VBOUGFDFSG56VWF3A, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:36.496700709 (528h2m36.367783744s) [noise=true , address=145.79.227.48:4301 , piece_id=2W7VO7654QKTKK5YFWBOTK4H57I3PJ4UIUEISTLOIR2F5WY6BSTQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:36.508879120 (528h2m36.379962165s) [noise=true , address=145.79.227.37:4565 , piece_id=5ZCRHMG2CQPEM4A5R5ZWLUB4KZO2VHD2TDT4AP7Q6IKBYR7YABZQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:36.511295511 (528h2m36.382378556s) [noise=true , address=145.79.227.36:4519 , piece_id=UEZWKVXIDLC2JIMRPYDFBBES25JWBFEO3JVQ4ESQDOPPXWASPLFQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:36.673785822 (528h2m36.544868857s) [noise=true , address=77.90.31.14:4661 , piece_id=AQFGSNIIUH5ZAA5DSUNN7GZTV3JJ3YRQTVY5XLDLPHPCIYQ4U6QQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:37.005236026 (528h2m36.876319081s) [noise=true , address=45.91.116.4:4937 , piece_id=B2U3DALRGH5U2FJLQTTVD2C345ZKIWXP3F57PFDBQNBJAPQY4ARQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:37.081575021 (528h2m36.952658056s) [noise=true , address=45.91.116.8:4048 , piece_id=52W7HESUL4YA6OMET3H64EJXHDBESCUC5VAJEEEXEPPM6HBNW7BQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:37.081596342 (528h2m36.952679377s) [seg_pos={0 0}, upload=463153]: Segment reached optimal threshold of 13 pieces. Dec 27 15:31:37.081961918 (528h2m36.953044963s) [noise=true , address=145.79.227.33:4869 , piece_id=CNIVSRNCUSZJFORFT4SSJLEYACBKLZSW6YPTZPAVC5SHDLJKLPOA, seg_pos={0 0}, upload=463153]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 27 15:31:37.081972548 (528h2m36.953055593s) [noise=true , address=145.79.227.33:4869 , piece_id=CNIVSRNCUSZJFORFT4SSJLEYACBKLZSW6YPTZPAVC5SHDLJKLPOA, seg_pos={0 0}, upload=463153]: Long tail cancellation due to optimal threshold Dec 27 15:31:37.082138894 (528h2m36.95322194s) [noise=true , address=45.91.116.10:4084 , piece_id=JUKG7B5WVJ7PWWINVH6VLQH4HVCYHG6IDO2ZD5Y5LBCIJRUCEVMQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:36398->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:36398->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36398->45.91.116.10:4084: use of closed network connection Dec 27 15:31:37.082155006 (528h2m36.953238041s) [noise=true , address=45.91.116.10:4084 , piece_id=JUKG7B5WVJ7PWWINVH6VLQH4HVCYHG6IDO2ZD5Y5LBCIJRUCEVMQ, seg_pos={0 0}, upload=463153]: Long tail cancellation due to optimal threshold Dec 27 15:31:37.082289120 (528h2m36.953372155s) [noise=true , address=45.91.116.12:4225 , piece_id=X2CMM65PCOTDV6SIR6ZNN7F72OICXTQ44CHCKBH62CE45K4EXKJA, seg_pos={0 0}, upload=463153]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:58418->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:58418->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58418->45.91.116.12:4225: use of closed network connection Dec 27 15:31:37.082299571 (528h2m36.953382626s) [noise=true , address=45.91.116.12:4225 , piece_id=X2CMM65PCOTDV6SIR6ZNN7F72OICXTQ44CHCKBH62CE45K4EXKJA, seg_pos={0 0}, upload=463153]: Long tail cancellation due to optimal threshold Dec 27 15:31:37.082325521 (528h2m36.953408566s) [seg_pos={0 0}, upload=463153]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:31:37.082441892 (528h2m36.953524937s) [upload=463153]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:31:37.082447492 (528h2m36.953530527s) [upload=463153]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:31:37.380252539 (528h2m37.251335575s) [upload=463153]: Done uploading segments... Dec 27 15:31:37.380273720 (528h2m37.251356765s) [upload=463153]: Upload finished. err: <nil> Duration: 5.706454341s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.......X..X.X..X...XXXXXXX..X.X..X.X.....X...X......X.X..X.] duration: 6.102248738s trace: https://jaeger.dev.storj.io/trace/522a43b73399bbba FAIL (2025-12-27 15:32:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 52W7HESUL4YA6OMET3H64EJXHDBESCUC5VAJEEEXEPPM6HBNW7BQ) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: AQFGSNIIUH5ZAA5DSUNN7GZTV3JJ3YRQTVY5XLDLPHPCIYQ4U6QQ) context deadline exceeded ======LOGS====== 2m34.086203997s) [upload=463153]: Uploading segments... Dec 27 15:31:34.215346692 (528h2m34.086429737s) [upload=463153]: Got next segment. Inline: false Dec 27 15:31:34.215374916 (528h2m34.086457951s) [upload=463153]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:31:34.215379064 (528h2m34.086462109s) [upload=463153]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:31:34.462608585 (528h2m34.33369162s) [seg_pos={0 0}, upload=463153]: Begin upload segment... Dec 27 15:31:34.462659352 (528h2m34.333742398s) [seg_pos={0 0}, upload=463153]: Done begin upload segment. Dec 27 15:31:34.462748422 (528h2m34.333831467s) [noise=true , address=145.79.227.52:4445 , piece_id=X2ON65EDXEJVR2NXKMUYGJYJMA62KZ3QXG4VBOUGFDFSG56VWF3A, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.462796123 (528h2m34.333879168s) [noise=true , address=77.90.31.14:4661 , piece_id=AQFGSNIIUH5ZAA5DSUNN7GZTV3JJ3YRQTVY5XLDLPHPCIYQ4U6QQ, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.463437474 (528h2m34.334520519s) [noise=true , address=145.79.227.42:4123 , piece_id=YFKS7437G3M3SMTEHSTUYXMZO2CS7SCZ4YHZODSH4FSL5D64OENA, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.463844980 (528h2m34.334928025s) [noise=true , address=145.79.227.51:4409 , piece_id=CHN6PACDXRAKJECHHWIQWYMXNIOMZCSX6X6YBPJTFBMR5UXC35JA, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.464152144 (528h2m34.335235179s) [noise=true , address=145.79.227.38:4979 , piece_id=UN7DNPRLE2HIO2B3KUXX5Z5554JFMB6XR7ELGUGVGVBZFTXEDODQ, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.464449932 (528h2m34.335532967s) [noise=true , address=145.79.227.11:4729 , piece_id=JWNDP4YNP5FHTRASKVF2LJHZASNOK7T5KRYLEMEIDUGNC2TZ4WCQ, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.464789368 (528h2m34.335872413s) [noise=true , address=145.79.227.3:4149 , piece_id=32LS4UUSAWIDNE6UE4GP2YIXPILHS5IBN6OAPAZ5VA6SQVK42WOQ, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.465139475 (528h2m34.33622251s) [noise=true , address=45.91.116.4:4937 , piece_id=B2U3DALRGH5U2FJLQTTVD2C345ZKIWXP3F57PFDBQNBJAPQY4ARQ, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.465596745 (528h2m34.33667979s) [noise=true , address=45.91.116.10:4084 , piece_id=JUKG7B5WVJ7PWWINVH6VLQH4HVCYHG6IDO2ZD5Y5LBCIJRUCEVMQ, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.467395972 (528h2m34.338479017s) [noise=true , address=45.91.116.12:4225 , piece_id=X2CMM65PCOTDV6SIR6ZNN7F72OICXTQ44CHCKBH62CE45K4EXKJA, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.468960661 (528h2m34.340043706s) [noise=true , address=145.79.227.33:4869 , piece_id=CNIVSRNCUSZJFORFT4SSJLEYACBKLZSW6YPTZPAVC5SHDLJKLPOA, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.470669756 (528h2m34.341752801s) [noise=true , address=145.79.227.50:4373 , piece_id=3QOJVY4KLUIVWT5CYJ3PHS4ZYW4DFYMIB375UR2HBNB4FJEYDMBA, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.472588489 (528h2m34.343671524s) [noise=true , address=145.79.227.36:4519 , piece_id=UEZWKVXIDLC2JIMRPYDFBBES25JWBFEO3JVQ4ESQDOPPXWASPLFQ, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.475052040 (528h2m34.346135075s) [noise=true , address=145.79.227.37:4565 , piece_id=5ZCRHMG2CQPEM4A5R5ZWLUB4KZO2VHD2TDT4AP7Q6IKBYR7YABZQ, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.478883847 (528h2m34.349966892s) [noise=true , address=45.91.116.8:4048 , piece_id=52W7HESUL4YA6OMET3H64EJXHDBESCUC5VAJEEEXEPPM6HBNW7BQ, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.482498600 (528h2m34.353581645s) [noise=true , address=145.79.227.48:4301 , piece_id=2W7VO7654QKTKK5YFWBOTK4H57I3PJ4UIUEISTLOIR2F5WY6BSTQ, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:36.079768051 (528h2m35.950851096s) [upload=463153]: Next returned nil segment Dec 27 15:31:36.079782369 (528h2m35.950865414s) [upload=463153]: Waiting for error group managing segments... Dec 27 15:31:36.456664874 (528h2m36.327747919s) [noise=true , address=145.79.227.51:4409 , piece_id=CHN6PACDXRAKJECHHWIQWYMXNIOMZCSX6X6YBPJTFBMR5UXC35JA, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:36.465589868 (528h2m36.336672913s) [noise=true , address=145.79.227.3:4149 , piece_id=32LS4UUSAWIDNE6UE4GP2YIXPILHS5IBN6OAPAZ5VA6SQVK42WOQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:36.470502723 (528h2m36.341585778s) [noise=true , address=145.79.227.11:4729 , piece_id=JWNDP4YNP5FHTRASKVF2LJHZASNOK7T5KRYLEMEIDUGNC2TZ4WCQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:36.478656260 (528h2m36.349739305s) [noise=true , address=145.79.227.42:4123 , piece_id=YFKS7437G3M3SMTEHSTUYXMZO2CS7SCZ4YHZODSH4FSL5D64OENA, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:36.480936943 (528h2m36.352019978s) [noise=true , address=145.79.227.50:4373 , piece_id=3QOJVY4KLUIVWT5CYJ3PHS4ZYW4DFYMIB375UR2HBNB4FJEYDMBA, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:36.492065783 (528h2m36.363148818s) [noise=true , address=145.79.227.38:4979 , piece_id=UN7DNPRLE2HIO2B3KUXX5Z5554JFMB6XR7ELGUGVGVBZFTXEDODQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:36.494214175 (528h2m36.36529722s) [noise=true , address=145.79.227.52:4445 , piece_id=X2ON65EDXEJVR2NXKMUYGJYJMA62KZ3QXG4VBOUGFDFSG56VWF3A, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:36.496700709 (528h2m36.367783744s) [noise=true , address=145.79.227.48:4301 , piece_id=2W7VO7654QKTKK5YFWBOTK4H57I3PJ4UIUEISTLOIR2F5WY6BSTQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:36.508879120 (528h2m36.379962165s) [noise=true , address=145.79.227.37:4565 , piece_id=5ZCRHMG2CQPEM4A5R5ZWLUB4KZO2VHD2TDT4AP7Q6IKBYR7YABZQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:36.511295511 (528h2m36.382378556s) [noise=true , address=145.79.227.36:4519 , piece_id=UEZWKVXIDLC2JIMRPYDFBBES25JWBFEO3JVQ4ESQDOPPXWASPLFQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:36.673785822 (528h2m36.544868857s) [noise=true , address=77.90.31.14:4661 , piece_id=AQFGSNIIUH5ZAA5DSUNN7GZTV3JJ3YRQTVY5XLDLPHPCIYQ4U6QQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:37.005236026 (528h2m36.876319081s) [noise=true , address=45.91.116.4:4937 , piece_id=B2U3DALRGH5U2FJLQTTVD2C345ZKIWXP3F57PFDBQNBJAPQY4ARQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:37.081575021 (528h2m36.952658056s) [noise=true , address=45.91.116.8:4048 , piece_id=52W7HESUL4YA6OMET3H64EJXHDBESCUC5VAJEEEXEPPM6HBNW7BQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:37.081596342 (528h2m36.952679377s) [seg_pos={0 0}, upload=463153]: Segment reached optimal threshold of 13 pieces. Dec 27 15:31:37.081961918 (528h2m36.953044963s) [noise=true , address=145.79.227.33:4869 , piece_id=CNIVSRNCUSZJFORFT4SSJLEYACBKLZSW6YPTZPAVC5SHDLJKLPOA, seg_pos={0 0}, upload=463153]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 27 15:31:37.081972548 (528h2m36.953055593s) [noise=true , address=145.79.227.33:4869 , piece_id=CNIVSRNCUSZJFORFT4SSJLEYACBKLZSW6YPTZPAVC5SHDLJKLPOA, seg_pos={0 0}, upload=463153]: Long tail cancellation due to optimal threshold Dec 27 15:31:37.082138894 (528h2m36.95322194s) [noise=true , address=45.91.116.10:4084 , piece_id=JUKG7B5WVJ7PWWINVH6VLQH4HVCYHG6IDO2ZD5Y5LBCIJRUCEVMQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:36398->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:36398->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36398->45.91.116.10:4084: use of closed network connection Dec 27 15:31:37.082155006 (528h2m36.953238041s) [noise=true , address=45.91.116.10:4084 , piece_id=JUKG7B5WVJ7PWWINVH6VLQH4HVCYHG6IDO2ZD5Y5LBCIJRUCEVMQ, seg_pos={0 0}, upload=463153]: Long tail cancellation due to optimal threshold Dec 27 15:31:37.082289120 (528h2m36.953372155s) [noise=true , address=45.91.116.12:4225 , piece_id=X2CMM65PCOTDV6SIR6ZNN7F72OICXTQ44CHCKBH62CE45K4EXKJA, seg_pos={0 0}, upload=463153]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:58418->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:58418->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58418->45.91.116.12:4225: use of closed network connection Dec 27 15:31:37.082299571 (528h2m36.953382626s) [noise=true , address=45.91.116.12:4225 , piece_id=X2CMM65PCOTDV6SIR6ZNN7F72OICXTQ44CHCKBH62CE45K4EXKJA, seg_pos={0 0}, upload=463153]: Long tail cancellation due to optimal threshold Dec 27 15:31:37.082325521 (528h2m36.953408566s) [seg_pos={0 0}, upload=463153]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:31:37.082441892 (528h2m36.953524937s) [upload=463153]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:31:37.082447492 (528h2m36.953530527s) [upload=463153]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:31:37.380252539 (528h2m37.251335575s) [upload=463153]: Done uploading segments... Dec 27 15:31:37.380273720 (528h2m37.251356765s) [upload=463153]: Upload finished. err: <nil> Duration: 6.102272132s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX.......X..X.X..X...XXXXXXX..X.X..X.X.....X...X......X.X..X] duration: 30.003773982s trace: https://jaeger.dev.storj.io/trace/522a43b73399bbba FAIL (2025-12-27 15:32:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 52W7HESUL4YA6OMET3H64EJXHDBESCUC5VAJEEEXEPPM6HBNW7BQ) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: AQFGSNIIUH5ZAA5DSUNN7GZTV3JJ3YRQTVY5XLDLPHPCIYQ4U6QQ) context deadline exceeded ======LOGS====== 2m34.086203997s) [upload=463153]: Uploading segments... Dec 27 15:31:34.215346692 (528h2m34.086429737s) [upload=463153]: Got next segment. Inline: false Dec 27 15:31:34.215374916 (528h2m34.086457951s) [upload=463153]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 15:31:34.215379064 (528h2m34.086462109s) [upload=463153]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 15:31:34.462608585 (528h2m34.33369162s) [seg_pos={0 0}, upload=463153]: Begin upload segment... Dec 27 15:31:34.462659352 (528h2m34.333742398s) [seg_pos={0 0}, upload=463153]: Done begin upload segment. Dec 27 15:31:34.462748422 (528h2m34.333831467s) [noise=true , address=145.79.227.52:4445 , piece_id=X2ON65EDXEJVR2NXKMUYGJYJMA62KZ3QXG4VBOUGFDFSG56VWF3A, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.462796123 (528h2m34.333879168s) [noise=true , address=77.90.31.14:4661 , piece_id=AQFGSNIIUH5ZAA5DSUNN7GZTV3JJ3YRQTVY5XLDLPHPCIYQ4U6QQ, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.463437474 (528h2m34.334520519s) [noise=true , address=145.79.227.42:4123 , piece_id=YFKS7437G3M3SMTEHSTUYXMZO2CS7SCZ4YHZODSH4FSL5D64OENA, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.463844980 (528h2m34.334928025s) [noise=true , address=145.79.227.51:4409 , piece_id=CHN6PACDXRAKJECHHWIQWYMXNIOMZCSX6X6YBPJTFBMR5UXC35JA, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.464152144 (528h2m34.335235179s) [noise=true , address=145.79.227.38:4979 , piece_id=UN7DNPRLE2HIO2B3KUXX5Z5554JFMB6XR7ELGUGVGVBZFTXEDODQ, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.464449932 (528h2m34.335532967s) [noise=true , address=145.79.227.11:4729 , piece_id=JWNDP4YNP5FHTRASKVF2LJHZASNOK7T5KRYLEMEIDUGNC2TZ4WCQ, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.464789368 (528h2m34.335872413s) [noise=true , address=145.79.227.3:4149 , piece_id=32LS4UUSAWIDNE6UE4GP2YIXPILHS5IBN6OAPAZ5VA6SQVK42WOQ, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.465139475 (528h2m34.33622251s) [noise=true , address=45.91.116.4:4937 , piece_id=B2U3DALRGH5U2FJLQTTVD2C345ZKIWXP3F57PFDBQNBJAPQY4ARQ, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.465596745 (528h2m34.33667979s) [noise=true , address=45.91.116.10:4084 , piece_id=JUKG7B5WVJ7PWWINVH6VLQH4HVCYHG6IDO2ZD5Y5LBCIJRUCEVMQ, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.467395972 (528h2m34.338479017s) [noise=true , address=45.91.116.12:4225 , piece_id=X2CMM65PCOTDV6SIR6ZNN7F72OICXTQ44CHCKBH62CE45K4EXKJA, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.468960661 (528h2m34.340043706s) [noise=true , address=145.79.227.33:4869 , piece_id=CNIVSRNCUSZJFORFT4SSJLEYACBKLZSW6YPTZPAVC5SHDLJKLPOA, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.470669756 (528h2m34.341752801s) [noise=true , address=145.79.227.50:4373 , piece_id=3QOJVY4KLUIVWT5CYJ3PHS4ZYW4DFYMIB375UR2HBNB4FJEYDMBA, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.472588489 (528h2m34.343671524s) [noise=true , address=145.79.227.36:4519 , piece_id=UEZWKVXIDLC2JIMRPYDFBBES25JWBFEO3JVQ4ESQDOPPXWASPLFQ, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.475052040 (528h2m34.346135075s) [noise=true , address=145.79.227.37:4565 , piece_id=5ZCRHMG2CQPEM4A5R5ZWLUB4KZO2VHD2TDT4AP7Q6IKBYR7YABZQ, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.478883847 (528h2m34.349966892s) [noise=true , address=45.91.116.8:4048 , piece_id=52W7HESUL4YA6OMET3H64EJXHDBESCUC5VAJEEEXEPPM6HBNW7BQ, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:34.482498600 (528h2m34.353581645s) [noise=true , address=145.79.227.48:4301 , piece_id=2W7VO7654QKTKK5YFWBOTK4H57I3PJ4UIUEISTLOIR2F5WY6BSTQ, seg_pos={0 0}, upload=463153]: Uploading piece... Dec 27 15:31:36.079768051 (528h2m35.950851096s) [upload=463153]: Next returned nil segment Dec 27 15:31:36.079782369 (528h2m35.950865414s) [upload=463153]: Waiting for error group managing segments... Dec 27 15:31:36.456664874 (528h2m36.327747919s) [noise=true , address=145.79.227.51:4409 , piece_id=CHN6PACDXRAKJECHHWIQWYMXNIOMZCSX6X6YBPJTFBMR5UXC35JA, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:36.465589868 (528h2m36.336672913s) [noise=true , address=145.79.227.3:4149 , piece_id=32LS4UUSAWIDNE6UE4GP2YIXPILHS5IBN6OAPAZ5VA6SQVK42WOQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:36.470502723 (528h2m36.341585778s) [noise=true , address=145.79.227.11:4729 , piece_id=JWNDP4YNP5FHTRASKVF2LJHZASNOK7T5KRYLEMEIDUGNC2TZ4WCQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:36.478656260 (528h2m36.349739305s) [noise=true , address=145.79.227.42:4123 , piece_id=YFKS7437G3M3SMTEHSTUYXMZO2CS7SCZ4YHZODSH4FSL5D64OENA, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:36.480936943 (528h2m36.352019978s) [noise=true , address=145.79.227.50:4373 , piece_id=3QOJVY4KLUIVWT5CYJ3PHS4ZYW4DFYMIB375UR2HBNB4FJEYDMBA, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:36.492065783 (528h2m36.363148818s) [noise=true , address=145.79.227.38:4979 , piece_id=UN7DNPRLE2HIO2B3KUXX5Z5554JFMB6XR7ELGUGVGVBZFTXEDODQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:36.494214175 (528h2m36.36529722s) [noise=true , address=145.79.227.52:4445 , piece_id=X2ON65EDXEJVR2NXKMUYGJYJMA62KZ3QXG4VBOUGFDFSG56VWF3A, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:36.496700709 (528h2m36.367783744s) [noise=true , address=145.79.227.48:4301 , piece_id=2W7VO7654QKTKK5YFWBOTK4H57I3PJ4UIUEISTLOIR2F5WY6BSTQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:36.508879120 (528h2m36.379962165s) [noise=true , address=145.79.227.37:4565 , piece_id=5ZCRHMG2CQPEM4A5R5ZWLUB4KZO2VHD2TDT4AP7Q6IKBYR7YABZQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:36.511295511 (528h2m36.382378556s) [noise=true , address=145.79.227.36:4519 , piece_id=UEZWKVXIDLC2JIMRPYDFBBES25JWBFEO3JVQ4ESQDOPPXWASPLFQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:36.673785822 (528h2m36.544868857s) [noise=true , address=77.90.31.14:4661 , piece_id=AQFGSNIIUH5ZAA5DSUNN7GZTV3JJ3YRQTVY5XLDLPHPCIYQ4U6QQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:37.005236026 (528h2m36.876319081s) [noise=true , address=45.91.116.4:4937 , piece_id=B2U3DALRGH5U2FJLQTTVD2C345ZKIWXP3F57PFDBQNBJAPQY4ARQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:37.081575021 (528h2m36.952658056s) [noise=true , address=45.91.116.8:4048 , piece_id=52W7HESUL4YA6OMET3H64EJXHDBESCUC5VAJEEEXEPPM6HBNW7BQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: <nil> Dec 27 15:31:37.081596342 (528h2m36.952679377s) [seg_pos={0 0}, upload=463153]: Segment reached optimal threshold of 13 pieces. Dec 27 15:31:37.081961918 (528h2m36.953044963s) [noise=true , address=145.79.227.33:4869 , piece_id=CNIVSRNCUSZJFORFT4SSJLEYACBKLZSW6YPTZPAVC5SHDLJKLPOA, seg_pos={0 0}, upload=463153]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 27 15:31:37.081972548 (528h2m36.953055593s) [noise=true , address=145.79.227.33:4869 , piece_id=CNIVSRNCUSZJFORFT4SSJLEYACBKLZSW6YPTZPAVC5SHDLJKLPOA, seg_pos={0 0}, upload=463153]: Long tail cancellation due to optimal threshold Dec 27 15:31:37.082138894 (528h2m36.95322194s) [noise=true , address=45.91.116.10:4084 , piece_id=JUKG7B5WVJ7PWWINVH6VLQH4HVCYHG6IDO2ZD5Y5LBCIJRUCEVMQ, seg_pos={0 0}, upload=463153]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:36398->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:36398->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36398->45.91.116.10:4084: use of closed network connection Dec 27 15:31:37.082155006 (528h2m36.953238041s) [noise=true , address=45.91.116.10:4084 , piece_id=JUKG7B5WVJ7PWWINVH6VLQH4HVCYHG6IDO2ZD5Y5LBCIJRUCEVMQ, seg_pos={0 0}, upload=463153]: Long tail cancellation due to optimal threshold Dec 27 15:31:37.082289120 (528h2m36.953372155s) [noise=true , address=45.91.116.12:4225 , piece_id=X2CMM65PCOTDV6SIR6ZNN7F72OICXTQ44CHCKBH62CE45K4EXKJA, seg_pos={0 0}, upload=463153]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:58418->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:58418->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58418->45.91.116.12:4225: use of closed network connection Dec 27 15:31:37.082299571 (528h2m36.953382626s) [noise=true , address=45.91.116.12:4225 , piece_id=X2CMM65PCOTDV6SIR6ZNN7F72OICXTQ44CHCKBH62CE45K4EXKJA, seg_pos={0 0}, upload=463153]: Long tail cancellation due to optimal threshold Dec 27 15:31:37.082325521 (528h2m36.953408566s) [seg_pos={0 0}, upload=463153]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 15:31:37.082441892 (528h2m36.953524937s) [upload=463153]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:31:37.082447492 (528h2m36.953530527s) [upload=463153]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:31:37.380252539 (528h2m37.251335575s) [upload=463153]: Done uploading segments... Dec 27 15:31:37.380273720 (528h2m37.251356765s) [upload=463153]: Upload finished. err: <nil> Duration: 30.003800031s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X..............X....X...................X.....XX......X..] duration: 51.107385063s trace: https://jaeger.dev.storj.io/trace/24120a988203c524 FAIL (2025-12-27 15:31:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 191MKpEnoarS7sK4Gs63ZM8zGc5jevFX8VeqViP82duW33nJJz, Piece ID: 6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A) context deadline exceeded; piecestore: (Node ID: 1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, Piece ID: JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA) context deadline exceeded; piecestore: (Node ID: 12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, Piece ID: L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA) context deadline exceeded ======LOGS====== GEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.918056404 (528h0m9.789139439s) [noise=true , address=77.90.31.13:4645 , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.918708286 (528h0m9.789791331s) [noise=true , address=77.90.31.12:4593 , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.919456690 (528h0m9.790539736s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920039571 (528h0m9.791122616s) [noise=true , address=145.79.227.35:4512 , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920276141 (528h0m9.791359186s) [noise=true , address=145.79.227.34:4926 , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920474529 (528h0m9.791557564s) [noise=true , address=145.79.227.13:4816 , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.921046528 (528h0m9.792129563s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.921603759 (528h0m9.792686794s) [noise=true , address=77.90.31.21:4883 , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.922032846 (528h0m9.793115881s) [noise=true , address=145.79.227.44:4210 , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.923593568 (528h0m9.794676613s) [noise=true , address=45.91.116.10:4103 , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.924048133 (528h0m9.795131169s) [noise=true , address=77.90.31.18:4840 , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.924539719 (528h0m9.795622754s) [noise=true , address=64.49.13.139:4470 , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.925024673 (528h0m9.796107708s) [noise=true , address=145.79.227.49:4368 , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.925500990 (528h0m9.796584035s) [noise=true , address=77.90.31.17:4795 , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:11.205949065 (528h0m11.0770321s) [upload=463114]: Next returned nil segment Dec 27 15:29:11.205963343 (528h0m11.077046378s) [upload=463114]: Waiting for error group managing segments... Dec 27 15:29:11.610155861 (528h0m11.481238906s) [noise=true , address=64.49.13.139:4470 , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.812270605 (528h0m11.683353641s) [noise=true , address=145.79.227.35:4512 , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.818242477 (528h0m11.689325522s) [noise=true , address=145.79.227.13:4816 , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.938211621 (528h0m11.809294666s) [noise=true , address=145.79.227.49:4368 , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.064896693 (528h0m11.935979738s) [noise=true , address=77.90.31.17:4795 , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.070731273 (528h0m11.941814308s) [noise=true , address=77.90.31.21:4883 , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.107949174 (528h0m11.979032239s) [noise=true , address=77.90.31.11:4573 , piece_id=SRXHTEXDEKKJ2WPGY66BOAP7WD7Z75HK3B5DL2XIGFWDYQGI76YA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.151478532 (528h0m12.022561567s) [noise=true , address=77.90.31.13:4645 , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.335397247 (528h0m12.206480292s) [noise=true , address=45.91.116.10:4103 , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.410397992 (528h0m12.281481027s) [noise=true , address=77.90.31.18:4840 , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.667101894 (528h0m12.538184929s) [noise=true , address=77.90.31.12:4593 , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.785380610 (528h0m12.656463655s) [noise=true , address=145.79.227.44:4210 , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.828722731 (528h0m12.699805766s) [noise=true , address=193.149.164.143:4697 , piece_id=S4CQ3366KVFS5RMZR67MPGSLXKDWRAGEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.947757466 (528h0m12.818840501s) [noise=true , address=145.79.227.34:4926 , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.947777434 (528h0m12.818860469s) [seg_pos={0 0}, upload=463114]: Segment reached optimal threshold of 14 pieces. Dec 27 15:29:12.947947177 (528h0m12.819030212s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy): protocol: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection Dec 27 15:29:12.947953169 (528h0m12.819036204s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948200800 (528h0m12.819283845s) [noise=true , address=77.90.31.19:4869 , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8): protocol: read tcp 188.166.225.170:32932->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 15:29:12.948216079 (528h0m12.819299124s) [noise=true , address=77.90.31.19:4869 , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948310158 (528h0m12.819393203s) [noise=true , address=64.49.13.169:4711 , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection Dec 27 15:29:12.948316139 (528h0m12.819399174s) [noise=true , address=64.49.13.169:4711 , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948360164 (528h0m12.819443219s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 27 15:29:12.948372617 (528h0m12.819455652s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948398968 (528h0m12.819482013s) [seg_pos={0 0}, upload=463114]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:29:12.948477878 (528h0m12.819560913s) [upload=463114]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:29:12.948483889 (528h0m12.819566934s) [upload=463114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:29:13.237238452 (528h0m13.108321498s) [upload=463114]: Done uploading segments... Dec 27 15:29:13.237262939 (528h0m13.108345984s) [upload=463114]: Upload finished. err: <nil> Duration: 51.107405182s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [.XXX..X..XX.XXXX.....XXX.X.XXXX.X.XXXX...XXXX.XXXXX...X.....] duration: 3.434797318s trace: https://jaeger.dev.storj.io/trace/5973653b14c3bbcb FAIL (2025-12-27 15:31:29 UTC) orbiter warning: context deadline exceeded Duration: 3.434819521s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X..............X....X...................X.....XX......X.] duration: 58.584714521s trace: https://jaeger.dev.storj.io/trace/24120a988203c524 FAIL (2025-12-27 15:31:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 191MKpEnoarS7sK4Gs63ZM8zGc5jevFX8VeqViP82duW33nJJz, Piece ID: 6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A) context deadline exceeded; piecestore: (Node ID: 1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, Piece ID: JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA) context deadline exceeded; piecestore: (Node ID: 12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, Piece ID: L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA) context deadline exceeded ======LOGS====== GEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.918056404 (528h0m9.789139439s) [noise=true , address=77.90.31.13:4645 , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.918708286 (528h0m9.789791331s) [noise=true , address=77.90.31.12:4593 , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.919456690 (528h0m9.790539736s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920039571 (528h0m9.791122616s) [noise=true , address=145.79.227.35:4512 , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920276141 (528h0m9.791359186s) [noise=true , address=145.79.227.34:4926 , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.920474529 (528h0m9.791557564s) [noise=true , address=145.79.227.13:4816 , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.921046528 (528h0m9.792129563s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.921603759 (528h0m9.792686794s) [noise=true , address=77.90.31.21:4883 , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.922032846 (528h0m9.793115881s) [noise=true , address=145.79.227.44:4210 , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.923593568 (528h0m9.794676613s) [noise=true , address=45.91.116.10:4103 , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.924048133 (528h0m9.795131169s) [noise=true , address=77.90.31.18:4840 , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.924539719 (528h0m9.795622754s) [noise=true , address=64.49.13.139:4470 , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.925024673 (528h0m9.796107708s) [noise=true , address=145.79.227.49:4368 , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:09.925500990 (528h0m9.796584035s) [noise=true , address=77.90.31.17:4795 , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Uploading piece... Dec 27 15:29:11.205949065 (528h0m11.0770321s) [upload=463114]: Next returned nil segment Dec 27 15:29:11.205963343 (528h0m11.077046378s) [upload=463114]: Waiting for error group managing segments... Dec 27 15:29:11.610155861 (528h0m11.481238906s) [noise=true , address=64.49.13.139:4470 , piece_id=3MAQ4R3XTRQT72O3DEIQSAFZAGT2CJTMPT2IKLCDLIVLPESQRVGA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.812270605 (528h0m11.683353641s) [noise=true , address=145.79.227.35:4512 , piece_id=DX4PHB5M4NDOFTWWEGJM43D4VBIRIGADX4HILEMRO2ZU4OV2S47Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.818242477 (528h0m11.689325522s) [noise=true , address=145.79.227.13:4816 , piece_id=2OCASX7RNCMKBIVDVELFO7WIHKZMY27RII5C2BWIZXIG33BBAV5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:11.938211621 (528h0m11.809294666s) [noise=true , address=145.79.227.49:4368 , piece_id=3LXJYQGC74WHBFCISTNZMBVLSUGSMU7XOZIFP5ZRU57TXJ3YTM4Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.064896693 (528h0m11.935979738s) [noise=true , address=77.90.31.17:4795 , piece_id=FTJLV46VS5BACPHHVVOCU7EEOEU3Q66P3B3WTPUT3ACG6GLYTDUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.070731273 (528h0m11.941814308s) [noise=true , address=77.90.31.21:4883 , piece_id=6GXHQB47DWIXBK4ZTC5QIA4362JRMBOQKXXGBZ3M6FVCLFSZIE2A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.107949174 (528h0m11.979032239s) [noise=true , address=77.90.31.11:4573 , piece_id=SRXHTEXDEKKJ2WPGY66BOAP7WD7Z75HK3B5DL2XIGFWDYQGI76YA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.151478532 (528h0m12.022561567s) [noise=true , address=77.90.31.13:4645 , piece_id=JNNC647LJKVXZIIWY4GXSKOJ3OY3WWGPHWKFWGO5TI6M2SWIXHIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.335397247 (528h0m12.206480292s) [noise=true , address=45.91.116.10:4103 , piece_id=QU2UHH6G7QETR2DF3A6U3NZLYXY3WD5JO4GKU3HRUO35CJJP5LJA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.410397992 (528h0m12.281481027s) [noise=true , address=77.90.31.18:4840 , piece_id=L2BJ6DADQGARNEFIF7M4AKQ4RDDDER64NZUKOYKAT7QBMMOWVEIA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.667101894 (528h0m12.538184929s) [noise=true , address=77.90.31.12:4593 , piece_id=RMUWIRJ2XRORPV4RJKRL2JUZZOMMOGO3TDTWM22B7LTCERERYHWQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.785380610 (528h0m12.656463655s) [noise=true , address=145.79.227.44:4210 , piece_id=JMSSYMTYXUCJRFCOUCVRK4W6KX7LTGSCKKAO7XC7UGWKNGHNEMQQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.828722731 (528h0m12.699805766s) [noise=true , address=193.149.164.143:4697 , piece_id=S4CQ3366KVFS5RMZR67MPGSLXKDWRAGEJCQQVP2NVYJ6LZWA3HUA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.947757466 (528h0m12.818840501s) [noise=true , address=145.79.227.34:4926 , piece_id=LE5TDKNEJ2QPKW2ZIMMPBO56TAQRGFTSCAR5IHD66X4E6FIYXT5Q, seg_pos={0 0}, upload=463114]: Done uploading piece. err: <nil> Dec 27 15:29:12.947777434 (528h0m12.818860469s) [seg_pos={0 0}, upload=463114]: Segment reached optimal threshold of 14 pieces. Dec 27 15:29:12.947947177 (528h0m12.819030212s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12q2kob5oYxuxwUDh4vjNDBpW4c6LndrjHbsJ4VRDiBsLxjgLdy): protocol: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36046->77.90.31.14:4695: use of closed network connection Dec 27 15:29:12.947953169 (528h0m12.819036204s) [noise=true , address=77.90.31.14:4695 , piece_id=H265WN7TXOO3LG4QTFAB5S3TE2MUON7OCCQ7H5PA5NIBCEVGOM4A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948200800 (528h0m12.819283845s) [noise=true , address=77.90.31.19:4869 , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8): protocol: read tcp 188.166.225.170:32932->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 15:29:12.948216079 (528h0m12.819299124s) [noise=true , address=77.90.31.19:4869 , piece_id=KENCENCTEBHPVIPCECDQYLIWHYLT6G7ZETYMN4DA6LPQRLVQETNA, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948310158 (528h0m12.819393203s) [noise=true , address=64.49.13.169:4711 , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51826->64.49.13.169:4711: use of closed network connection Dec 27 15:29:12.948316139 (528h0m12.819399174s) [noise=true , address=64.49.13.169:4711 , piece_id=IMYFGTTZJUS5CEYCF75SUFUIGA3QFQYTFVCKYKFBM2DDQIKQHS7A, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948360164 (528h0m12.819443219s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: expected piece hash; context canceled; EOF Dec 27 15:29:12.948372617 (528h0m12.819455652s) [noise=true , address=145.79.227.40:4069 , piece_id=Z3ZYDWOIMGFRIRYP4WKYSL4YIJF7TZT73EOVUYQOOMNM42PGWHBQ, seg_pos={0 0}, upload=463114]: Long tail cancellation due to optimal threshold Dec 27 15:29:12.948398968 (528h0m12.819482013s) [seg_pos={0 0}, upload=463114]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:29:12.948477878 (528h0m12.819560913s) [upload=463114]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:29:12.948483889 (528h0m12.819566934s) [upload=463114]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:29:13.237238452 (528h0m13.108321498s) [upload=463114]: Done uploading segments... Dec 27 15:29:13.237262939 (528h0m13.108345984s) [upload=463114]: Upload finished. err: <nil> Duration: 58.584738598s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)